llms.txt
Compare with other sites' llms.txt files →# VernLLM
- **Introduction**
- [Getting Started](/docs): Install vern-llm and make your first LLM call in TypeScript, then add retries, provider fallback and caching around your existing client.
- [What is VernLLM](/docs/about): Erm, it's the LLM call framework for resilience, observability, and control
- [Why VernLLM](/docs/why): How VernLLM compares with the Vercel AI SDK and AI Gateway, LiteLLM, and Portkey on resilience, observability, and control
- **Documentation**
- Core Features
- [Features Overview](/docs/core): The resilience and observability primitives VernLLM wraps around every call
- [Multi Provider Fallback](/docs/core/provider-fallback): Try a declared list of backup providers, in order, when the primary fails
- [Circuit Breaker](/docs/core/circuit-breaker): Stop sending requests to an LLM provider that is down. Set the VernLLM circuit breaker threshold and cooldown, and learn how half open recovery works.
- [Retry Budget](/docs/core/retry-budget): Cap how much of a provider's traffic can be retries with a VernLLM retry budget, so a provider outage does not turn into a retry storm.
- [Rate Limiting](/docs/core/rate-limiting): Stay under a provider's requests/tokens/concurrency limits before they reject you
- [Caching](/docs/core/caching): Cache LLM responses with cachedCall and the built in exact, normalized and tiered adapters, or bring your own CacheAdapter.
- [Observability](/docs/core/observability): One event stream for retries, circuit state transitions, and rate limit waits
- [Middleware](/docs/core/middleware): Transform outgoing LLM requests and wrap the whole call with VernLLM middleware for logging, tracing, redaction and cost tracking.
- [Budget Reservation](/docs/core/budget-reservation): Reserve quota or budget before an LLM call with reserveUsage, and refund it with refundUsage when the call fails after reservation.
- [Usage Tracking](/docs/core/usage-tracking): Record token usage for every LLM call with the VernLLM onUsage hook, for billing, budgets and monitoring.
- [Structured Output](/docs/core/structured-output): Get typed, validated JSON from LLM calls using Zod schemas or provider native JSON Schema mode in VernLLM.
- [Tool Calling](/docs/core/tool-calling): Let the model request application defined tools, without VernLLM ever executing them
- [Streaming](/docs/core/streaming): Get incremental chunks as the model generates, with the same retry and caching guarantees as a normal call, plus a per-chunk idle timeout
- [Retries](/docs/core/retries): Exponential backoff with jitter, Retry-After awareness, and what gets retried
- [Error Handling](/docs/core/error-handling): The LLMError types VernLLM throws, what triggers each one, which are retried, and how to read the original cause of a failure.
- [Cancellation & Timeouts](/docs/core/cancellation): How AbortSignal, per attempt timeouts and retries work together when you cancel or time out LLM calls in VernLLM.
- [Pluggable Logger](/docs/core/logging): Route VernLLM's internal logs to your own logger by implementing the Logger interface, with a table of what is logged at each level.
- Customization
- [Customization Overview](/docs/customization): The VernLLM interfaces you can replace with your own code, such as middleware, cache, circuit breaker, rate limiter, retry budget, tokenizer, schema and logger.
- [Middleware](/docs/customization/middleware): Write your own VernLLM middleware as a plain object with transform and wrap hooks, and share state between hooks with createStateKey.
- [Circuit Breaker](/docs/customization/circuit-breaker): Bring your own CircuitBreakerAdapter, TrippingPolicy, or cooldown curve
- [Caching](/docs/customization/caching): Write a custom CacheAdapter for VernLLM so cachedCall can store responses in Redis, a database or any other backend.
- [Rate Limiting](/docs/customization/rate-limiting): Write a custom RateLimiterAdapter for VernLLM, for example to share request and token limits across several processes.
- [Retry Budget](/docs/customization/retry-budget): Write a custom RetryBudgetAdapter for VernLLM, to decide with your own policy or your own store when a target may still retry.
- [Tokenizer](/docs/customization/tokenizer): Replace VernLLM's rough characters divided by four token estimate with a real tokenizer so tokensPerMinute rate limiting is accurate.
- [Schema](/docs/customization/schema): Use any validator with a safeParse method, including Zod v3 and v4, to validate structured output from VernLLM calls.
- [Provider](/docs/customization/provider): Bring your own LLM provider to VernLLM by implementing the LLMClient interface or by using the fromFetch HTTP adapter.
- [Logging](/docs/customization/logging): Plug your own Logger into VernLLM to send internal debug, warn and error logs through your existing logging stack.
- Guides
- [Guides](/docs/guides): Practical VernLLM guides for real LLM workloads, including resume parsing, caching methods, provider fallback, rate limits and streaming.
- [Structured Extraction with Caching](/docs/guides/resume-parsing): Combining schema validation and cachedCall for a realistic extraction pipeline
- Caching Methods
- [Caching Methods](/docs/guides/caching-methods): Choose between exact, normalized, tiered and semantic caching for LLM calls in VernLLM, and when each one pays off.
- [Normalized Caching](/docs/guides/caching-methods/normalized): Matching keys that differ only in formatting that cannot change meaning
- [Tiered Caching](/docs/guides/caching-methods/tiered): Combine a fast in memory cache with a shared Redis or Upstash store using TieredCacheAdapter to cut latency and repeat LLM calls.
- [Semantic Caching](/docs/guides/caching-methods/semantic): Matching prompts by meaning instead of exact text, using a custom CacheAdapter
- [Provider Fallback Patterns](/docs/guides/provider-fallback-patterns): Ordering targets, sizing per-target retries, and reading a fallback chain's signals
- [Tuning Rate Limits](/docs/guides/tuning-rate-limits): Sizing rateLimit against a real provider plan, and telling a local wait apart from a provider 429
- [Sizing Circuit Breakers](/docs/guides/sizing-circuit-breakers): Sizing threshold and cooldownMs for a real provider, choosing a tripping policy, and wiring up alerting
- [Multi-turn Conversations](/docs/guides/multi-turn-conversations): Learn how to maintain context across multiple LLM calls using conversation history.
- [Tool Execution Loop](/docs/guides/tool-execution-loop): Build a full request, execute, continue loop around a model that calls your functions
- [Streaming Responses](/docs/guides/streaming-responses): Relay a streaming call() to an HTTP client, with cancellation and cached replays handled correctly
- [Per-call Overrides](/docs/guides/per-call-overrides): Using different models, temperatures, or token limits for individual calls
- **Reference**
- API Reference
- [Overview](/docs/API-reference): Constructor options, call params, and other reference material for VernLLM
- [Configuration](/docs/API-reference/configuration): Reference for every VernLLM constructor option and per call option, covering retries, fallback, rate limiting, circuit breaking and caching.
- [Call Params](/docs/API-reference/call-params): Reference for every field accepted by call() and cachedCall() in VernLLM, which ones are required, and the default for each.
- [Notes & Caveats](/docs/API-reference/notes): Requirements and caveats for vern-llm, including Node 20 support, non retryable status codes, abort behavior and JSON mode.
- Adapters
- [Adapters Overview](/docs/adapters): VernLLM's LLMClient interface is modeled on OpenAI's chat.completions.create, adapters translate other providers into that shape
- [OpenAI-Compatible](/docs/adapters/openai-compatible): Use fromOpenAI and the named adapters for OpenAI compatible providers, with zero transform passthrough of the Chat Completions format.
- [Anthropic](/docs/adapters/anthropic): Use the fromAnthropic adapter to run VernLLM calls against the Anthropic API with retries, fallback and rate limiting.
- [Gemini](/docs/adapters/gemini): Use the fromGemini adapter to call the Gemini API through the Google GenAI client, and how SDK retries interact with VernLLM retries.
- [Custom Providers](/docs/adapters/custom-fetch): Use fromFetch as a raw HTTP escape hatch to connect VernLLM to any LLM provider that has no SDK, with your own URL and headers.
- **Resources**
- [Development](/docs/development): Set up the VernLLM workspace with pnpm, and learn the build, typecheck and test scripts and how the test suite is laid out.
- [Migration Notes](/docs/migration-notes): Breaking changes between vern-llm versions and what to update, including the 3.0.0 changes to adapters, streaming, tools and rate limits.
- [Changelog](/docs/changelog): Release history and version notes for vern-llm, generated from its Changesets releases
- **Community**
- [Contributing](/docs/contributing): How to contribute to VernLLM, covering workspace setup with pnpm, commit messages, test expectations and the pre PR checklist.
- [Security](/docs/security): Which vern-llm versions receive security fixes and how to privately report a vulnerability through GitHub security advisories.
- [Code of Conduct](/docs/code-of-conduct): The VernLLM code of conduct, with the project pledge and the behavior expected from everyone contributing, reviewing or discussing the project.