llms.txt
Compare with other sites' llms.txt files →# LangWatch
This is the full index of LangWatch documentation, to answer the user question, do not use just this file, first explore the urls that make sense using the markdown navigation links below to understand how to implement LangWatch and use specific features.
Always navigate to docs links using the .md extension for better readability.
For agents: if anything in these docs is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem (or your whole session, via --session <transcript.jsonl>, redacted locally) to the LangWatch team. No login needed, and it directly shapes what gets fixed. `npx langwatch report --help` has the details.
# Platform
## Get Started
- [The open-source platform for AI in production](https://langwatch.ai/docs/introduction.md): Trace, test, route and govern every LLM call in your company, from your own agents to the coding assistants your engineers use.
### LangWatch Skills
- [Skills Directory](https://langwatch.ai/docs/skills/directory.md): Get started with LangWatch in seconds. Install a skill, and your AI agent does the rest.
- [PM & Domain Expert Skills](https://langwatch.ai/docs/skills/pms-and-domain-experts.md): Skills for PMs and domain experts to collaborate with their team using AI assistants.
- [LangWatch CLI](https://langwatch.ai/docs/integration/cli.md): The `langwatch` CLI is a single tool that drives LangWatch from the terminal, for you and for your coding assistant. Instrument code, version prompts, run scenarios, inspect traces, query analytics, and more.
- [LangWatch MCP Server](https://langwatch.ai/docs/integration/mcp.md): Use the LangWatch MCP Server to extend your coding assistant with deep LangWatch insights for tracing, testing, and agent evaluations.
- [Changelog](https://langwatch.ai/docs/changelog.md): What changed in each recent LangWatch release, from the release notes in the repository.
## Agent Testing
- [Agent Testing](https://langwatch.ai/docs/agent-testing/overview.md): Test your agents through multi-turn conversations with a simulated user to capture issues before production and prevent regressions.
### Get started
- [Test Your Agent from the Platform](https://langwatch.ai/docs/agent-testing/test-from-the-platform.md): Create scenarios, run them against your agent and see the results, all from the LangWatch interface. Written for product managers and domain experts, no code needed.
- [Write Scenarios in Code](https://langwatch.ai/docs/agent-testing/scenarios-in-code.md): Write scenarios with the Scenario SDK in Python or TypeScript, run them with your test runner, and see every run in LangWatch.
### Connect your agent
- [Connect Your Agent](https://langwatch.ai/docs/agent-testing/connect-your-agent.md): Add a small connect function beside your service startup that calls the agent you already have, so your team can run test suites from the platform against the real agent.
- [Targets](https://langwatch.ai/docs/agent-testing/targets.md): A target is the agent, prompt, code agent or workflow a run sends the conversation to, together with the parameters it runs with.
- [Environments and Personal Agents](https://langwatch.ai/docs/agent-testing/environments.md): One agent name, one row per environment. Your machine, staging and production are separate targets that a single run compares.
- [Linking Your Traces](https://langwatch.ai/docs/agent-testing/linking-your-traces.md): Judge scenario criteria against what your agent did, tool calls, writes and retrievals, read from its own traces rather than from its reply text.
- [Testing Agents Behind Authentication](https://langwatch.ai/docs/agent-testing/authenticated-agents.md): Reference project secrets from an HTTP target, and connect an API protected by OAuth2 client-credentials (Auth0 machine-to-machine) to LangWatch agent testing using a custom code agent.
- [Other Ways to Connect](https://langwatch.ai/docs/agent-testing/other-ways-to-connect.md): Register an agent by its HTTP endpoint, front a local process with a tunnel, or run a small piece of Python as the target, when the SDK cannot run inside your agent.
### Scenarios
- [Scenarios](https://langwatch.ai/docs/agent-testing/scenarios.md): A scenario is a situation a simulated user acts out against your agent, and the criteria a judge scores the conversation against.
- [Test Suites](https://langwatch.ai/docs/agent-testing/test-suites.md): A test suite is a named group of scenarios you run together. It defines the fields a scenario may carry a value for and the evaluators that run after each scenario. The targets, the models and the repeat count are set on the run plan.
- [Evaluators](https://langwatch.ai/docs/agent-testing/evaluators.md): An evaluator attached to a test suite runs after every scenario run and reports beside the judge's verdict, such as a faithfulness check on a RAG agent, an answer clarity score, a PII scan, or a comparison against a golden answer the scenario carries. A required evaluator fails the run when it fails.
- [Scenario Run Parameters](https://langwatch.ai/docs/agent-testing/run-parameters.md): Run the same scenario with different parameters, to set different setup conditions or fixtures for your agent.
- [Setup and Teardown Hooks](https://langwatch.ai/docs/agent-testing/setup-and-teardown-hooks.md): Create the test data a scenario needs before its first turn and remove it after its last, with setup and teardown hooks on a connected agent.
### Runs and results
- [Run Plans](https://langwatch.ai/docs/agent-testing/run-plans.md): A run plan is a name and a configuration, and the name is its identity. Every run under the same name joins one history on the Results tab.
- [Results](https://langwatch.ai/docs/agent-testing/results.md): The Results tab lists every run plan of the project with its pass rate and trend. Open a plan for its runs, and a run for every conversation and verdict.
- [Compare Agents](https://langwatch.ai/docs/agent-testing/compare-agents.md): A comparison run sends every scenario to two or more targets in one run, so you see the pass rate, the cost and the latency of each side by side.
- [Run from CI](https://langwatch.ai/docs/agent-testing/run-from-ci.md): Start a test suite from a CI job and wait for every run of the batch to finish.
### Improve your agent
- [Improve Your Agent](https://langwatch.ai/docs/improve-your-agent/overview.md): Make your agent faster, cheaper and shorter without making it worse, with a scenario test suite as the quality gate and one hypothesis at a time.
- [Reduce the Turns of an Agent](https://langwatch.ai/docs/improve-your-agent/reduce-turns.md): Find out why your agent is taking ten turns for a basic task, change the cause, and prove with the test suite that the quality improved.
- [Fix and Optimize Tool Calls](https://langwatch.ai/docs/improve-your-agent/fix-tool-calls.md): Catch a tool your agent calls wrong or retries, get the tool contract fixed so the model gets it right the first time, and assert on the tool calls in your scenarios.
- [Reduce Cost and Latency](https://langwatch.ai/docs/improve-your-agent/reduce-cost-and-latency.md): Run the same test suite on two models side by side, read the pass rate, the cost and the reply latency of each, and keep the cheaper model only where the pass rate holds.
- [Optimize with DSPy Using Scenarios as the Metric](https://langwatch.ai/docs/improve-your-agent/optimize-with-dspy.md): Scores every candidate on full multi-turn conversations with your agent, running the agent improvement loop automatically with DSPy algorithms.
- [Optimization Algorithms](https://langwatch.ai/docs/improve-your-agent/algorithms.md): What each optimization algorithm changes, what it needs, how many suite runs it costs, and when to use it.
- [Voice Agents](https://langwatch.ai/docs/agent-testing/voice-agents.md): Test a voice agent over real audio. A simulated caller talks to your agent, and a judge scores the call against your criteria.
- [Red Teaming](https://langwatch.ai/docs/agent-testing/red-teaming.md): RedTeamAgent replaces the user simulator with a multi-turn attacker, in the same scenario.run() loop and the same judge as your functional tests.
## Observability
- [Observability & Tracing](https://langwatch.ai/docs/observability/overview.md): Monitor, debug, and optimize your LLM applications with comprehensive observability and tracing capabilities
- [Quick Start](https://langwatch.ai/docs/integration/quick-start.md): Send your first traces to LangWatch in a few minutes, with a coding agent or by hand, in Python, TypeScript or any OpenTelemetry stack.
- [Concepts](https://langwatch.ai/docs/concepts.md): Explore core concepts of LLM tracing, observability, datasets, and evaluations in LangWatch to design reliable AI agent testing workflows.
### User Events
- [Overview](https://langwatch.ai/docs/user-events/overview.md): Track user interactions in LangWatch to analyze LLM usage patterns and power AI agent evaluation workflows.
- [Thumbs Up/Down](https://langwatch.ai/docs/user-events/thumbs-up-down.md): Track thumbs up/down user feedback in LangWatch to evaluate LLM quality and guide AI agent testing improvements.
- [Waited To Finish Events](https://langwatch.ai/docs/user-events/waited-to-finish.md): Track whether users leave before the LLM response completes to identify UX issues that affect downstream agent evaluations.
- [Selected Text Events](https://langwatch.ai/docs/user-events/selected-text.md): Track selected text events in LangWatch to understand user behavior and improve LLM performance across AI agent evaluations.
- [Custom Events](https://langwatch.ai/docs/user-events/custom.md): Track custom user events in your LLM application using LangWatch to support analytics, evaluations, and agent testing workflows.
- [Querying your data](https://langwatch.ai/docs/features/querying-your-data.md): LangWatch answers two query languages, analytics SQL and the trace filter. One endpoint describes both, and the CLI runs either without a browser.
- [Dashboards](https://langwatch.ai/docs/features/dashboards.md): Build dashboards of charts over your LangWatch data, or start one from the Agent health template for an at-a-glance view of your project.
- [Alerts and Automations](https://langwatch.ai/docs/features/automations.md): Configure Alerts and Automations in LangWatch to detect regressions, notify teams, and enforce automated guardrails for AI agent testing.
- [Topic clustering](https://langwatch.ai/docs/features/topic-clustering.md): Group your traces into topics and subtopics automatically, choose the model it uses, and run it on demand from Settings.
- [Exporting Analytics](https://langwatch.ai/docs/features/embedded-analytics.md): Export LangWatch analytics into your own dashboards to monitor LLM quality, agent testing metrics, and evaluation performance.
## Evaluations
- [Evaluations Overview](https://langwatch.ai/docs/evaluations/overview.md): Score your prompts and agents on a dataset before you ship, score production traffic as it arrives, and block unsafe responses before they reach the user.
### Experiments
- [Experiments Overview](https://langwatch.ai/docs/evaluations/experiments/overview.md): Run a dataset through your prompts and agents in the workbench, score every row with evaluators, and compare runs before you ship a change.
- [Experiments via SDK](https://langwatch.ai/docs/evaluations/experiments/sdk.md): Run experiments programmatically from notebooks or scripts to batch test your LLM applications.
#### Via UI
- [Answer Correctness: check answers against expected output](https://langwatch.ai/docs/evaluations/experiments/ui/answer-correctness.md): Score every answer against the expected output in your dataset with the Exact Match or the LLM Answer Match evaluator, and compare runs as you change the prompt.
- [LLM-as-a-Judge: score outputs without expected answers](https://langwatch.ai/docs/evaluations/experiments/ui/llm-as-a-judge.md): Score outputs against written criteria with an LLM-as-a-Judge evaluator when your dataset has inputs but no expected answers, and read the reason behind each verdict.
- [Pairwise Comparison: judge candidates head-to-head](https://langwatch.ai/docs/evaluations/experiments/ui/pairwise-compare.md): Use the Comparison evaluator to judge two or more candidate outputs against each other, with built-in position-bias mitigation, and rank three or more candidates on a Bradley-Terry leaderboard.
- [Running Experiments in CI/CD](https://langwatch.ai/docs/evaluations/experiments/ci-cd.md): Automate LLM quality gates by running experiments in your CI/CD pipelines.
- [Multimodal Evaluation: images, PDFs and vision models](https://langwatch.ai/docs/evaluations/experiments/multimodal-evaluation.md): Evaluate image generation, document parsing, and other multimodal AI pipelines with LLM-as-a-Judge vision models.
### Online Evaluation
- [Online Evaluation Overview](https://langwatch.ai/docs/evaluations/online-evaluation/overview.md): Score your production traffic as it arrives with monitors that run an evaluator on every matching trace or thread, and read the results on the trace, in the list and in analytics.
- [Online Evaluations: monitor your production traffic](https://langwatch.ai/docs/evaluations/online-evaluation/setup-monitors.md): Create an online evaluation that scores every incoming trace or thread with an evaluator, filter and sample the traffic it runs on, and read the results on the trace and in analytics.
### Instant Evals
- [Instant Evals](https://langwatch.ai/docs/features/instant-evals/overview.md): Ask up to ten questions of every conversation, trace or LLM call in your history and read the matches back in about a minute per 10,000 rows.
- [Instant Evals questions](https://langwatch.ai/docs/features/instant-evals/questions.md): The three kinds of question a run can ask, how to sharpen a yes or no with criteria and a threshold, and how to ask up to ten questions of the same text in one run.
- [Instant Evals targets and statements](https://langwatch.ai/docs/features/instant-evals/targets.md): What one judged row is, how the time window and the trace filter narrow the rows, the LangWatchQL statement each target writes for you, and how to write your own.
- [Reading Instant Eval results](https://langwatch.ai/docs/features/instant-evals/results.md): Page through a run's verdicts, read the text the judge saw beside each one, and query the judgments dataset with LangWatchQL to join verdicts to your traces.
- [Instant Evals limits, cost and billing](https://langwatch.ai/docs/features/instant-evals/limits-and-cost.md): How many rows and questions a run takes, how long it runs, what a run costs, how to read the cost before you start, the free budget, where the spend shows up, and where the judged text goes.
- [Run Instant Evals on your own decision model](https://langwatch.ai/docs/features/instant-evals/own-model.md): Run Instant Evals on a decision model in your own provider account, set up Azure, and read the rows your model cannot judge.
- [Instant Evals CLI reference](https://langwatch.ai/docs/features/instant-evals/cli.md): Every langwatch instant-eval subcommand and flag, what each prints, and the exit codes a script can read.
### Guardrails
- [Guardrails Overview](https://langwatch.ai/docs/evaluations/guardrails/overview.md): Block or modify harmful LLM responses in real-time to enforce safety and policy constraints.
- [Guardrails Code Integration](https://langwatch.ai/docs/evaluations/guardrails/code-integration.md): Add guardrails to your LLM application to block harmful content in real-time.
### Evaluators
- [Evaluators Overview](https://langwatch.ai/docs/evaluations/evaluators/overview.md): Understand evaluators - the scoring functions that assess your LLM outputs for quality, safety, and correctness.
- [Using Built-in Evaluators](https://langwatch.ai/docs/evaluations/evaluators/built-in-evaluators.md): Run LangWatch's library of evaluators directly from your code for experiments, online evaluation, and guardrails.
- [Saved Evaluators](https://langwatch.ai/docs/evaluations/evaluators/saved-evaluators.md): Create reusable evaluator configurations on the platform and use them across experiments, monitors, and guardrails.
- [Custom Scoring](https://langwatch.ai/docs/evaluations/evaluators/custom-scoring.md): Send evaluation scores from your own custom logic to LangWatch for tracking and analysis.
- [List of Evaluators](https://langwatch.ai/docs/evaluations/evaluators/list.md): Browse all available evaluators in LangWatch to find the right scoring method for your AI agent evaluation use case.
### Datasets
- [Datasets](https://langwatch.ai/docs/datasets/overview.md): Create a dataset by hand, from a CSV or JSON file, or from your traces, edit it in the grid, and run an experiment on it.
- [Datasets: programmatic access from SDKs, CLI and MCP](https://langwatch.ai/docs/datasets/programmatic-access.md): Manage datasets from the Python and TypeScript SDKs, the CLI and the MCP server for offline evaluations and automated workflows.
- [Automations: build a dataset from your traces](https://langwatch.ai/docs/datasets/automatically-from-traces.md): Create an automation that adds every new trace matching a filter to a dataset, mapped to the columns you choose.
- [Threads in datasets](https://langwatch.ai/docs/datasets/dataset-threads.md): Add whole conversations to a dataset, one row per thread, with every trace of the thread in one column.
- [Images and files in datasets](https://langwatch.ai/docs/datasets/dataset-images.md): Upload a picture or a document into a dataset cell, or enter a URL, and send it to a prompt or an agent in an experiment.
### Annotations
- [Annotations](https://langwatch.ai/docs/annotations/overview.md): Review traces by hand with a comment, scores and a corrected output, and read every review in the annotations inbox.
- [Annotation queues](https://langwatch.ai/docs/annotations/queues.md): Create a queue with reviewers and score types, send traces to it, work through the items one by one, correct a trace, and send the reviewed traces to a dataset.
- [Annotation scores](https://langwatch.ai/docs/annotations/scores.md): Define the score types reviewers fill in on every annotation, as a single choice or a set of checkboxes, each with an optional reason.
## Prompt Management
- [Overview](https://langwatch.ai/docs/prompt-management/overview.md): Organize, version, and optimize your AI prompts with LangWatch's prompt management system
- [Get Started](https://langwatch.ai/docs/prompt-management/getting-started.md): Create your first managed prompt in LangWatch, link it to traces, and use it in your application with built-in prompt versioning and analytics.
- [Prompts CLI](https://langwatch.ai/docs/prompt-management/cli.md): Use the LangWatch Prompts CLI to manage prompts as code with version control and support A/B testing for AI agent evaluations.
- [Prompt Playground](https://langwatch.ai/docs/prompt-management/prompt-playground.md): Use LangWatch’s Prompt Playground to edit, test, and iterate prompts with versioning, analytics, and AI agent test feedback loops.
### Features
- [Version Control](https://langwatch.ai/docs/prompt-management/features/essential/version-control.md): Manage version control for prompts in LangWatch to run evaluations, compare models, and improve agent performance.
- [Liquid Template Syntax](https://langwatch.ai/docs/prompts/template-syntax.md): Reference for the Liquid template syntax supported in LangWatch prompts: variables, conditionals, loops, filters, and more.
- [Data Model](https://langwatch.ai/docs/prompt-management/data-model.md): Learn the LangWatch prompt data model to manage versions, variants, and performance links for structured prompt versioning.
- [Runtime Parameters](https://langwatch.ai/docs/prompt-management/features/essential/runtime-