llms.txt
Compare with other sites' llms.txt files →# Confident AI
> Confident AI is an AI quality platform built for enterprise teams to standardize evals and observability across the organization. Platform teams set one consistent standard, product teams measure how their AI apps perform before launch and monitor them live with online evals and signals on real traffic. That standard extends to security through native red teaming, and is enforced automatically by AI governance, an organization-wide gate that blocks anything failing its evals or red-team checks before it ships, and holds live applications to the same bar in production. Confident AI also builds and maintains DeepEval, the open-source LLM evaluation framework, and DeepTeam, the open-source LLM red teaming framework; the platform's metrics are DeepEval's metrics.
## Products
- [LLM Evaluation](https://www.confident-ai.com/products/llm-evaluation): 50+ research-backed metrics, multi-turn simulation, human metric alignment, and CI/CD regression testing.
- [LLM Observability](https://www.confident-ai.com/products/llm-observability): Tracing with evaluation on every trace, span, and thread — drift detection, alerting, and dataset curation from production.
- [AI Governance](https://www.confident-ai.com/products/ai-governance): Audit trails, RBAC, and compliance evidence for enterprise AI programs.
- [AI Red Teaming](https://www.confident-ai.com/products/ai-red-teaming): Adversarial safety testing based on OWASP Top 10 for LLMs and NIST AI RMF.
- [Pricing](https://www.confident-ai.com/pricing)
## Migrate to Confident AI
- [Stitching tools together around Langfuse? Migrate to Confident AI.](https://www.confident-ai.com/migrate/langfuse-alternative): Langfuse shows you what happened. Confident AI scores every trace, curates datasets from production, catches drift and simulates conversations. Switch in three steps and keep your OpenTelemetry spans.
- [LangSmith traces your chains. Confident AI grades every one of them.](https://www.confident-ai.com/migrate/langsmith-alternative): LangSmith traces your chains. Confident AI grades every trace, curates datasets from production, catches drift and simulates conversations. Switch in three steps and keep your LangChain spans.
- [Arize watches your traces. Confident AI improves your app.](https://www.confident-ai.com/migrate/arize-phoenix-alternative): Arize Phoenix and Arize AX watch your traces. Confident AI scores every one of them, curates datasets from production, catches drift and simulates conversations. Switch in three steps and keep your OpenInference spans.
- [Braintrust scores the traces you sample. Confident AI scores them all.](https://www.confident-ai.com/migrate/braintrust-alternative): Braintrust scores the traces you sample. Confident AI scores them all, curates datasets from production, catches drift and simulates conversations. Switch in three steps and keep your OpenTelemetry spans.
## Documentation
- [README.MD](https://www.confident-ai.com/docs)
- [Reference](https://www.confident-ai.com/docs/reference)
- [Integrations & OTEL](https://www.confident-ai.com/docs/integrations)
- [Skills & Plugins](https://www.confident-ai.com/docs/coding-agents/skills/confident-client)
- [Platform Settings](https://www.confident-ai.com/docs/settings)
- [Self-Hosting](https://www.confident-ai.com/docs/self-hosting)
- [Guides](https://www.confident-ai.com/docs/guides)
- [Changelog](https://www.confident-ai.com/docs/changelog)
- [Full documentation index](https://www.confident-ai.com/docs/llms.txt): Every documentation page, grouped by section.
- [Full documentation text](https://www.confident-ai.com/llms-full.txt): The entire documentation as a single markdown file.
Any documentation page is served as markdown by appending .md to its URL.
## Agents
- [Agent authentication](https://www.confident-ai.com/auth.md): How agents authenticate: API keys for the REST API, OAuth with dynamic registration for the MCP server.
- [Agent resources](https://www.confident-ai.com/.well-known/ai-catalog.json): Machine-readable catalog of the MCP server, OpenAPI spec, agent skills, and API catalog.
## Guides
- [AI Agent Testing: How to Test Tool Calling, Regressions, and Failure Handling](https://www.confident-ai.com/knowledge-base/guides/ai-agent-testing.md): A how-to guide for testing AI agents before they ship: writing test cases for tool selection, arguments, and trajectories, simulating tool failures and preventing loops, the three-tier CI/CD strategy, and how product managers should read agent test results when making the ship decision.
- [AI Production Issue Detection: A Failure Taxonomy and Detection Framework](https://www.confident-ai.com/knowledge-base/guides/ai-production-issue-detection-framework.md): A vendor-neutral taxonomy of the six ways LLM applications fail in production — tool misuse, context loss, goal drift, silent quality degradation, retrieval failure, and policy violations — with a diagnostic table and a five-layer detection framework for catching each one.
- [AI Governance and Audit Trails for Enterprise LLM Observability](https://www.confident-ai.com/knowledge-base/guides/enterprise-ai-governance-audit-trails.md): How enterprises turn LLM observability into compliance evidence: immutable audit trails, RBAC and SSO patterns, data residency, and a mapping from SOC 2, HIPAA, GDPR, the EU AI Act, and NIST AI RMF to the observability records that satisfy each one.
- [LLM Error Analysis: How to Trace Failures Back to Root Cause](https://www.confident-ai.com/knowledge-base/guides/llm-error-analysis-methodology.md): The six-step methodology for going from 'this test case scored poorly' to a shipped fix: reading judge reasoning, categorizing failures, localizing root causes with traces, fixing the right layer, and converting every confirmed failure into permanent regression coverage.
- [LLM Evaluation for Healthcare: Mapping Metrics to Clinical Use Cases](https://www.confident-ai.com/knowledge-base/guides/llm-evaluation-for-healthcare.md): How to evaluate LLM applications in clinical settings: a mapping from five healthcare use cases — intake, documentation, extraction, clinician Q&A, triage — to the evaluation metrics that catch their failure modes, and the clinician-validation loop that makes automated scores trustworthy.
- [LLM Evaluation for Startups: A Quickstart Guide](https://www.confident-ai.com/knowledge-base/guides/llm-evaluation-for-startups-quickstart.md): The one-afternoon path from zero evals to a CI-gated evaluation loop: a comparison of the five ways startups approach evaluation, the four setup steps from first test case to release gate, and the production loop that grows coverage without headcount.
- [LLM Regression Testing: How to Gate Every Change with CI/CD Evals](https://www.confident-ai.com/knowledge-base/guides/llm-regression-testing.md): A step-by-step guide to LLM regression testing: building a golden dataset that reflects real usage, choosing metrics per use case, setting pass/fail thresholds that block bad deploys, wiring evals into CI/CD, and extending the suite to AI agents — plus the four mistakes that quietly undermine all of it.
- [RAG Evaluation: Metrics, CI/CD, and Production Monitoring](https://www.confident-ai.com/knowledge-base/guides/rag-evaluation.md): The definitive guide to evaluating RAG pipelines: starting with end-to-end outcomes, measuring retrieval and generation separately, calibrating metrics and thresholds, building datasets, gating changes in CI/CD, and catching production degradation.
- [What Is LLM Tracing? Traces, Spans, and Threads Explained](https://www.confident-ai.com/knowledge-base/guides/what-is-llm-tracing.md): A concept-first guide to LLM tracing: what traces, spans, and threads are, how OpenTelemetry's GenAI semantic conventions standardize them, the four instrumentation approaches compared, and what separates tracing an AI application from tracing a microservice.
## Playbook
- [Playbook Overview](https://www.confident-ai.com/knowledge-base/playbook/playbook-overview.md): What this handbook is for, who should read it, and what you will walk away with.
- [What Makes a Good Eval](https://www.confident-ai.com/knowledge-base/playbook/what-makes-a-good-eval.md): Good evaluation is automated metrics locked to human judgment — neither side alone is enough.
- [What Should I Measure?](https://www.confident-ai.com/knowledge-base/playbook/what-should-i-measure.md): Before you pick a metric, figure out which business outcome it needs to predict — everything else follows from that.
- [When Should I Start Tracing?](https://www.confident-ai.com/knowledge-base/playbook/when-should-i-start-tracing.md): Set up tracing before you need it — everything else (datasets, annotations, evals) depends on having the data flowing first.
- [User-Facing vs. Non-User-Facing Apps](https://www.confident-ai.com/knowledge-base/playbook/user-facing-vs-non-user-facing-apps.md): The same LLM stack does not imply the same definition of quality — user-facing and internal apps optimize different dimensions.
- [Single-Turn vs. Multi-Turn Use Cases](https://www.confident-ai.com/knowledge-base/playbook/single-turn-vs-multi-turn-evaluation.md): Multi-turn failures show up across turns — not in any single response — which is why a separate evaluation strategy matters.
- [Dev, Staging, and Production](https://www.confident-ai.com/knowledge-base/playbook/dev-staging-and-production.md): Evaluation is different work in each environment — dev is for iteration, staging is for regression, production is for monitoring.
- [Setting Up Trigger Moments (Online Evals)](https://www.confident-ai.com/knowledge-base/playbook/setting-up-trigger-moments-online-evals.md): How to choose where to run online evaluations in your LLM app, and in what order.
- [Setting Up AI Agent Observability](https://www.confident-ai.com/knowledge-base/playbook/ai-agent-observability.md): What agent observability actually captures, why traditional application monitoring and generic LLM observability miss agent failures, and how to turn traces into a quality loop instead of a log dump.
- [Evaluating AI Agents](https://www.confident-ai.com/knowledge-base/playbook/how-to-evaluate-ai-agents.md): What to measure at each layer of an agent, how to build a test harness that survives contact with production, and how to gate releases on agent quality without slowing the team down.
- [Setting Up Multi-Turn Agent Observability](https://www.confident-ai.com/knowledge-base/playbook/multi-turn-observability.md): The unit of quality for a multi-turn agent is not the request — it is the thread. How to instrument production agents and chatbots so the conversation is a first-class object in the trace store, not something you reconstruct from a session ID after a complaint comes in.
- [Evaluating Multi-Turn Chatbots](https://www.confident-ai.com/knowledge-base/playbook/evaluating-multi-turn-chatbots.md): A chatbot can score green on every individual reply and still fail the user's actual request twelve turns later. How to evaluate both trace-level turns and conversation-level outcomes with scenario-based simulation and CI gates.
## Comparisons
- [7 Best LLM Inference Providers in 2026](https://www.confident-ai.com/knowledge-base/compare/best-llm-inference-providers-2026.md): Compare the seven best LLM inference providers in 2026, including Morph, Together AI, and Groq, on dedicated capacity, serverless pricing, and operations.
- [8 Best On-Call Incident Management Software Tools in 2026](https://www.confident-ai.com/knowledge-base/compare/best-on-call-incident-management-software-2026.md): Compare the eight best on-call incident management software tools in 2026, including OnPage, PagerDuty, incident.io, xMatters, AlertOps, Better Stack, SolarWinds Incident Response, and FireHydrant.
- [7 best voice AI evaluation and observability tools in 2026](https://www.confident-ai.com/knowledge-base/compare/best-voice-ai-evaluation-and-observability-tools-2026.md): The 7 best voice AI evaluation and observability tools in 2026, ranked on audio metrics, caller simulation, stack integrations and live-call monitoring.
- [8 best evaluation tools for conversational AI in 2026](https://www.confident-ai.com/knowledge-base/compare/best-evaluation-tools-for-conversational-ai.md): The 8 best evaluation tools for conversational AI in 2026, ranked by how well they test multi-turn conversations — simulating realistic dialogues, scoring coherence and context retention across turns, and catching the failures that only appear after the third or fourth exchange.
- [10 best QA tools for AI agents in 2026](https://www.confident-ai.com/knowledge-base/compare/best-qa-tools-for-ai-agents.md): The 10 best QA tools for AI agents in 2026, ranked by how precisely they localize agent failures — span-level evaluation on tool calls and reasoning steps, fresh simulated benchmarks, CI/CD regression gates, and QA workflows that don't route every test cycle through engineering.
- [Top 5 AI testing tools in 2026](https://www.confident-ai.com/knowledge-base/compare/top-5-ai-testing-tools.md): The top 5 AI testing tools in 2026, ranked by how completely they test the application that actually ships — reliable eval metrics, whole-app endpoint testing, multi-turn simulation, regression tracking across versions, and online evals on live production traffic.
- [Top 5 LLM testing tools in 2026](https://www.confident-ai.com/knowledge-base/compare/top-5-llm-testing-tools.md): The top 5 LLM testing tools in 2026, ranked by benchmark discipline — reliable evaluation metrics, versioned dataset curation, endpoint-based whole-app testing, multi-turn simulation, regression tracking across versions, and online evals on live production traffic.
- [8 best AI production issue detection and alerting tools in 2026](https://www.confident-ai.com/knowledge-base/compare/best-ai-production-issue-detection-tools.md): The 8 best tools for AI production issue detection and alerting in 2026, ranked by how automatically they surface failing runs, hallucinations, wrong tool calls, frustrated users, and new topics — and how reliably they page your team when quality drops, not just when latency spikes.
- [8 best LLM observability tools for enterprise in 2026](https://www.confident-ai.com/knowledge-base/compare/best-enterprise-llm-observability-tools.md): The 8 best enterprise LLM observability tools in 2026, ranked by how well they combine evaluation-first quality monitoring with the security, compliance, access control, and cross-team governance a large organization needs to run AI in production at scale.
- [8 best LLM observability tools for startups in 2026](https://www.confident-ai.com/knowledge-base/compare/best-observability-tools-for-startups.md): The 8 best LLM observability tools for startups in 2026, ranked by how little setup they need and how much they automate afterward — one-line instrumentation, then a constant stream of surfaced failures, drift, and quality drops a small team can keep improving without manually digging through traces.
- [9 Best AI Evaluation Tools for CI/CD Pipelines in 2026](https://www.confident-ai.com/knowledge-base/compare/best-ai-evaluation-tools-for-ci-cd.md): Compare nine AI evaluation tools for CI/CD by quality coverage, whole-app execution, governed release policy, trusted metrics, versioned benchmarks, human alignment, and audit-ready evidence.
- [10 Best AI Quality Platforms for Human Annotators and Subject Matter Experts (2026)](https://www.confident-ai.com/knowledge-base/compare/best-ai-quality-platforms-for-human-annotation.md): Compare ten evaluation-first AI quality platforms for curating datasets, labeling outputs, aligning automated metrics, reviewing production failures, and turning expert corrections into regression coverage.
- [7 Best RAG Evaluation Tools for Retrieval and Generation Quality (2026)](https://www.confident-ai.com/knowledge-base/compare/best-rag-evaluation-tools.md): Compare seven RAG evaluation tools by how clearly they find retrieval and answer failures, test application changes, involve reviewers, and monitor production quality.
- [7 Best Web Search APIs for Grounding LLMs in 2026](https://www.confident-ai.com/knowledge-base/compare/best-web-search-apis-grounding-llms-reducing-hallucinations-2026.md): Compare the seven best web search APIs for grounding LLMs in 2026, including Firecrawl, Brave Search, Exa, Tavily, Parallel, Google Search grounding, and SerpApi.
- [7 Best AI Evaluation Tools for Enterprises in 2026](https://www.confident-ai.com/knowledge-base/compare/best-ai-evaluation-tools-for-enterprises-2026.md): Compare the 7 best AI evaluation tools for enterprises in 2026. We rank platforms by their ability to standardize evals and observability across the org, enforce one quality standard through automatic governance, run native red teaming, and meet enterprise security, compliance, and deployment requirements.
- [Top 8 Platforms for Pre-Deployment AI Testing in 2026](https://www.confident-ai.com/knowledge-base/compare/top-8-platforms-for-pre-deployment-ai-testing-2026.md): Compare the top 8 platforms for pre-deployment AI testing in 2026. We rank tools by evaluation depth, testing the app as deployed, multi-turn simulation, CI/CD regression gates, automatic release blocking, and security testing.
- [9 Best LLM Evaluation Tools for Product Managers in 2026](https://www.confident-ai.com/knowledge-base/compare/best-llm-evaluation-tools-for-product-managers-2026.md): Compare the 9 best LLM evaluation tools for product managers in 2026. We rank platforms by no-code accessibility, custom metrics and alignment, prompt and model experiments, production-to-dataset workflows, monitoring with dashboards and signals, and cross-functional pricing.
- [6 Best LLM Evaluation Tools for Startups in 2026](https://www.confident-ai.com/knowledge-base/compare/best-llm-evaluation-tools-for-startups-2026.md): Compare the 6 best LLM evaluation tools for startups in 2026. We rank platforms by automation and setup speed, dataset generation and curation, production-trace workflows, metric recommendations, CI/CD and scheduled evals, and startup-friendly pricing.
- [Top 6 Human-in-the-Loop Tools for AI Agent Evaluation (2026, Tested and Reviewed)](https://www.confident-ai.com/knowledge-base/compare/best-human-in-the-loop-ai-agent-evaluation-tools-2026.md): AI agents fail across tool calls, retrieval, and handoffs, and automated metrics miss a lot of it. We reviewed the six human-in-the-loop tools that get SMEs and QA into AI agent evaluation and turn their judgment into aligned metrics, new metrics, and regression datasets.
- [Top 8 CI/CD Tools for AI Applications in 2026](https://www.confident-ai.com/knowledge-base/compare/best-ci-cd-tools-ai-applications-2026.md): The eight best CI/CD tools for AI applications in 2026, ranked for LLM regression testing, release gates, CI/CD reports, industry-grade metrics, benchmark curation, metric alignment, AI failure insights, and advanced analytics.
- [Best 7 Tools for Testing LLM Apps Before Production in 2026](https://www.confident-ai.com/knowledge-base/compare/best-tools-testing-llm-apps-before-production-2026.md): The best tools for pre-production LLM app testing, ranked by how well they test whole-app behavior, use reliable metrics, curate benchmarks, catch regressions, simulate user journeys, and support human-in-the-loop review before production.
- [Top 8 AI Agent Observability Platforms for 2026](https://www.confident-ai.com/knowledge-base/compare/best-ai-agent-observability-tools-2026.md): Tracing has commoditized. The eight AI agent observability platforms that matter in 2026 are the ones that score what they capture, surface failing runs without manual querying, and turn production traces into the next test cycle.
- [5 Best AI Red Teaming Tools to Find AI Security Vulnerabilities in 2026](https://www.confident-ai.com/knowledge-base/compare/best-ai-red-teaming-tools-2026.md): A