llms.txt
Compare with other sites' llms.txt files →# evalship
> evalship tests, monitors and optimizes every AI call your product makes. Get notified whenever your AI fails, and see what's wrong and how to fix it.
In production, your app sends evalship its AI calls, from its own code, through OpenRouter's Broadcast or as OpenTelemetry traces: evalship checks each one, groups what's wrong into issues and emails the moment one opens, with the calls behind it and a line for your coding agent to fix it. Your eval workflow keeps running in your own GitHub Actions with your own keys, and uploads a results file from pytest, Vitest, Jest, promptfoo, Langfuse, Braintrust or LangSmith (JUnit XML, promptfoo JSON, pytest-json-report JSON, Vitest or Jest JSON, an Inspect AI log or evalship JSON) that evalship reads when the run finishes. Teams that keep model keys off GitHub run the evals on their own machines instead, and their coding agent sends the results. Setup is a prompt for your coding agent: it reviews your evals, wires them into CI or your machine and opens a setup pull request. Before the agent finishes a change to a prompt, model or AI call, it runs the evals the change affects and sends evalship the diff with the results. Optimize has the agent try prompt and model changes on your machine, and evalship says which win beyond chance. evalship works next to eval platforms such as Langfuse, Braintrust and LangSmith rather than replacing them. Public repositories are free.
## Product
- [Detect errors in your AI calls](https://evalship.com/product/monitor): Every AI call your app makes in production, checked the moment it comes in, with an email when one fails, promises something or shares private data.
- [Review your evals](https://evalship.com/product/review): At setup, your coding agent reviews every scorer, judge and prompt in your LLM evals and finds their flaws, each with a fix.
- [Pull request reports](https://evalship.com/product/pull-requests): On every pull request, the LLM evals the change really broke and the likely cause, with flaky evals set aside and untested AI calls flagged.
- [Optimize prompts and models](https://evalship.com/product/optimize): Your coding agent tries prompt and model changes on your machine, and evalship says which win beyond chance, and by how much.
- [Eval runs and weekly report](https://evalship.com/product/test): Every eval run from GitHub and your engineers' machines with what to do next, the week's insights, and a Monday email listing what needs your team.
- [How it works](https://evalship.com/product/how-it-works): Support bot ELIZA from setup on: evalship finds the flaws in its evals, blocks a pull request that breaks a rule, then finds a better prompt.
- [Integrations](https://evalship.com/product/integrations): evalship reads results from pytest, Vitest, Jest, promptfoo, Langfuse, Braintrust and LangSmith, next to the eval platform you keep.
- [evalship for coding agents](https://evalship.com/product/agents): Before your coding agent finishes a change to a prompt or model, it runs the evals before and after, and evalship says if it got worse. No account.
## Docs
- [Quickstart](https://evalship.com/docs/quickstart.md): evalship works in two parts. Setup reviews your evals and shows what they miss, with a fix for each flaw.
- [Monitor](https://evalship.com/docs/monitor.md): evalship checks every AI call your app makes in production and emails you the moment one fails, promises something, shares private data or frustrates a user.
- [What your agent does](https://evalship.com/docs/agent.md): Setup is a prompt your coding agent follows in your repository.
- [The review](https://evalship.com/docs/audit.md): The review is your coding agent's look at your evals, written while it follows the setup prompt.
- [What evalship checks](https://evalship.com/docs/checks.md): evalship checks your evals twice. Once at setup, the review asks whether your evals can catch a bad change at all.
- [How the PR comment works](https://evalship.com/docs/comparison.md): When an eval workflow finishes, evalship stores its results as a run for that commit, compares the run with a baseline, and reports the difference in a comment…
- [Untested changes](https://evalship.com/docs/untested-changes.md): On every pull request, evalship finds the AI calls and prompt files the pull request adds or changes, and checks whether any eval exercises them.
- [Weekly report](https://evalship.com/docs/weekly-report.md): Every Monday at 8:00 UTC, evalship emails each account's weekly report to everyone who reads its reports: the people the account invited, and GitHub users with…
- [Optimize](https://evalship.com/docs/optimize.md): Your coding agent finds changes your evals can prove better, runs them on your machine, and evalship says which win beyond chance.
- [Badges](https://evalship.com/docs/badges.md): evalship serves README badges with live eval results from your default branch.
- [Integrations](https://evalship.com/docs/integrations.md): evalship reads the results your evals already write, from pytest, Vitest, Jest, promptfoo, Langfuse, Braintrust or LangSmith.
- [Connecting CI](https://evalship.com/docs/ci.md): evalship reads eval results from GitHub Actions workflow artifacts.
- [Result formats](https://evalship.com/docs/formats.md): evalship reads six result formats. Each result becomes the same eval record: a suite, a key, pass or fail, and optionally a score, threshold, duration, cost,…
- [Configuration](https://evalship.com/docs/configuration.md): evalship works without configuration. To change its behavior, add .evalship.yml to the root of your repository.
- [Pricing and billing](https://evalship.com/docs/pricing.md): evalship's plans and prices, the launch offer, how users are counted, additional usage, trials and billing.
## About evalship
- [FAQ](https://evalship.com/faq.md): Common questions about evalship: running evals, frameworks, CI, coding agents, privacy, gating, and using it next to Langfuse, Braintrust or LangSmith.
- [Security](https://evalship.com/security.md): What evalship keeps of your customers' data and source code, what goes to a model, how it handles your secrets, and how it reaches your repositories.
- [Data and privacy](https://evalship.com/privacy.md): This page lists what evalship stores, what it reads from GitHub, what it sends to a model, and who processes it.
- [Data processing agreement](https://evalship.com/dpa.md): This agreement is part of the terms between you and Garm Tech B.V. ("we", "us"), and covers the personal data evalship processes on your behalf.
- [Terms of service](https://evalship.com/terms.md): The terms for using evalship.com and the evalship GitHub App.
## Articles
- [Recorded LLM replies in Rails tests pass any prompt change](https://evalship.com/articles/recorded-llm-replies-in-rails-tests): In 16 of 19 Ruby apps that replay their LLM calls in tests, the replay ignores the request, so any prompt or model change passes.
- [LLM evals in Ruby, Go and Rust, without Python or Node](https://evalship.com/articles/llm-evals-in-ruby-go-rust): Run LLM evals in a Ruby, Go or Rust app without adding Node or Python to CI: call your real code, check in code first, run each test case 3 times.
- [LLM evals that can't fail: flaws in 10 open-source eval suites](https://evalship.com/articles/llm-eval-flaws): Every one of 10 open-source LLM eval suites had a test case that couldn't fail. The flaws we found most, with examples and how to check yours.
- [LLM evals on pull requests: how the tools compare (2026)](https://evalship.com/articles/llm-evals-on-pull-requests): promptfoo, Braintrust, Langfuse, LangSmith, DeepEval and Raindrop on pull requests: what each compares with main, and what fails the check.
- [LLM as a judge: five mistakes in real eval suites](https://evalship.com/articles/llm-as-a-judge-mistakes): In 6 of 9 open-source eval suites, the LLM judge had a flaw, most often grading its own model's answers. Five mistakes, and how to fix each.
- [Run your Claude evals in CI on your Claude plan](https://evalship.com/articles/claude-evals-on-your-plan): Claude Max, Ultra and Team plans include $100 to $500 a month of API credit. Pay for your evals in CI with it, or with your subscription.
- [Langfuse experiments on every pull request](https://evalship.com/articles/langfuse-experiments-in-ci): Run Langfuse experiments on every pull request and see which test cases the change broke, compared with your default branch.
- [Braintrust evals on every pull request](https://evalship.com/articles/braintrust-evals-in-ci): Run Braintrust evals on every pull request and see which test cases the change broke, not only how the averages moved.
- [LangSmith evaluations on every pull request](https://evalship.com/articles/langsmith-evals-in-ci): Run LangSmith evaluations on every pull request, from pytest, Vitest, Jest or evaluate(), and see which examples the change broke.
- [LLM evals with pytest in GitHub Actions](https://evalship.com/articles/llm-evals-with-pytest): Run LLM evals with pytest in GitHub Actions: scores in the JUnit XML, test names that last, and a build that's red only when something broke.
- [LLM evals with Vitest in GitHub Actions](https://evalship.com/articles/llm-evals-with-vitest): Run LLM evals with Vitest in GitHub Actions: JUnit XML with the console kept, scores recorded with annotate, and why retry hides flaky evals.
- [LLM evals with Jest in GitHub Actions](https://evalship.com/articles/llm-evals-with-jest): Run LLM evals with Jest in GitHub Actions: jest-junit settings that keep names stable, scores in the XML, and why retries hide flaky evals.
- [Replacing OpenAI Evals: promptfoo in CI without flaky reds](https://evalship.com/articles/openai-evals-to-promptfoo): OpenAI Evals shuts down on November 30. Run the evals in promptfoo on every pull request, without a gate that goes red when nothing broke.
- [How noisy are LLM evals in CI?](https://evalship.com/articles/eval-noise): How often passing LLM evals fail by chance in CI, measured on public repositories, and how big a break your suite can see through it.
## AI is never wrong: the AI incidents log
- [AI is never wrong](https://aineverwrong.com/): times AI got it wrong, what each cost, with sources
- [UK prosecutor cites a case that doesn't exist after an AI search](https://aineverwrong.com/cps-prosecutor-cites-nonexistent-case-after-ai-search): Internal review
- [Lawsuit alleges ChatGPT talked a woman off her seizure medication](https://aineverwrong.com/lawsuit-alleges-chatgpt-coached-woman-off-seizure-drug): Wrongful death suit
- [Lawyer fined $5,000 for an undisclosed AI-written brief](https://aineverwrong.com/la-film-school-suit-lawyer-fined-ai-brief): $5,000 fine
- [South African labor arbitrator's ruling voided over a fake AI case](https://aineverwrong.com/south-africa-ccma-ruling-voided-over-fake-ai-case): Ruling declared nullity
- [Just Eat pulls AI menu photos that put sausages in a glass of cola](https://aineverwrong.com/just-eat-pauses-ai-menu-photos): AI trial paused
- [Meta pauses Muse chat sharing after 6,800 shared chats surface in DNS](https://aineverwrong.com/meta-muse-shared-chats-exposed-dns): 6,800 chats exposed
- [California courts cancel AI translation rollout after errors surface](https://aineverwrong.com/california-courts-cancel-ai-translation-rollout): Rollout canceled
- [Deloitte's WA university merger report cites sources that don't exist](https://aineverwrong.com/deloitte-wa-university-merger-report-ai-references): Refund sought
- [OpenAI's test agents broke into Australian government systems](https://aineverwrong.com/openai-agents-breach-australian-government-sites): Medicare portal breached
- [Google's Gemini broke into three companies during a security test](https://aineverwrong.com/google-gemini-breaks-into-three-companies-during-test): 3 companies breached
- [Oklahoma judge admits his custody order cited fake cases from ChatGPT](https://aineverwrong.com/oklahoma-judge-custody-order-chatgpt-fake-cases): State investigation
- [UK tribunal strikes off a lawyer over an AI-drafted defense](https://aineverwrong.com/uk-tribunal-strikes-off-lawyer-ai-drafted-defense): Struck off
- [India's top court voids a ₹425 crore penalty built on fake AI cases](https://aineverwrong.com/india-supreme-court-voids-customs-penalty-fake-ai-cases): ₹425 crore fine voided
- [NHS watchdog finds AI scribes put wrong diagnoses into records](https://aineverwrong.com/nhs-watchdog-finds-ai-scribes-put-wrong-diagnoses-in-records): Wrong diagnosis recorded
- [Claude deletes a developer's 700 GB home folder in a safety test](https://aineverwrong.com/claude-deletes-developers-700gb-home-folder-in-a-safety-test): 700 GB deleted
- [Georgia judge doubles a lawyer's fine for lying about AI use](https://aineverwrong.com/georgia-judge-doubles-fine-for-lawyer-who-denied-using-ai): $8,000 fine
- [Florida appeals court finds 46 AI hallucinations, sanctions lawyer](https://aineverwrong.com/florida-appeals-court-finds-46-ai-hallucinations-sanctions-lawyer): $1,500 fine plus fees
- [Indian court voids a tax officer's GST order built on fake AI cases](https://aineverwrong.com/gujarat-high-court-voids-gst-order-built-on-fake-ai-cases): GST order set aside
- [AI scribe falsely records that an Australian patient took mushrooms](https://aineverwrong.com/ai-scribe-falsely-records-australian-patient-took-mushrooms): False drug use on record
- [AI agent cancels another gym member's booking without being asked](https://aineverwrong.com/openclaw-agent-cancels-another-gym-members-booking): Member lost class spot
- [Connecticut's top court sanctions lawyer after ChatGPT altered cites](https://aineverwrong.com/connecticut-supreme-court-sanctions-lawyer-after-chatgpt-altered-cites): $2,000 and 6 CLE hours
- [Illinois court fines a lawyer $15,000 for ChatGPT's false citations](https://aineverwrong.com/illinois-court-fines-lawyer-15000-for-chatgpt-false-citations): $15,000 fine
- [Shared Claude chats with medical and business data show up on Google](https://aineverwrong.com/shared-claude-chats-with-sensitive-data-show-up-on-google): Chats exposed in search
- [State Department shows an AI-made Africa map with countries misplaced](https://aineverwrong.com/state-department-shows-ai-made-africa-map-with-countries-misplaced): Countries mislabeled
- [Lawyer fined over ChatGPT and Claude citations; expert report tossed](https://aineverwrong.com/ledoux-lawyer-fined-over-chatgpt-and-claude-citations): $3,000 sanction
- [OpenAI's test models escape their sandbox and break into Hugging Face](https://aineverwrong.com/openai-test-models-escape-sandbox-and-break-into-hugging-face): Hugging Face breached
- [OpenAI admits its GPT-5.6 Sol coding agent sometimes deletes files](https://aineverwrong.com/openai-admits-gpt-5-6-sol-sometimes-deletes-users-files): Files and database wiped
- [California appeals court fines a lawyer for AI-added fake quotes](https://aineverwrong.com/california-appeals-court-fines-lawyer-for-ai-added-fake-quotes): $1,500 sanction
- [Marie Claire Australia deletes an article with AI notes left in](https://aineverwrong.com/marie-claire-australia-deletes-article-with-ai-notes-left-in): Article deleted
- [Appeals court refers a lawyer who cited fake AI cases twice](https://aineverwrong.com/eleventh-circuit-refers-lawyer-who-cited-fake-ai-cases-twice): Referred for discipline
- [India's Supreme Court voids tribunal rulings that cited fake AI cases](https://aineverwrong.com/india-supreme-court-voids-tribunal-rulings-built-on-fake-ai-cases): Two rulings set aside
- [KPMG pulls an agentic AI report over false case studies](https://aineverwrong.com/kpmg-pulls-agentic-ai-report-false-case-studies): Report pulled
- [BMW Toronto honors a buyback price its chatbot offered by mistake](https://aineverwrong.com/bmw-toronto-chatbot-buyback-offer): CA$7,000 lost
- [Judge removes all four lawyers in a Mississippi case over fake cases](https://aineverwrong.com/aberdeen-mississippi-lawyers-removed-ai-citations): $8,000 fines, 4 removed
- [Berlingske's AI tool invents experts and quotes in a news story](https://aineverwrong.com/berlingske-ai-invented-experts): Journalist suspended
- [Doubao misquotes a flight fee, then pledges 600 yuan it never pays](https://aineverwrong.com/doubao-flight-refund-compensation-pledge): 600 yuan unpaid
- [A book on AI and truth prints fake quotes made by AI tools](https://aineverwrong.com/future-of-truth-book-ai-fake-quotes): Fake quotes in print
- [AI name reader stumbles at a community college graduation](https://aineverwrong.com/glendale-graduation-ai-name-reader-fails): Public apology
- [Lawyers apologize for Claude-made quotes in a federal layoffs case](https://aineverwrong.com/binnall-claude-phantom-quotes-layoffs-case): Sworn apologies filed
- [EY Canada pulls a cyber report with AI-hallucinated references](https://aineverwrong.com/ey-canada-pulls-report-hallucinated-references): Report pulled
- [A German court holds a clinic to its chatbot's false claims](https://aineverwrong.com/clinic-chatbot-false-credentials): Court order
- [Judge voids DOGE's humanities grant cuts made with ChatGPT's help](https://aineverwrong.com/neh-grant-cuts-chatgpt-ruled-unlawful): Cuts ruled unlawful
- [Georgia prosecutor's AI citations void a ruling in a murder case](https://aineverwrong.com/georgia-prosecutor-ai-citations-murder-case-order-vacated): Trial order vacated
- [South Africa suspends two officials over AI-invented policy sources](https://aineverwrong.com/home-affairs-suspends-officials-ai-references): 2 officials suspended
- [South Africa withdraws its draft AI policy over fake AI citations](https://aineverwrong.com/south-africa-withdraws-ai-policy-fake-citations): Draft policy withdrawn
- [Cursor agent deletes a startup's production database and backups](https://aineverwrong.com/pocketos-cursor-agent-deletes-database): Database wiped
- [Sullivan & Cromwell apologizes for AI-made citations in a filing](https://aineverwrong.com/sullivan-cromwell-apologizes-ai-errors-prince-group): Apology to the judge
- [New York Times prints an AI-made quote from Canada's Poilievre](https://aineverwrong.com/nyt-poilievre-ai-quote): Correction after 17 days
- [KBS's AI subtitles turn a NASA launch term into a slur](https://aineverwrong.com/kbs-ai-subtitles-artemis-slur): Public apology
- [Mediahuis suspends a senior journalist over AI-invented quotes](https://aineverwrong.com/mediahuis-fellow-ai-invented-quotes): Journalist suspended
- [Meta's internal AI agent's bad advice exposes data to staff](https://aineverwrong.com/meta-internal-ai-agent-exposes-data-to-staff): Sev 1 data exposure
- [Lawsuit says Google's Gemini fed a man's delusions before his death](https://aineverwrong.com/gemini-wrongful-death-lawsuit-delusions): Wrongful-death suit
- [UK prosecutors cite two nonexistent cases in extradition appeals](https://aineverwrong.com/cps-cites-nonexistent-cases-extradition): Two fake cases cited
- [Claude Code's Terraform run wipes DataTalks.Club's production setup](https://aineverwrong.com/claude-code-terraform-destroy-datatalks-club): Platform down a day
- [OpenClaw agent deletes a Meta director's email despite stop orders](https://aineverwrong.com/openclaw-agent-deletes-meta-directors-email): Emails deleted
- [Amazon's Kiro coding agent is tied to a 13-hour AWS outage](https://aineverwrong.com/kiro-agent-tied-to-aws-outage): 13-hour outage
- [Microsoft 365 Copilot summarizes emails labeled confidential](https://aineverwrong.com/microsoft-copilot-summarizes-confidential-emails): Confidentiality bypassed
- [Ars Technica retracts a story with quotes an AI tool made up](https://aineverwrong.com/ars-technica-ai-fabricated-quotes): Article retracted
- [Judge enters default judgment over a lawyer's repeated AI citations](https://aineverwrong.com/flycatcher-default-judgment-ai-citations): Client lost by default
- [Tour site's AI article sends tourists to hot springs that don't exist](https:/