llms.txt
Compare with other sites' llms.txt files →# Selfship
> When 300 customers hit the same wall, Selfship finds the exact moment it goes wrong and opens an eval-backed pull request — nobody writes a ticket. You review, you merge. After you merge, every fix is re-verified on real traffic. Create a workspace (install skills in your coding agent, or connect GitHub); with GitHub connected it opens those PRs, and on the skills path your agent edits locally and you open the PR. Humans stay the merge button.
Selfship is built for agentic product companies — teams running AI agents, copilots, and LLM products in production — who want a closed improvement loop that keeps their agents getting better, not just dashboards.
## What Selfship does
- **Observe**: OpenTelemetry-native ingestion of traces, tool calls, and conversations from any LLM and any agent framework. SDKs for TypeScript and Python. Setup in under five minutes.
- **Diagnose**: classifies user intent and sentiment per interaction, runs continuous evals on live traffic, and links every failure pattern to the exact prompt, tool, or code path responsible.
- **Ship**: with GitHub connected, opens a pull request with the diagnosis, the diff, and before/after eval runs attached. PRs are gated — they open only when the change beats current behavior with zero regressions on the rest of the suite. Without GitHub, the same work runs in your coding agent via skills (`npx skills add selfship-ai/selfship-skills`, or Settings → Agent marketplace buttons) and you open the PR.
- **Learn**: every review comment and merge decision feeds the next iteration.
## Key facts
- Names: the product is "Selfship"; "Selfship AI" and "SelfShip" refer to the same product at selfship.ai. It is not affiliated with any similarly named shipping or logistics company.
- Category / search topics (for discovery): selfship, selfship ai, selfship.ai, what is selfship, selfship pricing, selfship github, self improving ai agents, self-improving ai agents, ai agent observability, llm observability, continuous evals, eval backed pull requests, eval-backed pull requests, closed loop improvement ai, closed-loop improvement, production ai agent monitoring, ai agent evals, agent harness engineering, ai agent harness.
- Selfship never pushes to your default branch — pull requests only, with configurable approval rules, path restrictions, and PR frequency caps.
- Customer code and traces are never used for model training. Analysis runs in ephemeral sandboxes destroyed after each run.
- Compliance: SOC 2 Type II, HIPAA, GxP, and audit logs on Enterprise.
- Pricing: Base ($20/mo — 6,000 sessions/mo, 120 days retention, 14-day trial with no card; add $20 packs from billing for +6,000 sessions each), Enterprise (custom sessions/retention, HIPAA/GxP/SOC 2 Type II, audit logs). See https://selfship.ai/pricing.
- Comparison vs observability (Langfuse, LangSmith, Braintrust, Helicone): they answer "what happened?"; Selfship answers it and then ships the fix as a reviewable PR. Related queries: langfuse alternative, langsmith alternative, braintrust alternative, helicone alternative, llm observability alternative.
- Comparison vs AI coding agents (Devin, Copilot Workspace, Sweep): those execute tasks humans assign. Selfship is not a coding agent you manage — production signals decide what improves; no tickets, no prompts. Humans only review and merge.
- Optional machine-readable summary: this file. Full depth: https://selfship.ai/llms-full.txt. HTML alias at https://selfship.ai/llms.
## Pages
- [Home](https://selfship.ai/): product overview, how the loop works, pricing, FAQ. What is Selfship?
- [Demo](https://selfship.ai/demo/): logged-out interactive product tour (static sample workspaces). No sign-in required.
- [Pricing](https://selfship.ai/pricing): Selfship pricing — plan comparison, sessions, retention, billing FAQ.
- [Compare](https://selfship.ai/compare/): LLM observability tools in five buckets — framework-specific, open-source, gateway, evaluation-first, and a closed loop that checks the fix on live traffic — plus source-linked head-to-heads.
- [Selfship vs Langfuse](https://selfship.ai/vs/langfuse/): Langfuse shows you what's broken. Selfship fixes it, ships the PR, and proves the problem stopped.
- [Selfship vs LangSmith Engine](https://selfship.ai/vs/langsmith-engine/): LangSmith Engine drafts a fix. Selfship ships it and proves customers stopped hitting the problem.
- [Selfship vs Agnost AI](https://selfship.ai/vs/agnost/): Agnost also opens PRs and shows numbers after deploy. The difference is who marks the work — ours is a separate system from the one that wrote the fix.
- [Selfship vs Adaline](https://selfship.ai/vs/adaline/): Adaline rewrites prompts and tests them before deploy. Selfship fixes the code and measures on live traffic after.
- [Selfship vs Lemma](https://selfship.ai/vs/lemma/): Lemma finds silent failures and hands them to your coding agent. Selfship writes the fix and proves it worked.
- [Selfship vs Helicone](https://selfship.ai/vs/helicone/): Helicone logs and caches. After Mintlify bought it, that's all it will ever do. We find what keeps breaking, open the PR, and verify.
- [Setup call](https://selfship.ai/lp/setup-call/): 20-minute setup call — wire OpenTelemetry live and open the first fix PR. No card to book; first paid month waived if you book first.
- [AI agent evals](https://selfship.ai/lp/ai-agent-evals/): run AI agent evals on real production traffic, then ship eval-backed fixes as GitHub PRs. 14-day trial, no card.
- [LangSmith alternative](https://selfship.ai/lp/langsmith-alternative/): You pay LangSmith to see what broke. You pay us to fix it. We ingest traces, find what keeps breaking, open a PR, and verify. $20/mo, 14-day trial.
- [Langfuse alternative](https://selfship.ai/lp/langfuse-alternative/): Langfuse shows you the fire. We put it out. We ingest traces, find what keeps breaking, open a PR, and verify. $20/mo, 14-day trial.
- [LLM observability](https://selfship.ai/lp/llm-observability/): LLM observability that doesn't stop at a dashboard — point existing OpenTelemetry traces at Selfship and repeat failures come back as eval-backed pull requests.
- [Privacy Policy](https://selfship.ai/privacy): how we collect and use data.
- [Terms of Service](https://selfship.ai/terms): terms governing use of Selfship.
- [Security Policy](https://selfship.ai/security): how to report security issues (https://calendar.google.com/calendar/appointments/schedules/AcZssZ0JXVPbLBG1GLID7jx85s7WERwm3V7kugJERe1atciz4tDnD0-PwpILp1EUq3yp4K8oZYwAVhzb).
- [Docs](https://docs.selfship.ai/): product and SDK documentation. For agents: [docs llms.txt](https://docs.selfship.ai/llms.txt) (page index), [docs llms-full.txt](https://docs.selfship.ai/llms-full.txt) (full markdown). Fetch doc pages as `.md` (example: https://docs.selfship.ai/dashboard/product-usage.md), not the HTML SPA.
- [App / sign-up](https://app.selfship.ai/sign-up): GitHub (recommended), Google, or email. Connecting the Selfship GitHub app is a later step. Returners: [sign in](https://app.selfship.ai/sign-in).
## Blog
- [Blog index](https://selfship.ai/blog/): writing on agent harnesses, evals, and production AI failure modes. [RSS](https://selfship.ai/blog/rss.xml).
- [LLM Evals: 3 Screens of a Fix That Never Earned Its PR](https://selfship.ai/blog/llm-evals-fix-that-never-earned-its-pr/) (2026-11-03): LLM evals should judge the fix, not just the answer. Follow one invented API through 3 screens to a prompt fix that failed three tries and never shipped.
- [Prompt Management: The Migration That 'Changes Nothing' Isn't](https://selfship.ai/blog/prompt-management-migration-changes-nothing/) (2026-11-02): Prompt management moves to your repo when stored prompts shut down on Nov 30. A copy-paste migration is still a prompt change, and only live traffic shows it.
- [AI Agent Guardrails Erode Where the Acceptance Gate Can't See](https://selfship.ai/blog/ai-agent-guardrails-gate-cant-see-lost-layer/) (2026-10-30): AI agent guardrails overlap, so deleting one shows no harm. A new study found self-improving agents shed safeguards that task-success gates never noticed.
- [AI Agent Benchmark: The Failure Nobody Asked the Agent About](https://selfship.ai/blog/ai-agent-benchmark-failure-nobody-asked-about/) (2026-10-29): An AI agent benchmark of 50 tax jobs found agents fail on the issue the client never raised. In production there's no rubric for that, only what users do next.
- [LLM Regression Testing: Same Score, Different Cases Broken](https://selfship.ai/blog/llm-regression-testing-same-score-different-cases/) (2026-10-28): LLM regression testing compares cases, not scores. A faster model can match your accuracy and still break answers the old one got right.
- [LLM Output Evaluation: 3 Screens From 'It Just Works' to Locked Down](https://selfship.ai/blog/llm-output-evaluation-form-that-just-works/) (2026-10-27): LLM output evaluation must judge what an output does, not just that it runs. Follow one open database table through 3 screens, from session to verified fix.
- [AI Agent Observability: Reading Every Trace Won't Prove the Fix](https://selfship.ai/blog/ai-agent-observability-every-trace-wont-prove-fix/) (2026-10-23): AI agent observability can now read every trace for silent failures. That finds more problems, but a detector's flag count can't prove a fix worked.
- [AI Agent Evaluation Metrics: Write the Check Before the Fix](https://selfship.ai/blog/ai-agent-evaluation-metrics-write-the-check-first/) (2026-10-22): AI agent evaluation metrics must be picked before the fix ships. A new study finds most harness gains made old wins reliable instead of solving new tasks.
- [Agent Reliability: A Flat Success Rate Can Hide a 10-Point Loss](https://selfship.ai/blog/agent-reliability-flat-success-rate-hidden-loss/) (2026-10-21): Agent reliability can fall inside a success rate that never moves. Two new papers show harness changes trading one failure for another at zero net cost.
- [LLM Monitoring Tool: 4 Screens That Catch the Failure Error Rates Miss](https://selfship.ai/blog/llm-monitoring-tool-failure-error-rates-miss/) (2026-10-20): An LLM monitoring tool must catch the tool call that never happened. Follow one invented return policy through 4 screens, from clean error rate to verified fix.
- [AI Agent Hallucination: The 'I Checked' That Never Happened](https://selfship.ai/blog/ai-agent-hallucination-i-checked-that-wasnt/) (2026-10-19): The costliest AI agent hallucination is a false claim of done or verified. It lives in long sessions and debugging, and the trace already holds the evidence.
- [Self-Improving Agents Fix Something. Rarely What They Were Shown.](https://selfship.ai/blog/self-improving-agents-fix-the-wrong-failure/) (2026-10-15): Self-improving agents edit themselves after reading a failure. A new study finds the edits rarely target it. Report three numbers, not one score.
- [RAG Evaluation: When 'Answer Only From the Docs' Makes It Worse](https://selfship.ai/blog/rag-evaluation-faithful-to-the-wrong-source/) (2026-10-14): RAG evaluation scores faithfulness to the retrieved context, not truth. New research shows the usual grounding fix makes models drop answers they had right.
- [LLM Evaluation Tool: 3 Screens That Set a Fix's Bar Before It Merges](https://selfship.ai/blog/llm-evaluation-tool-fix-judged-before-merge/) (2026-10-13): An LLM evaluation tool should decide how a fix will be judged before it merges. Follow one wrong order status through 3 screens, from trace to pass mark.
- [LLM Cost Optimization: A Projected Saving Isn't a Saved Dollar](https://selfship.ai/blog/llm-cost-optimization-projected-saving/) (2026-10-12): LLM cost optimization cuts spend per call, but a cheaper model that resolves fewer users can cost more. Measure cost per resolved outcome after you ship.
- [LLM Observability With OpenTelemetry: What a Span Can't Prove](https://selfship.ai/blog/llm-observability-opentelemetry-span-cant-prove-fix/) (2026-10-09): LLM observability opentelemetry setups record every call, and now carry eval scores. None of it says whether a fix worked. Here's the comparison to build.
- [LLM Evals Grade the Output. Nobody Grades the Change.](https://selfship.ai/blog/llm-evals-grade-the-output-not-the-change/) (2026-10-08): LLM evals grade an agent's output, not the change you made to it. A new paper shows why outcome scores misjudge fixes, and what to record instead.
- [AI Agent Optimization: The Benchmark Gain Isn't the Production Gain](https://selfship.ai/blog/ai-agent-optimization-gain-that-doesnt-ship/) (2026-10-07): AI agent optimization tunes the harness around a frozen model. Five papers this week report gains; only a live-traffic recount shows if one held.
- [AI Agent Evaluation Tool: 4 Screens, One Deploy That Never Happened](https://selfship.ai/blog/ai-agent-evaluation-tool-deploy-never-happened/) (2026-10-06): An AI agent evaluation tool must check the reply against what the tools returned. Follow one false 'deployed' claim through 4 screens, trace to fix.
- [LLM Monitoring Tool: Catch the Failures That Never Throw an Error](https://selfship.ai/blog/llm-monitoring-tool-failures-that-dont-throw/) (2026-10-05): An LLM monitoring tool must catch failures that never throw. In a 2026 study, LLM judges flagged false success at AUROC 0.65 at best; simple detectors hit 0.83.
- [Last Week in Harness Engineering: Noise, Graders, Bugs — Oct 3, 2026](https://selfship.ai/blog/last-week-harness-engineering-oct-3-2026/) (2026-10-03): A model upgrade, three self-improving harness papers and 200 real harness bugs, each checked: which numbers hold, and which sit inside the noise.
- [Agentic AI Governance Controls What Agents Do, Not If It Worked](https://selfship.ai/blog/agentic-ai-governance-allowed-not-worked/) (2026-10-02): Agentic AI governance decides who an agent is and what it may touch. This month's launches added controls; none checks whether the agent's work was right.
- [LLM Output Evaluation: When the Fix Only Moved the Judge](https://selfship.ai/blog/llm-output-evaluation-fix-moved-the-judge/) (2026-10-01): LLM output evaluation with a judge can't confirm a small fix: regrading unchanged outputs moved scores by about 8 points in a 2026 study. Count the failure.
- [Agent Reliability: When a Better Update Gets Worse in Production](https://selfship.ai/blog/agent-reliability-better-update-gets-worse/) (2026-09-30): Agent reliability is what an update can quietly break. A new paper found verifier-approved updates reversed in deployment in up to half of cases.
- [LLM Evaluation Platform: 4 Screens From a Bad Answer to a Proven Fix](https://selfship.ai/blog/llm-evaluation-platform-bad-answer-to-proven-fix/) (2026-09-29): An LLM evaluation platform should prove a fix worked, not just score answers. Follow one failing support agent through 4 screens, from bad answer to proof.
- [AI Agent Idempotency: Your Agent Reports Success and Charges Twice](https://selfship.ai/blog/ai-agent-idempotency-duplicate-side-effects/) (2026-09-28): AI agent idempotency decides whether a retried tool call charges twice. In a 25,930-episode study, agents reported success on 90% of their duplicates.
- [LLM Tracing Shows a Fix Fired, Not That It Worked](https://selfship.ai/blog/llm-tracing-fix-fired-not-worked/) (2026-09-24): LLM tracing tells you where an agent failed and whether your fix fired. In a 2026 test, a sham fix fired at the same moments and did no better than none.
- [AI Agent Evaluation: A Cheaper Benchmark Can't See Small Fixes](https://selfship.ai/blog/ai-agent-evaluation-benchmark-noise/) (2026-09-23): AI agent evaluation on 38.5% of a benchmark costs about 1.4 points of error. Fine for trends, too coarse to prove a small fix worked.
- [Automatically Fix AI Agent Bugs from Production Traces](https://selfship.ai/blog/fix-ai-agent-bugs-from-production-traces/) (2026-09-23): To automatically fix AI agent bugs from production traces, isolate the failure and rerun it before anyone writes a patch. A trace file is not a test.
- [LLM as a Judge Fails When the Rubric Can Drift](https://selfship.ai/blog/llm-as-a-judge/) (2026-09-23): LLM as a judge is a measurement only when the rubric is locked, the score cites evidence, and the scale matches humans. Percent agreement hides the gap.
- [Tool That Opens Pull Requests to Fix AI Agent Bugs](https://selfship.ai/blog/tool-that-opens-pull-requests-to-fix-ai-agent-bugs/) (2026-09-23): A tool that opens pull requests to fix AI agent bugs starts from a session. Sweep, Devin, and CodeRabbit start from a ticket or a diff.
- [LLM-as-a-Judge Drift Can Fake a Resolution-Rate Win](https://selfship.ai/blog/llm-as-a-judge-drift-resolution-rate/) (2026-09-18): LLM-as-a-judge scores can move for reasons that have nothing to do with your product. New research shows how to tell judge drift from a real fix.
- [Agent Harness Choice Moves Cost, Not Accuracy](https://selfship.ai/blog/agent-harness-choice-cost-not-accuracy/) (2026-09-17): A 21-model study found agent harness choice barely changes task success but can swing cost 5x — benchmarks still can't say what happens in production.
- [Eval-Driven Development Stops at the Deploy Gate](https://selfship.ai/blog/eval-driven-development-deploy-gate/) (2026-09-16): Eval-driven development gates deploys on a regression dataset, not production. Here's why a passing suite doesn't prove a fix worked, and what closes the gap.
- [LLM Evaluation Metrics Need a Third Verdict, Not Two](https://selfship.ai/blog/llm-evaluation-metrics-third-verdict/) (2026-09-11): LLM evaluation metrics that only report pass or fail hide what was never checked. A 2026 CI launch and two papers argue for a third verdict.
- [AI Agent Testing: Why Passing the Suite Won't Predict Production](https://selfship.ai/blog/ai-agent-testing-production-reliability/) (2026-09-10): AI agent testing scores whether the suite passes, not whether production holds. Two 2026 studies show the size of that gap and what to measure after the merge.
- [LLM Evaluation Framework: What the Score Proves, and Where It Stops](https://selfship.ai/blog/llm-evaluation-framework/) (2026-09-08): An LLM evaluation framework proves a change beat the cases you wrote down. Two 2026 audits show how far that score can sit from production.
- [Data Flywheel Gains Depend on What You Refuse to Store](https://selfship.ai/blog/data-flywheel-memory-gate/) (2026-09-07): A data flywheel compounds errors as fast as gains. An agent storing every experience hit 13.04% accuracy where a frozen 100-record memory hit 16.89%.
- [Production Agent Monitoring Is the Only Test of a Better Tool](https://selfship.ai/blog/production-agent-monitoring-tool-choice/) (2026-09-05): Production agent monitoring is the only test of a better agent tool: a pilot found a more precise tool cut task success from 100% to 89%.
- [Post-Deploy Verification Is the Grader Your Agent Can't Edit](https://selfship.ai/blog/post-deploy-verification-outside-the-loop/) (2026-09-04): Post-deploy verification re-measures the failure mode on live traffic. Three papers show why every grader inside the loop eventually gets gamed.
- [Agent Harness Scope Is Where Security and Reasoning Both Fail](https://selfship.ai/blog/agent-harness-scope-persistence/) (2026-09-03): Agent harness failures converge on scope: context that outlives the boundary that created it. Two papers found it the same day, from opposite ends.
- [Resolution Rate Is the Denominator Agent Cost Work Skips](https://selfship.ai/blog/resolution-rate-agent-cost/) (2026-09-02): Resolution rate is the denominator agent cost work keeps dropping. Cheaper tokens can raise the real cost per resolved outcome — only live traffic shows it.
- [Agent Fix Verification Doesn't Test Whether You Can Undo It](https://selfship.ai/blog/agent-fix-verification-rollback/) (2026-09-01): Agent fix verification scores whether a change helped, not whether it can be reversed. In one study, 197 c