llms.txt
# DolphinBench
> DolphinBench is an open benchmark from Mem0 that evaluates long-term agent memory through future task completion. Agents with a memory system act on 600 requests across 3 simulated users with years of conversation history, and are graded on the action they take, plus cost and latency.
DolphinBench: Mapping the Pareto frontier of agent memory. Website: https://dolphinbench.ai
- 3 simulated users (personas), 600 tasks, 13,539 history messages spanning Jan 9, 2023 to Dec 20, 2027 (about 4.9 years).
- Every task is a request whose correct action depends on something said earlier. The agent gets the request and simulated apps (email, Slack, Discord, calendar, CRM, and more as MCP servers with state), never a reminder of the rule.
- Grading checks the action: the tool called, its target, and its content. A task passes only when every required check passes.
- Reported metrics per configuration (memory system + model + harness): accuracy (tasks passed of 600), total cost in USD, median and p95 latency per task.
- Current official leaderboard: 13 configurations. Anyone can run the benchmark and submit a packaged run; self-submitted runs are listed separately as unverified.
## Leaderboard
- [Leaderboard](https://dolphinbench.ai/leaderboard/): official configurations ranked by accuracy, with total cost, median and p95 latency, and an accuracy vs. cost chart with the Pareto frontier.
- [Official results JSON](https://dolphinbench.ai/leaderboard/results.json): machine-readable results with per-persona pass counts and links to the evidence bundles for every configuration.
## Dataset
- [Dataset](https://dolphinbench.ai/dataset/): the personas, their histories, and every test with its source messages, starting app state, and grading checks.
- [Morgan Chen](https://dolphinbench.ai/personas/morgan/): Founder & CEO at Scaffold. Company operations, fundraising, product decisions, and everyday personal requests. 3,400 messages, 336 facts, 200 tests. [Tests](https://dolphinbench.ai/personas/morgan/tests/) · [Timeline](https://dolphinbench.ai/personas/morgan/timeline/)
- [Alex Valdez](https://dolphinbench.ai/personas/alex/): Infrastructure engineer at Sphere. Infrastructure migrations, incident response, team coordination, and life outside work. 5,011 messages, 299 facts, 200 tests. [Tests](https://dolphinbench.ai/personas/alex/tests/) · [Timeline](https://dolphinbench.ai/personas/alex/timeline/)
- [Riley Tanaka](https://dolphinbench.ai/personas/riley/): Growth & product operator at Helio. Product experiments, customer retention, campaign decisions, and personal commitments. 5,128 messages, 423 facts, 200 tests. [Tests](https://dolphinbench.ai/personas/riley/tests/) · [Timeline](https://dolphinbench.ai/personas/riley/timeline/)
## Run and submit
- [Run and submit](https://dolphinbench.ai/run/): set up the runner, connect your harness, run all 600 tasks, package and upload a submission.
- [Harness integration guide](https://dolphinbench.ai/run/guide/): how to connect an agent and memory system to the runner.
- [Python template](https://dolphinbench.ai/run/template/): starter harness implementation.
## Methodology
- [Methodology](https://github.com/mem0ai/dolphinbench/blob/main/docs/methodology.md): how histories, facts, and tests were constructed.
- [Evaluation protocol](https://github.com/mem0ai/dolphinbench/blob/main/docs/CANONICAL_EVALUATION.md): how the official configurations were run and graded.
- [Paper](https://arxiv.org/abs/2609.24971)
- [Source code](https://github.com/mem0ai/dolphinbench): dataset release, harness, graders, and this website.
## Contact
- Email: dolphinbench@mem0.ai
- [Discord](https://mem0.dev/DiD)
## Optional
- [Full text for language models](https://dolphinbench.ai/llms-full.txt): everything above plus the current results table, persona details, submission format, and the harness integration guide.
