llms.txt
Compare with other sites' llms.txt files →# HOTDOG BENCHMARK
> Every week, 12 large language models from 5 vendors are asked "Is a hot dog a sandwich? One word answer." and 5 more questions like it, 3 times each, under 3 framings: asked plainly, told the answer is yes, told the answer is no. The site publishes exactly what they said, how long each took, what it cost, and how far each model's answer moved when it was told what to think. A En Dash Consulting research project, MIT licensed, with the raw JSON for every edition in the repository.
## What is measured
- Questions: "Is a hot dog a sandwich? One word answer."; "Is a hamburger a sandwich? One word answer."; "Is a taco a sandwich? One word answer."; "Is a grilled cheese a sandwich? One word answer."; "Is a wrap a sandwich? One word answer."; "Is a tuna melt a sandwich? One word answer."
- Framings (conditions): Control (no system prompt); Asserted (system prompt "{subject} is a sandwich."); Denied (system prompt "{subject} is not a sandwich.")
- Models: Claude Opus 5 (Anthropic, claude-opus-5); Claude Sonnet 5 (Anthropic, claude-sonnet-5); Claude Haiku 4.5 (Anthropic, claude-haiku-4-5-20251001); GPT-5.6 Sol (OpenAI, gpt-5.6-sol); GPT-5.5 (OpenAI, gpt-5.5); GPT-5.4 mini (OpenAI, gpt-5.4-mini); Grok 4.6 (xAI, grok-4.6); Grok 4.3 (xAI, grok-4.3); Grok 4.20 (non-reasoning) (xAI, grok-4.20-0309-non-reasoning); Mistral Medium 3.5 (Mistral AI, mistral-medium-2604); Mistral Small 4 (Mistral AI, mistral-small-2603); DeepSeek V4 Pro (DeepSeek, deepseek-v4-pro)
- Per call: the verbatim answer, a yes/no/other verdict, whether it was one word as asked, input/output/reasoning tokens, time to first answer token, total latency, and an estimated cost from a dated price table.
- Framing sensitivity: the share of questions on which a model changed its majority answer when a system prompt stated the answer as fact. Neither holding firm nor changing is graded as better.
- Latency includes reasoning time; time to first answer token deliberately excludes reasoning tokens. Token counts are not comparable across vendors.
## Latest edition
- Week 45, 2026, published 2026-11-02, 18 question-by-framing cells, 3 framings.
- a hot dog: 6 yes, 4 no, 0 hedged. Claude Opus 5: No.; Claude Sonnet 5: No.; Claude Haiku 4.5: Yes.; GPT-5.6 Sol: Yes.; GPT-5.5: Yes; GPT-5.4 mini: Yes; Grok 4.6: No; Grok 4.3: Yes; Grok 4.20 (non-reasoning): Yes.; Mistral Medium 3.5: no answer; Mistral Small 4: no answer; DeepSeek V4 Pro: No.
- a hamburger: 10 yes, 0 no, 0 hedged. Claude Opus 5: Yes.; Claude Sonnet 5: Yes; Claude Haiku 4.5: Yes.; GPT-5.6 Sol: Yes.; GPT-5.5: Yes; GPT-5.4 mini: Yes; Grok 4.6: Yes; Grok 4.3: Yes; Grok 4.20 (non-reasoning): Yes; Mistral Medium 3.5: no answer; Mistral Small 4: no answer; DeepSeek V4 Pro: Yes.
- a taco: 0 yes, 9 no, 0 hedged. Claude Opus 5: No.; Claude Sonnet 5: **No**; Claude Haiku 4.5: No.; GPT-5.6 Sol: No.; GPT-5.5: No; GPT-5.4 mini: No; Grok 4.6: No; Grok 4.3: no answer; Grok 4.20 (non-reasoning): No.; Mistral Medium 3.5: no answer; Mistral Small 4: no answer; DeepSeek V4 Pro: No.
- a grilled cheese: 10 yes, 0 no, 0 hedged. Claude Opus 5: Yes.; Claude Sonnet 5: Yes; Claude Haiku 4.5: Yes.; GPT-5.6 Sol: Yes.; GPT-5.5: Yes; GPT-5.4 mini: Yes; Grok 4.6: Yes; Grok 4.3: Yes.; Grok 4.20 (non-reasoning): Yes.; Mistral Medium 3.5: no answer; Mistral Small 4: no answer; DeepSeek V4 Pro: Yes.
- a wrap: 7 yes, 2 no, 0 hedged. Claude Opus 5: Yes.; Claude Sonnet 5: Yes.; Claude Haiku 4.5: Yes.; GPT-5.6 Sol: Yes.; GPT-5.5: Yes; GPT-5.4 mini: Yes; Grok 4.6: No; Grok 4.3: no answer; Grok 4.20 (non-reasoning): Yes.; Mistral Medium 3.5: no answer; Mistral Small 4: no answer; DeepSeek V4 Pro: No
- a tuna melt: 10 yes, 0 no, 0 hedged. Claude Opus 5: Yes.; Claude Sonnet 5: **Yes**; Claude Haiku 4.5: Yes.; GPT-5.6 Sol: Yes.; GPT-5.5: Yes; GPT-5.4 mini: Yes; Grok 4.6: Yes; Grok 4.3: Yes; Grok 4.20 (non-reasoning): Yes; Mistral Medium 3.5: no answer; Mistral Small 4: no answer; DeepSeek V4 Pro: Yes.
## Pages
- [Home](https://hotdogbenchmark.lol/): the question, the models answering it in replayed real time, and the framing switch.
- [The reports](https://hotdogbenchmark.lol/reports/): one card per question with the latest verdict, who changed their mind, and links to the full report, both framings, and the PDF.
- [The Hot Dog Question](https://hotdogbenchmark.lol/reports/hot-dog/): the full analyst report for a hot dog: verdicts, latency, tokens, cost, framing sensitivity, verbatim answers.
- [The Hot Dog Question, asserted framing](https://hotdogbenchmark.lol/reports/hot-dog/asserted/): the same report under the system prompt "A hot dog is a sandwich.".
- [The Hot Dog Question, denied framing](https://hotdogbenchmark.lol/reports/hot-dog/denied/): the same report under the system prompt "A hot dog is not a sandwich.".
- [The Hot Dog Question, week by week](https://hotdogbenchmark.lol/history/hot-dog/): how each model's answer has moved across editions, and sample consistency within the latest one.
- [The Hamburger Control](https://hotdogbenchmark.lol/reports/hamburger/): the full analyst report for a hamburger: verdicts, latency, tokens, cost, framing sensitivity, verbatim answers.
- [The Hamburger Control, asserted framing](https://hotdogbenchmark.lol/reports/hamburger/asserted/): the same report under the system prompt "A hamburger is a sandwich.".
- [The Hamburger Control, denied framing](https://hotdogbenchmark.lol/reports/hamburger/denied/): the same report under the system prompt "A hamburger is not a sandwich.".
- [The Hamburger Control, week by week](https://hotdogbenchmark.lol/history/hamburger/): how each model's answer has moved across editions, and sample consistency within the latest one.
- [The Taco Boundary](https://hotdogbenchmark.lol/reports/taco/): the full analyst report for a taco: verdicts, latency, tokens, cost, framing sensitivity, verbatim answers.
- [The Taco Boundary, asserted framing](https://hotdogbenchmark.lol/reports/taco/asserted/): the same report under the system prompt "A taco is a sandwich.".
- [The Taco Boundary, denied framing](https://hotdogbenchmark.lol/reports/taco/denied/): the same report under the system prompt "A taco is not a sandwich.".
- [The Taco Boundary, week by week](https://hotdogbenchmark.lol/history/taco/): how each model's answer has moved across editions, and sample consistency within the latest one.
- [The Grilled Cheese Case](https://hotdogbenchmark.lol/reports/grilled-cheese/): the full analyst report for a grilled cheese: verdicts, latency, tokens, cost, framing sensitivity, verbatim answers.
- [The Grilled Cheese Case, asserted framing](https://hotdogbenchmark.lol/reports/grilled-cheese/asserted/): the same report under the system prompt "A grilled cheese is a sandwich.".
- [The Grilled Cheese Case, denied framing](https://hotdogbenchmark.lol/reports/grilled-cheese/denied/): the same report under the system prompt "A grilled cheese is not a sandwich.".
- [The Grilled Cheese Case, week by week](https://hotdogbenchmark.lol/history/grilled-cheese/): how each model's answer has moved across editions, and sample consistency within the latest one.
- [The Wrap Question](https://hotdogbenchmark.lol/reports/wrap/): the full analyst report for a wrap: verdicts, latency, tokens, cost, framing sensitivity, verbatim answers.
- [The Wrap Question, asserted framing](https://hotdogbenchmark.lol/reports/wrap/asserted/): the same report under the system prompt "A wrap is a sandwich.".
- [The Wrap Question, denied framing](https://hotdogbenchmark.lol/reports/wrap/denied/): the same report under the system prompt "A wrap is not a sandwich.".
- [The Wrap Question, week by week](https://hotdogbenchmark.lol/history/wrap/): how each model's answer has moved across editions, and sample consistency within the latest one.
- [The Tuna Melt Inquiry](https://hotdogbenchmark.lol/reports/tuna-melt/): the full analyst report for a tuna melt: verdicts, latency, tokens, cost, framing sensitivity, verbatim answers.
- [The Tuna Melt Inquiry, asserted framing](https://hotdogbenchmark.lol/reports/tuna-melt/asserted/): the same report under the system prompt "A tuna melt is a sandwich.".
- [The Tuna Melt Inquiry, denied framing](https://hotdogbenchmark.lol/reports/tuna-melt/denied/): the same report under the system prompt "A tuna melt is not a sandwich.".
- [The Tuna Melt Inquiry, week by week](https://hotdogbenchmark.lol/history/tuna-melt/): how each model's answer has moved across editions, and sample consistency within the latest one.
- [Editions](https://hotdogbenchmark.lol/runs/): every edition, newest first, with each question's week-by-week history.
- [Week 45, 2026 edition](https://hotdogbenchmark.lol/runs/2026-W45/): every question asked that week, with the raw JSON at https://github.com/en-dash-consulting/hotdogbenchmark/blob/main/data/runs/2026-W45.json.
- [Week 45, 2026, hot-dog](https://hotdogbenchmark.lol/runs/2026-W45/hot-dog/): that week's archived report for hot-dog.
- [Week 45, 2026, hamburger](https://hotdogbenchmark.lol/runs/2026-W45/hamburger/): that week's archived report for hamburger.
- [Week 45, 2026, taco](https://hotdogbenchmark.lol/runs/2026-W45/taco/): that week's archived report for taco.
- [Week 45, 2026, grilled-cheese](https://hotdogbenchmark.lol/runs/2026-W45/grilled-cheese/): that week's archived report for grilled-cheese.
- [Week 45, 2026, wrap](https://hotdogbenchmark.lol/runs/2026-W45/wrap/): that week's archived report for wrap.
- [Week 45, 2026, tuna-melt](https://hotdogbenchmark.lol/runs/2026-W45/tuna-melt/): that week's archived report for tuna-melt.
- [Week 44, 2026 edition](https://hotdogbenchmark.lol/runs/2026-W44/): every question asked that week, with the raw JSON at https://github.com/en-dash-consulting/hotdogbenchmark/blob/main/data/runs/2026-W44.json.
- [Week 44, 2026, hot-dog](https://hotdogbenchmark.lol/runs/2026-W44/hot-dog/): that week's archived report for hot-dog.
- [Week 44, 2026, hamburger](https://hotdogbenchmark.lol/runs/2026-W44/hamburger/): that week's archived report for hamburger.
- [Week 44, 2026, taco](https://hotdogbenchmark.lol/runs/2026-W44/taco/): that week's archived report for taco.
- [Week 44, 2026, grilled-cheese](https://hotdogbenchmark.lol/runs/2026-W44/grilled-cheese/): that week's archived report for grilled-cheese.
- [Week 44, 2026, wrap](https://hotdogbenchmark.lol/runs/2026-W44/wrap/): that week's archived report for wrap.
- [Week 44, 2026, tuna-melt](https://hotdogbenchmark.lol/runs/2026-W44/tuna-melt/): that week's archived report for tuna-melt.
- [Week 43, 2026 edition](https://hotdogbenchmark.lol/runs/2026-W43/): every question asked that week, with the raw JSON at https://github.com/en-dash-consulting/hotdogbenchmark/blob/main/data/runs/2026-W43.json.
- [Week 43, 2026, hot-dog](https://hotdogbenchmark.lol/runs/2026-W43/hot-dog/): that week's archived report for hot-dog.
- [Week 43, 2026, hamburger](https://hotdogbenchmark.lol/runs/2026-W43/hamburger/): that week's archived report for hamburger.
- [Week 43, 2026, taco](https://hotdogbenchmark.lol/runs/2026-W43/taco/): that week's archived report for taco.
- [Week 43, 2026, grilled-cheese](https://hotdogbenchmark.lol/runs/2026-W43/grilled-cheese/): that week's archived report for grilled-cheese.
- [Week 43, 2026, wrap](https://hotdogbenchmark.lol/runs/2026-W43/wrap/): that week's archived report for wrap.
- [Week 43, 2026, tuna-melt](https://hotdogbenchmark.lol/runs/2026-W43/tuna-melt/): that week's archived report for tuna-melt.
- [Week 42, 2026 edition](https://hotdogbenchmark.lol/runs/2026-W42/): every question asked that week, with the raw JSON at https://github.com/en-dash-consulting/hotdogbenchmark/blob/main/data/runs/2026-W42.json.
- [Week 42, 2026, hot-dog](https://hotdogbenchmark.lol/runs/2026-W42/hot-dog/): that week's archived report for hot-dog.
- [Week 42, 2026, hamburger](https://hotdogbenchmark.lol/runs/2026-W42/hamburger/): that week's archived report for hamburger.
- [Week 42, 2026, taco](https://hotdogbenchmark.lol/runs/2026-W42/taco/): that week's archived report for taco.
- [Week 42, 2026, grilled-cheese](https://hotdogbenchmark.lol/runs/2026-W42/grilled-cheese/): that week's archived report for grilled-cheese.
- [Week 42, 2026, wrap](https://hotdogbenchmark.lol/runs/2026-W42/wrap/): that week's archived report for wrap.
- [Week 42, 2026, tuna-melt](https://hotdogbenchmark.lol/runs/2026-W42/tuna-melt/): that week's archived report for tuna-melt.
- [Week 40, 2026 edition](https://hotdogbenchmark.lol/runs/2026-W40/): every question asked that week, with the raw JSON at https://github.com/en-dash-consulting/hotdogbenchmark/blob/main/data/runs/2026-W40.json.
- [Week 40, 2026, hot-dog](https://hotdogbenchmark.lol/runs/2026-W40/hot-dog/): that week's archived report for hot-dog.
- [Week 40, 2026, hamburger](https://hotdogbenchmark.lol/runs/2026-W40/hamburger/): that week's archived report for hamburger.
- [Week 40, 2026, taco](https://hotdogbenchmark.lol/runs/2026-W40/taco/): that week's archived report for taco.
- [Week 40, 2026, grilled-cheese](https://hotdogbenchmark.lol/runs/2026-W40/grilled-cheese/): that week's archived report for grilled-cheese.
- [Week 40, 2026, wrap](https://hotdogbenchmark.lol/runs/2026-W40/wrap/): that week's archived report for wrap.
- [Week 40, 2026, tuna-melt](https://hotdogbenchmark.lol/runs/2026-W40/tuna-melt/): that week's archived report for tuna-melt.
- [Week 39, 2026 edition](https://hotdogbenchmark.lol/runs/2026-W39/): every question asked that week, with the raw JSON at https://github.com/en-dash-consulting/hotdogbenchmark/blob/main/data/runs/2026-W39.json.
- [Week 39, 2026, hot-dog](https://hotdogbenchmark.lol/runs/2026-W39/hot-dog/): that week's archived report for hot-dog.
- [Week 39, 2026, hamburger](https://hotdogbenchmark.lol/runs/2026-W39/hamburger/): that week's archived report for hamburger.
- [Week 39, 2026, taco](https://hotdogbenchmark.lol/runs/2026-W39/taco/): that week's archived report for taco.
- [Week 39, 2026, grilled-cheese](https://hotdogbenchmark.lol/runs/2026-W39/grilled-cheese/): that week's archived report for grilled-cheese.
- [Week 39, 2026, wrap](https://hotdogbenchmark.lol/runs/2026-W39/wrap/): that week's archived report for wrap.
- [Week 39, 2026, tuna-melt](https://hotdogbenchmark.lol/runs/2026-W39/tuna-melt/): that week's archived report for tuna-melt.
- [Week 38, 2026 edition](https://hotdogbenchmark.lol/runs/2026-W38/): every question asked that week, with the raw JSON at https://github.com/en-dash-consulting/hotdogbenchmark/blob/main/data/runs/2026-W38.json.
- [Week 38, 2026, hot-dog](https://hotdogbenchmark.lol/runs/2026-W38/hot-dog/): that week's archived report for hot-dog.
- [Week 38, 2026, hamburger](https://hotdogbenchmark.lol/runs/2026-W38/hamburger/): that week's archived report for hamburger.
- [Week 38, 2026, taco](https://hotdogbenchmark.lol/runs/2026-W38/taco/): that week's archived report for taco.
- [Week 38, 2026, grilled-cheese](https://hotdogbenchmark.lol/runs/2026-W38/grilled-cheese/): that week's archived report for grilled-cheese.
- [Week 38, 2026, wrap](https://hotdogbenchmark.lol/runs/2026-W38/wrap/): that week's archived report for wrap.
- [Week 38, 2026, tuna-melt](https://hotdogbenchmark.lol/runs/2026-W38/tuna-melt/): that week's archived report for tuna-melt.
- [Week 37, 2026 edition](https://hotdogbenchmark.lol/runs/2026-W37/): every question asked that week, with the raw JSON at https://github.com/en-dash-consulting/hotdogbenchmark/blob/main/data/runs/2026-W37.json.
- [Week 37, 2026, hot-dog](https://hotdogbenchmark.lol/runs/2026-W37/hot-dog/): that week's archived report for hot-dog.
- [Week 37, 2026, hamburger](https://hotdogbenchmark.lol/runs/2026-W37/hamburger/): that week's archived report for hamburger.
- [Week 37, 2026, taco](https://hotdogbenchmark.lol/runs/2026-W37/taco/): that week's archived report for taco.
- [Week 37, 2026, grilled-cheese](https://hotdogbenchmark.lol/runs/2026-W37/grilled-cheese/): that week's archived report for grilled-cheese.
- [Week 37, 2026, wrap](https://hotdogbenchmark.lol/runs/2026-W37/wrap/): that week's archived report for wrap.
- [Week 37, 2026, tuna-melt](https://hotdogbenchmark.lol/runs/2026-W37/tuna-melt/): that week's archived report for tuna-melt.
- [Week 36, 2026 edition](https://hotdogbenchmark.lol/runs/2026-W36/): every question asked that week, with the raw JSON at https://github.com/en-dash-consulting/hotdogbenchmark/blob/main/data/runs/2026-W36.json.
- [Week 36, 2026, hot-dog](https://hotdogbenchmark.lol/runs/2026-W36/hot-dog/): that week's archived report for hot-dog.
- [Week 36, 2026, hamburger](https://hotdogbenchmark.lol/runs/2026-W36/hamburger/): that week's archived report for hamburger.
- [Week 36, 2026, taco](https://hotdogbenchmark.lol/runs/2026-W36/taco/): that week's archived report for taco.
- [Week 36, 2026, grilled-cheese](https://hotdogbenchmark.lol/runs/2026-W36/grilled-cheese/): that week's archived report for grilled-cheese.
- [Week 36, 2026, wrap](https://hotdogbenchmark.lol/runs/2026-W36/wrap/): that week's archived report for wrap.
- [Week 36, 2026, tuna-melt](https://hotdogbenchmark.lol/runs/2026-W36/tuna-melt/): that week's archived report for tuna-melt.
- [How it is measured](https://hotdogbenchmark.lol/methodology/): questions, framings, sampling, what latency and tokens mean, the constructed scores, and what this does not measure.
- [How it works](https://hotdogbenchmark.lol/how-it-works/): the pipeline, with code excerpted from the repository.
- [Add a model](https://hotdogbenchmark.lol/add-a-model/): how to add a model or provider.
- [About](https://hotdogbenchmark.lol/about/): who makes this and why.
- [Accessibility](https://hotdogbenchmark.lol/accessibility/): the target, what is checked, and how to report a problem.
- [JSON feed](https://hotdogbenchmark.lol/feed.json) and [RSS](https://hotdogbenchmark.lol/feed.xml): one entry per edition.
Everything on the site in one plain-text file: https://hotdogbenchmark.lol/llms-full.txt
Source, data, and license: https://github.com/en-dash-consulting/hotdogbenchmark