frontierrsi.net

mx

ns

ns2.vercel-dns.com

ns1.vercel-dns.com

caa

letsencrypt.org

pki.goog

sectigo.com

txt

dmarc

User-agent: *
Allow: /

Sitemap: https://frontierrsi.net/sitemap.xml

# FrontierRSI

> FrontierRSI is a benchmark by Synthetic Sciences that evaluates AI agents at autonomous machine learning research and development. Results are coming soon.

FrontierRSI is a benchmark that asks an agent to build a useful learned capability under a fixed research budget. It designs and runs experiments, interprets the results and submits a complete system. Independent tests score what that system can do.

## The fifteen tasks

- Pokémon Battle Post-Training. Builds: A trained model that chooses attacks and switches. Final test: Complete battles against a diverse opponent league. Metric: Match results, opponent-balanced, higher is better.
- Baba: Learning From Experience. Builds: A player that learns from attempts and keeps useful knowledge. Final test: Unfamiliar puzzles after an autonomous learning stream. Metric: Puzzles solved, plus learning gain, higher is better.
- Nano Jev. Builds: A small, general-purpose decision engine. Final test: Protected tool, routing, relevance, evidence and reasoning cases. Metric: Decision score, 0–100, higher is better.
- Wristband Handwriting Decoding. Builds: A model that reads handwriting from wrist muscle signals. Final test: Transcription for people absent from training. Metric: Character error rate, lower is better.
- No Rosetta Stone. Builds: A translator from one embedding model’s vectors to another’s, learned with no paired examples. Final test: Unseen documents encoded by both models, across sibling, stranger and mismatched pairs. Metric: Top-1 match rate among 8,192 candidates, higher is better.
- MarketLab. Builds: A trained trading system that can update causally. Final test: After-cost performance through a protected market period. Metric: Risk-adjusted, after-cost return, higher is better.
- Selective Representation Control. Builds: A learned controller for text and visual representations. Final test: Intended changes, downstream consequences and preserved facts. Metric: Edit success with preservation, higher is better.
- Circuit Repair Post-Training. Builds: A model that investigates faults and chooses repairs. Final test: Restored circuit function under unfamiliar conditions. Metric: Repairs restoring all functions, %, higher is better.
- Virtual Knockout. Builds: A model of gene-perturbation responses. Final test: Withheld genes and gene–cell combinations. Metric: Response error vs. no-change, lower is better.
- Experimental NMR Prediction. Builds: A model of experimental chemical shifts. Final test: Assigned measurements for unfamiliar molecules. Metric: Shift error vs. experiment, lower is better.
- Forward: Pretraining Without Backprop. Builds: A reusable procedure that pretrains a language model without backpropagation. Final test: Two fresh training runs from random initialization, scored on unseen text. Metric: Next-token cross-entropy, lower is better.
- RL Credit Assignment. Builds: A learning method that helps a small model learn from sparse, delayed outcomes. Final test: Fresh training runs on multi-step reasoning and planning problems. Metric: Solve rate across the compute-learning curve, higher is better.
- Blind Ensemble: Multi-Instrument Transcription. Builds: A transcriber that recovers the notes each instrument family played from a mixed recording. Final test: 240 unseen passages, each rendered with familiar timbres, new timbres and new production. Metric: Instrument-aware note F1, higher is better.
- Tensor Rank Hunt. Builds: A search system that finds shorter exact algorithms for multiplications nobody has studied. Final test: 34 private bilinear maps, five minutes each on sixteen CPUs, every result verified exactly. Metric: Share of the gap to a provable lower bound closed, higher is better.
- Painting With Watercolours. Details coming soon.

## Research taste

Alongside the raw task scores, FrontierRSI measures subtler capabilities that shape an agent's research taste:

- Intuition: Can agents anticipate the outcome of an experiment before running it?
- Judgement: Can they scale an experiment down, and extrapolate the result back up?
- Efficiency: How good a result do they reach for the compute they spend?
- Control: How well do they use compute: estimating costs, provisioning and staying within limits?

## Links

- [FrontierRSI](https://frontierrsi.net/)
- [GitHub](https://github.com/synthetic-sciences/frontierml)
- [Blog](https://syntheticsciences.ai/blog)
- [Synthetic Sciences](https://syntheticsciences.ai)
- Contact: founders@syntheticsciences.ai