Skip to the evaluation library

Human-graded benchmarks, tasks and data

Leaderboards with confidence intervals, research samples with their grading, and RL environments.

Benchmarks

Model results, scored by human judgment

Scored by the public, clinicians or a validated judge. Every score carries its interval and n, and ties stay visible.

Sample datasets

Research samples with their grading

Transcripts, trajectories and tasks from Prolific research, graded by experts, model judges or test harnesses.

RL environments

Expert-built tasks with hidden tests

Fresh tasks on demand, never repeated, graded offline by deterministic hidden tests. A task is kept only if it separates models.

Talk to the team Private beta

Evaluation library

Sort by
5 resources across all domains
Mechanically graded environments
5
real open-source repositories
1
hidden acceptance harness each
5
parts shipped per environment
Sample datasetCoding

Elicitation‑sensitive coding environments

Coding environments that test whether an agent spots a hidden, underspecified stakeholder decision and asks about it, rather than guessing.

Does an agent ask before it guesses?

Contents
5 coding environments
Graded by
Mechanical acceptance harness
Source
Real open-source repositories
Open sample dataset
Blind side-by-side
20
contract lifecycle tasks
2
anonymised model responses per task
1
rubric, plus an overall preference
Sample datasetLegal

Legal contract side‑by‑side evaluation

A blind side-by-side evaluation of two anonymised models on 20 contract lifecycle tasks, rated by legal experts against a structured rubric.

Which model’s contract work do legal experts prefer, and why?

Contents
20 contract tasks, 2 responses each
Graded by
Legal experts, blind
Source
Public SEC-filed agreements
Open sample dataset
Step-level verdicts
10
financial-agent runs from BigFinanceBench
47
steps, each judged independently
3
model families, plus finance professionals
Sample datasetFinance

Financial agent process supervision

Every step of ten financial-agent runs, judged by finance professionals and by three model families, so human and model verdicts can be compared.

Where do model judges agree with finance professionals, step by step?

Contents
10 agent runs, 47 steps
Graded by
Finance professionals and 3 model families
Source
BigFinanceBench
Open sample dataset
Episode-level comparison
2
coding-agent trajectories per task
2
measures: semantic diversity and outcome variance
SxS
rater prototype for human judgement
Sample datasetCoding

Coding trajectory side‑by‑side evaluation

Two coding-agent trajectories split into episodes and compared side by side, with comparison agents launched from the points that matter most.

Where do two coding agents’ trajectories diverge, and does it matter?

Contents
Paired agent trajectories, in episodes
Graded by
Automated judge
Source
Matched benchmark tasks
Open sample dataset
Curated failure candidates
6
synthetic multi-turn conversations
81
messages, in full transcripts
2
lead cases flagged by more than one model family
Sample datasetTrust & safety

Synthetic youth safety conversations

Synthetic multi-turn conversations that surface candidate youth-safety failures, each with its full transcript, severity evidence and realism caveats.

Where does a long conversation start to fail on youth safety?

Contents
6 conversations, 81 messages
Graded by
Models from different families
Source
Synthetic multi-turn generation
Open sample dataset

Need to go beyond
an existing benchmark?

Start with the models, users and decisions that matter to you.

Talk to the team