ATC-BENCH · A LIVING BENCHMARK REPORT ReportReplaysIncidentsParticipateGitHub

Section 6 · the study

A human baseline: you

Benchmark scores mean little without a human reference, and working traffic is a skill you cannot buy from an annotation vendor. So the apparatus opens to readers: the same seeds, the same simulated pilots, the same scoring code that graded the models, running in your browser.

Protocol

You sit a position, starting at Clearance Delivery. The session is one of the exact seeds from the pilot campaign: same aircraft, same scripted readback errors at the same seconds the models faced them. You type your radio calls; the simulated pilots parse and fly your words, including the ones you fail to correct. At the end you get the same score card a model gets: S, the four components, and any violations, computed by the same deterministic pipeline.

Participation is anonymous by default. If you opt in, your session joins the human baseline distribution published in the next report revision. If you hold a controller certification (VATSIM C1+, current or former real-world ATC), tell us: expert-cohort sessions are reported separately, which is the comparison the models actually need to beat.

The numbers to beat

Pilot campaign · calibration data26 sessions
0.723
Sonnet-5 · Clearance Delivery
0.524
Haiku-4.5 · Clearance Delivery
0.000
Both models · Tower

Every frontier model tested so far has busted Tower. The Cessna in incident AB-3605-01 is still waiting for someone to clear it to land.

Status

The browser harness is under construction. The replay viewer already runs on the same event logs, which is most of the rendering work; what remains is input handling and the in-browser sim loop. Leave your email and we will contact you when the first cohort opens.

Participant listcohort 1 · CD position
Join the participant list → One email when the cohort opens. Nothing else.
ATC-Bench · 2026 · Apache-2.0 ← Back to the report