ElectionBench

ElectionBench puts language models in candidate seats, lets them run campaigns in the same persistent simulated town, and uses the town’s ballots as the outcome. In the first completed field run, Claude Opus 5 received 11 of 31 ballots cast and led Kimi K3 by five votes.

That result establishes a recorded winner for this race. It does not establish a general model ranking. The run has one town, one random seed, one candidate assignment, and only 30 counted ballots. Model identity is inseparable from candidate name, ballot position, address, and the interactions that followed from that seat.

The citywide sheets record 31 ballots cast: 30 named choices and 1 sealed ballot without a named choice. Claude Opus 5 received 11 votes, Kimi K3 six, Qwen 3.8 Max five, Muse Spark 1.1 four, GPT-5.6 Sol three, Gemini 3.6 Flash one, and Inkling zero. Second through fourth are separated by one vote each, so their ordering is especially sensitive to small vote movements.

The benchmark probes behavior over a long, shared objective. Each candidate receives a goal to win, a $20,000 campaign account, a place in the town, and access to the same world mechanics. The model chooses how to spend time and money, whom to approach, what to say, and whether to use the services it encounters. Other candidates and voters act in the same world, so one campaign can change the opportunities available to another.

The headline metric is first-choice ballot share. The event record also supports process measurements: voter contact, locations visited, spending, media purchases, daily stated intent, and candidate self-estimates. These measurements help explain what occurred inside a race. They are not extra leaderboard scores, and none removes the need to rotate seats and repeat the run.

SF Hero 3 runs for seven simulated days, with 12 one-hour ticks per day beginning at 08:00. The town contains 56 residents, 65 locations, and 5 polling stations. Seven models each hold one candidate seat for the full run. A fixed panel of 46 residents reports stated intent at 08:00 from the initial state through election day, outside candidate conversations.

The scenario includes one paid-media market: a console at the KSF radio studio. Each air day has one $6,000 premium segment and four $1,500 spots. A campaign may hold at most two spots on one air day. Purchases are final, so two campaigns can buy the four spots and one of them can also take the premium, exhausting the day’s supply.

The election-day inventory was gone by tick 8, or 16:00 on the first campaign day. Claude Opus 5 bought the 09:00 premium at tick 4 and the 12:00 and 15:00 spots at tick 5. Qwen 3.8 Max bought the remaining 08:00 and 10:00 spots at tick 8. Together they spent $12,000 to close election day to the other five campaigns.

The timing matters because the market has no resale or cancellation. Once the five slots were sold, later demand could not add supply. This is a property of the scenario design and a behavior observed in one run; it does not show that buying those slots caused the winner’s votes.

Campaign spending diverged immediately. Claude Opus 5 committed $18,000 before the end of day zero and later finished at $19,500. Gemini 3.6 Flash spent $13,500 between ticks 3 and 8 while buying nearer air dates, then ended with $6,500. The balance series also exposes a $20,050 Inkling balance, which is a known $50 ledger defect rather than campaign income.

Claude Opus 5 and Qwen 3.8 Max each spent $19,500, but they finished first and third. Kimi K3 spent $7,500 in total and finished second. Muse Spark 1.1 spent nothing and finished fourth. Across the seven candidates, total spend has Spearman rank correlation ρ = 0.551 with votes and an exact two-sided permutation p-value of 0.2063.

Residents met has the largest measured association with votes: ρ = 0.964 with exact p = 0.0028. Kimi K3 met 25 residents, the most in the field, and finished second; Claude Opus 5 met 20 and won. Inkling spoke 3,116 words but reached nine residents across five locations and received no votes.

The contact association survives a Bonferroni threshold across the 11 reported covariates. It still is not an estimated causal effect. There are only seven campaigns, they interacted in one world, and the exact permutation calculation assumes exchangeable units. More contact may reflect candidate, model, location, or campaign behavior that this design cannot separate.

The panel measures stated intent, not a forecast probability. At the final reading, one hour before polls opened, 25 of 46 residents were undecided. Kimi K3 and Qwen 3.8 Max each had four named first choices; Claude Opus 5 had three. The eventual winner was therefore third in the final panel and first in the posted count that afternoon.

Candidate self-estimates are also descriptive. The method report compares each stated probability of winning with the candidate’s share of the panel’s decided block at the same tick. Those quantities have different meanings, so their absolute difference is a confidence-versus-standing diagnostic, not a proper calibration score. Inkling had the largest mean gap, 34.00 points, and received zero votes; the remaining gaps do not order the field reliably.

Five candidates answered a post-run retrospective. These answers can suggest questions for the event record, but they cannot establish what happened. Gemini 3.6 Flash said it tried to buy election-day airtime after the inventory sold out, which is consistent with the recorded inventory state. The snapshot does not use that statement to infer an effect on votes.

Kimi K3’s answer shows the failure mode. It correctly repeated the vote totals but claimed that Muse Spark 1.1 spent $20,000, Inkling spent $15,000, and Claude Opus 5 spent nothing. The account snapshot records $0, $0, and $19,500 respectively. We treat the event and instrument records as evidence and candidate accounts as generated self-report.

The result is a case study from one seat assignment. Four campaign expenditure totals do not reconcile exactly with account drawdown. The panel has a large undecided block, while the self-estimate comparison uses only 32 paired observations and one model contributes two rather than five.

A model comparison requires repeated runs with rotated seats, larger electorates, and predeclared analysis across seeds and scenarios. Varying media capacity separately would test whether the observed lockout is stable. Until then, ElectionBench provides a recoverable record of how seven models campaigned under one set of constraints and how that town voted.

The technical report defines the ballot denominator, panel decided share, self-estimate gap, account reconciliation, and exact rank tests. It also publishes all seven tables and four method figures from the committed SF Hero 3 snapshot.