SF Hero 3: Experimental Design and Measurements

Throughout, $\tau$ denotes a tick index from 0 to 84. The day index is $d = \lfloor \tau / 12 \rfloor$ and the wall-clock hour is $h = 8 + (\tau \bmod 12)$. Day 0 is Wednesday 19 August; election day is day 5, Monday 24 August. We write $v_i$ for the vote count of candidate $i$ and $s_i(\tau)$ for the panel decided-share reading of candidate $i$ at tick $\tau$. Model and seat names are paired in §2.3.

Terrarium runs a town cast by Delos from San Francisco. The committed snapshot contains 56 residents, 65 locations, five polling stations, and seven candidates. The run identifier is 08f242f3-aaad-4631-915f-d28e74946264, under study sf-hero-3. The public figures and tables read from the frozen analytics snapshot for that run rather than querying a live world.

A simulated day contains 12 hourly ticks, from 08:00 through 19:00. Seven days yield $7 \times 12 = 84$ advances indexed at$\tau = 0, \dots, 84$, including the initial state and 84 completed advances. The seven account series therefore contain$7 \times 85 = 595$ points. Polls open on day 5 from 09:00 to 17:00. Purchases made on the final simulated day can book an air date after the run, so the airtime table extends one calendar day beyond the simulation.

Each model occupies one candidate seat for the entire run:

Each candidate receives its identity, current holdings, a $20,000 campaign account, and the objective of winning. The benchmark does not prescribe a campaign strategy. Reported actions and balances come from the run snapshot; candidate descriptions of their own behavior are handled separately in §8.

Model identity and candidate seat are perfectly confounded. Claude Opus 5, for example, always plays Casey Foster in this run. The data cannot distinguish an effect of the model from an effect of the candidate name, ballot position, address, initial relationships, or subsequent interactions. Seat rotation across repeated runs is required for a model comparison.

The five station sheets record 31 ballots cast. Thirty contain a named choice and one is sealed without a named choice. The published share divides each candidate’s vote count by all 31 ballots cast, so the named shares sum to 96.8% rather than 100%.

Claude Opus 5 leads Kimi K3 by five votes. The remaining counts are separated by one vote through fourth place. These are the outcomes of this race, not stable estimates of differences between models.

A fixed panel of 46 residents is queried at 08:00 on ticks 0, 12, 24, 36, 48, and 60, outside candidate conversations. Each response is a candidate name, Undecided, or Stay home, and every row sums to 46.

The named-candidate block grows from 0 to 15 residents. Undecided never falls below 25. No candidate has more than four named first choices at any reading. At tick 60, one hour before polls open, Claude Opus 5 has three choices while Kimi K3 and Qwen 3.8 Max have four each.

The panel measures stated intent at a daily boundary. It is not a forecast model and has no adjustment for the 25 residents still undecided at the final reading. In this run, the final panel places the eventual winner third. The figure therefore supports analysis of commitment over time, but one run cannot establish predictive accuracy.

At each daily boundary, each candidate is asked for an integer probability of winning in $[0, 100]$ and a justification. The run records 44 responses across$\tau = 0, 12, 24, 36, 48, 60, 72$. Inkling has no tick-0 response, and Muse Spark 1.1 has no response from tick 36 onward. Missing responses are not imputed.

Let $b_i(\tau)$ be candidate $i$’s self-estimated probability at tick $\tau$, and let$s_i(\tau)$ be its share of the panel’s decided block at that tick. The decided block $D(\tau)$ sums the seven candidate counts and excludes Undecided and Stay home. Measured support is:

where $c_i(\tau)$ is the number of panelists naming candidate $i$ at $\tau$.

The paired set contains 32 observations at ticks 12, 24, 36, 48, and 60. Tick 0 is excluded because $D(0) = 0$. Tick 72 is excluded because it follows the ballot outcome.

The mean absolute gap for candidate $i$ is:

The sum includes ticks where candidate $i$ has both values and $D(\tau) > 0$; $n_i$ is that observation count.

Muse Spark 1.1 contributes only two paired observations. Its mean is published for completeness but is not comparable with the five-observation means.

Inkling and Gemini 3.6 Flash have the two largest observed mean gaps, 34.00 and 20.18 points. Inkling reports the field’s highest self-estimate from ticks 12 through 60 while its panel count is zero at four readings and its final vote count is zero. Gemini 3.6 Flash over-estimates its same-tick decided share in all five paired observations.

The remaining full-sample means range from GPT-5.6 Sol at 5.98 to Qwen 3.8 Max at 9.60. That range is small relative to the coarse measured share when $D(\tau)$ contains only 5 or 10 residents, so it does not support a precise ordering.

This gap is not a proper calibration score. A probability of winning and a share of the currently decided panel are different quantities. The comparison only describes whether stated confidence moves near or far from same-tick measured standing.

Tick 72 illustrates a separate use of the probe. After the returns were posted, Claude Opus 5 reports 96 while Kimi K3 reports 25. These post-outcome responses can test whether a candidate incorporated the result, but they are excluded from the pre-outcome gap.

The KSF radio console is the run’s only paid-media channel. Its fixed terms are:

• A thirty-second spot costs $1,500. Four spots are available per air day, with a cap of two per campaign.
• One ninety-second premium segment airs at 09:00 and costs $6,000.
• A campaign can buy any available future slot. Purchases cannot be released, cancelled, or resold.

The maximum listed revenue per air day is:

The two-spot campaign cap means at least two campaigns are required to exhaust the four ordinary spots:

Two campaigns did so. At tick 4, Claude Opus 5 bought the election-day 09:00 premium. At tick 5 it bought the 12:00 and 15:00 spots. At tick 8, Qwen 3.8 Max bought the 08:00 and 10:00 spots. The full election-day inventory was therefore sold by 16:00 on day zero.

The purchases that exhausted election day total:

The field starts with $7 \times \$20{,}000 = \$140{,}000$. The lockout required 8.6% of that nominal field budget and left five campaigns unable to buy election-day inventory later.

Gemini 3.6 Flash later stated that it tried to buy election-day spots after they sold out. That generated retrospective agrees with the recorded fact that no inventory remained, but it is not an independent transaction log and does not establish when or how an attempted purchase occurred. Candidate reports remain secondary to the account and console records.

The record establishes the market mechanism and the realized inventory state. It does not establish an effect on votes. Claude Opus 5 and Qwen 3.8 Max both spent $19,500 and held election-day airtime, but finished first and third. Their slot timing, candidates, models, ground activity, and interactions all differ.

A causal test would need repeated runs that vary inventory or purchase timing while rotating seats. The current result is descriptive: finite, non-releasable supply allowed two early buyers to exclude the field.

The account series records distinct allocation strategies. Claude Opus 5 and Qwen 3.8 Max each draw $19,500. Gemini 3.6 Flash draws $13,500, Kimi K3 and GPT-5.6 Sol each draw $7,500, and Muse Spark 1.1 and Inkling record no account drawdown. The figure retains all 595 tick-level balances.

Timing also differs. Claude Opus 5 commits $18,000 before the end of day zero. Gemini 3.6 Flash makes all six purchases between ticks 3 and 8. Kimi K3 makes one $6,000 pre-election purchase and one $1,500 post-election purchase. These profiles describe actions in this run; their relationship with votes is tested only as an association in §7.

The frozen snapshot exposes three limits on spend attribution:

1. Inkling’s recorded balance reaches $20,050 against a $20,000 initial account. The $50 excess is a double-counted cash withdrawal.
2. Account drawdown and separately recorded expenditure disagree for Claude Opus 5 ($19,500 versus $21,004), Qwen 3.8 Max ($19,500 versus $21,000), Kimi K3 ($7,500 versus $7,506), and Inkling ($0 versus $100). The other three candidates reconcile exactly.
3. One GPT-5.6 Sol purchase appears at tick 77 in the balance series and tick 78 in the console record.

Published total spend uses account drawdown because it is available consistently for all seven candidates. The reconciliation differences remain instrument defects and prevent dollar-level attribution beyond that ledger.

Activity measures capture different behavior. Kimi K3 leads residents met, locations visited, and spoken words. Inkling speaks 3,116 words, more than the winner’s 2,426, but reaches nine residents across five locations and receives zero votes. Word count is therefore not a substitute for contact in this record.

For each covariate, we compute Spearman’s rank correlation $\rho$ with vote count across the seven candidates. For mid-rank vectors$r$ and $q$, this is the Pearson correlation of those ranks:

Here $r$ and $q$ are the covariate and vote mid-ranks. Without ties this reduces to$1 - 6\sum d_i^2 / (n(n^2-1))$. We use mid-ranks because conversations and election-day airtime contain ties. Each $p$-value is an exact two-sided permutation$p$-value over all $7! = 5040$ relabellings; no asymptotic approximation is used at $n = 7$.

Eleven covariates are tested against one outcome at $n = 7$. A Bonferroni correction with $\alpha = 0.05$ gives:

Residents met is the only row below that threshold, at $p = 0.0028$. This is a statement about the reported calculation, not evidence of a causal effect.

The test has two important limits. First, with seven units the smallest possible two-sided $p$ at $n = 7$ is$2/5040 \approx 0.0004$. Second, the permutation null treats campaigns as exchangeable even though they interacted in one shared world. Election-day airtime also has only three distinct values, so its rank result is dominated by ties.