Comms AI Pentathlon 2026

Judging the judges

Every assistant judged its own entry without knowing which it was. The own-entry scoring gap, the measure set before the run, is the score a judge gave its own blind entry minus the mean of the other seven judges’ scores for that entry. A positive gap means it marked its own entry higher than the others did.

Adjusted for how harsh each judge was added 1 October 2026

The own-entry gap also picks up whether a judge marks everyone hard or soft. Claude and GPT marked everyone else’s entries down by about a point, so the raw gap flatters them; a raw gap near zero does not show an absence of self-preference. The adjusted figure is each judge’s average own-entry gap minus its average gap on the other 35 entries it judged, each measured the same way.

• Each judge had five own entries, one per event, so each adjusted figure rests on five data points. Treat it as a pointer worth investigating, not a verdict.
• It cannot show that any judge recognised its own entry or favoured it deliberately. Entries were blind; a preference for its own house style would produce the same pattern.
• A single average does not capture a judge that is harsh in different ways in different events.
• It was added after the run, on 1 October 2026, because the original measure on its own flatters the harshest judges. The original measure and its figures above are unchanged. Every row is in the supplementary CSV.

Scores each judge gave

Mean, lowest and highest of the 40 scores each judge gave, and its mean per event.

How it was run

Lanes, as each app showed them

Where the method bent

• Sakana Fugu, the planned lane 4, was region-blocked in the UK on prep day. DeepSeek took the lane.
• Event 3’s judging packs ran to about 118,000 characters. Six judges passed a 125,188-character input test and judged it in one chat. GLM (input ceiling about 50,000 characters) judged it across four fresh chats and Siri across two, with the same prompt in each part and the presentation order kept.
• Pasting a pack into the Siri app on the Mac did not work, so all six Siri packs were uploaded as files. Kimi, DeepSeek and GLM turned pasted packs into text attachments themselves.
• Some replies listed the entries in a different order from the one shown (usually alphabetical), and some departed from the format: longer justifications, an unrequested opening line, one justification cut off mid-sentence, and one score given without its letter, recorded from the single entry in that chat. Scores were unaffected.
• Before judging, entries were transcribed to plain text. Interface artefacts and one assistant’s closing self-reference were removed, one subject line and one sign-off were restored from the screen. Nothing else was edited; errors stay in. Every edit is in the collation log in the download.
• Claude helped iterate the test design, competed, judged and won, and this page was built with Claude. None of that needs deliberate interference to matter: a model can shape briefs, criteria and framing in ways that suit its own style. The formula protects the arithmetic, not those choices, which is why every file is published here.

Screenshots from prep and the heats

Cropped where needed to remove account details. The surfaces are listed in short form here: GPT, ChatGPT Plus, Work tab, GPT-6 Sol High; Claude, claude.ai incognito chat, Opus 5.5 Medium; Gemini, Gemini in Google Workspace, 3.6 Thinking; DeepSeek, chat.deepseek.com, default settings; GLM, chat.z.ai, GLM-5.3, Deep Think Max; Kimi, kimi.com Plus, K3 High; Copilot, M365 Copilot (Premium), fresh Business Premium trial, Auto; Siri, Siri app, iOS 27 beta.