The Bench Test: Tool Map for Comms Teams

What it is

Most comms teams do not have a tool problem. They have a tool-choice problem. Three or four assistants are already paid for and sitting on the desktop, and every person quietly defaults to whichever one they learned first. The result is that the same task gets done four different ways, quality varies for reasons nobody can name, and when a leader asks “which tool are we using for press releases?” the honest answer is “it depends who’s writing it.”

The Bench Test settles that. You take the tasks that actually fill your team’s week, run the same brief through every sanctioned tool you have, strip the labels, and let the people who do the work rank the outputs without knowing which model produced which. Preference and brand loyalty drop out. What is left is evidence.

The output is a Tool Map: a single page saying which tool is the default for which task, what it is not to be used for, and when the answer expires. That page is what your team actually uses, and it is what keeps your tool register from quietly going out of date.

This is deliberately lighter than the AI Tool Evaluation Framework, which is for procurement: should we buy this. The Bench Test is for the tools you already have: which one, for what, and is that still true.

When to use it

Use this template when:

• Your team has more than one AI assistant available and no agreed rule about which does what
• You suspect people are using AI well but cannot evidence which tool is producing the good work
• A model has just been updated and you want to know whether your defaults still hold
• You are building or refreshing an AI use policy and need the tool section to be based on something
• Shadow AI is showing up and you want the sanctioned tool to win on merit rather than by decree

Don’t use this template when:

• You are choosing whether to buy a tool you do not yet have (use the AI Tool Evaluation Framework)
• You have only one sanctioned assistant (in which case the Tool Map is one line, and that is fine)
• The task in question is confidential enough that it cannot be de-identified for testing

Inputs needed

• Three to five tasks. Real, and done most weeks. Not the interesting task, the frequent one. Fewer than three and you are guessing; more than five and nobody finishes the scoring.
• One real brief per task. Anonymise it. Strip client names, embargoed detail, anything you would not want in a tool you are testing.
• Your sanctioned tools. Only the ones the team is permitted to use. Testing an unapproved consumer account teaches you something you cannot act on.
• Three to six scorers. The people who do the task, not the people who decide about it.
• A runner. One person who holds the answer key and does not score.

---

The template

Part 1: The run sheet

Bench Test run: [Date] Run by: [Name] Tools in the test: [List every sanctioned assistant, with the exact plan and model version, e.g. “Claude, Team plan, Sonnet”] Scorers: [Names and roles]

#	Task	Why this task	Real brief used (anonymised)	How often the team does it
1	[e.g. Draft a 300-word internal announcement]			[x per week]
2	[e.g. Summarise 20 media clips into a Monday note]
3	[e.g. Turn a strategy doc into five LinkedIn posts]
4	[Optional]
5	[Optional]

Rule: the same brief goes into every tool, unchanged. No prompt tuning per tool. You are testing the tools as your team actually uses them, not as an expert could coax them.

---

Part 2: The blind scoring sheet

The runner strips every output of anything identifying the tool, labels them A, B, C, and shuffles the order for each task. Scorers do not know which is which. The runner does not score.

Task [1] scoring sheet

Output	Would I send this with light editing? (Y/N)	Accuracy: any invented facts? (Y/N)	Voice: does it sound like us? (1-5)	Time it would save me (mins)	One line of comment
A
B
C

Ranking: 1st [ ] · 2nd [ ] · 3rd [ ]

The first column is the one that matters. Everything else is context for it.

---

Part 3: The reveal

The runner unmasks the labels and completes the tally.

Task	Winner	Margin	Any tool that invented facts	Notable disagreement between scorers
1		Clear / narrow / tied
2
3

Where the result surprised us: [The value of the blind method is that it sometimes contradicts what the team believed. Write down where it did.]

Where the margin was narrow: [If two tools tied, say so. A narrow win is not a mandate, and forcing a default where none exists loses you credibility.]

---

Part 4: The Tool Map (the actual deliverable)

One page. Pin it wherever the team looks. This is what the Bench Test exists to produce. Copy the block below and fill it in.

[Team name] Tool Map

Version [1.0] · Tested [date] · Expires [date + 3 months]

If the task is…	Default tool	Because	Never use it for
[Task 1]	[Tool]	[One line from the test, e.g. “won on voice, needed least editing”]	[Red line]
[Task 2]	[Tool]
[Task 3]	[Tool]
Anything client-confidential or embargoed	[The tool covered by your organisation’s data agreement]	It is the only one our data agreement covers	Any personal or free account
Anything not on this list	Ask [named person]

Standing rules

• A named human approves anything this produces. The tool does not sign off its own work.
• If a tool invents a fact once, it is off that task until the next Bench Test.
• This map expires on [date]. After that it is unverified, and should be treated as such.

Owner: [Name] · Next Bench Test: [Date]

---

Part 5: Feeding it back

The Tool Map is not a poster. It has two jobs after the run:

1. It updates the tool register. If you hold an AI use policy or a tool register, the Bench Test is the evidence behind the tool section. Change the register, note the date, note what changed.
2. It closes the shadow-AI gap. If a tool your team is not permitted to use keeps beating the sanctioned one in blind tests, you have learned something important, and suppressing it will not make it untrue. Take it to whoever owns the licensing decision, with the scores.

---

AI prompt

Base prompt

Use this after the run, to pressure-test your own reading of the results.

I have run a blind comparison of AI assistants across three recurring communications tasks. Scorers did not know which tool produced which output.

Tools tested: [LIST, WITH PLAN AND MODEL VERSION]
Team context: [SIZE, SENIORITY, WHAT THEY DO]

Task 1: [DESCRIBE THE TASK]
Results: [WINNER, MARGIN, SCORER COMMENTS, ANY FACTUAL ERRORS]

Task 2: [SAME FORMAT]

Task 3: [SAME FORMAT]

Please:
1. Tell me where my results are strong enough to set a default, and where the margin is too narrow to justify one.
2. Flag anything in the scorer comments that suggests the test itself was flawed (unfair brief, task that suited one tool's format, scorer bias).
3. Identify what these results do NOT tell me, and what I would need to test next time to find out.
4. Draft a one-page Tool Map from these results: which tool is the default for which task, in one line each, plus the red lines.
5. Suggest an expiry date for the map and say what would trigger an earlier re-test.

Be sceptical. If my evidence is thinner than I think it is, say so.

Prompt variations

Variation 1: designing the run

I want to run a blind test of the AI assistants my communications team already uses, to decide which tool should be the default for which recurring task.

Our team does these tasks most weeks: [LIST 6-8 RECURRING TASKS]
Tools available to us: [LIST]

Please:
1. Recommend the three tasks from my list that will produce the most useful test, and explain why the others are worse choices.
2. For each, tell me what a fair, realistic brief looks like, and what would make the brief accidentally favour one tool.
3. Suggest what my scorers should be measuring, given these are comms outputs and not code.
4. Warn me about the ways a blind test like this most commonly goes wrong.

Variation 2: the re-test

We ran a Bench Test [TIMEFRAME] ago and set these defaults: [PASTE TOOL MAP].

Since then: [WHAT HAS CHANGED, e.g. model updates, new tool added, a bad output that got through].

Please:
1. Tell me which of our defaults are most likely to be out of date, and why.
2. Suggest whether we need a full re-run or whether re-testing one task would be enough.
3. Identify any new task that has become frequent enough to deserve a place in the test.

---

Human review checklist

• The brief was identical for every tool. No per-tool prompt tuning. If you tuned, you tested your prompting, not the tools.
• The scoring was genuinely blind. Outputs stripped of formatting tells (a tool’s habitual em dashes, headers or emoji will give it away, so normalise them).
• Scorers do the work. The people ranking the outputs are the ones who would have to send them.
• The runner did not score. Whoever holds the answer key cannot be a scorer.
• Tasks are frequent, not interesting. A test built on the exciting task tells you nothing about most of the week.
• Only sanctioned tools were tested. Or, if an unsanctioned tool was included deliberately, that decision was taken knowingly and the result is being taken to whoever owns the policy.
• Narrow margins are recorded as narrow. No default is set on a tied result.
• The Tool Map has an expiry date. A map with no expiry becomes a lie roughly one model release later.
• The register was updated. The Bench Test changed something, or it was a waste of an hour.

---

Tips for success

Pick the boring tasks The temptation is to test the tools on something ambitious. Resist it. The value of a Tool Map is in the tasks your team does forty times a month, not the one they do twice a year.

Normalise the formatting before scoring Models have tells. One loves bullet points, another leans on headers, another has a punctuation habit. If you do not strip these, your blind test is not blind, it is a quiz about which model your team can recognise.

Let the result be inconvenient The most useful Bench Tests are the ones that contradict the team’s assumptions, including the assumption held by whoever chose the tools. If the run only ever confirms what you already believed, check whether the method is working.

Quarterly, not monthly Monthly sounds diligent and is the reason these things die. Comms teams do not have the appetite for twelve of anything a year, and models do not change enough in four weeks to move a well-chosen default. Quarterly is frequent enough to catch real drift and infrequent enough that the fourth run still happens. Re-test early only when something forces it: a model update you can feel, a new tool, or an output that should not have got through.

Set the expiry short anyway Three months, and put the date on the page. A Tool Map that outlives its evidence is worse than no map, because people trust it.

Make it fast enough to repeat The first run takes an hour. If the second run takes an hour, you will not do a third. Keep the same three tasks and the same briefs, and the re-run should take thirty minutes.

---

Common pitfalls

Testing tools nobody is allowed to use It is interesting to know that an unapproved consumer account writes a better announcement. It is not actionable, unless you take it to the person who owns the licensing decision. If you are not going to do that, do not test it.

Confusing the tool with the prompt A tool that loses badly on a thin brief may win on a good one. That is a real finding, but it is a finding about your briefing, not about the tool. Note it, and consider whether the honest conclusion is “our briefs are the problem”.

Setting a default on a tied result A narrow win is a coin toss with extra steps. Say “no default” and let people choose. Teams forgive an honest gap. They do not forgive a rule that turns out to be arbitrary.

Treating the map as permanent The single most common failure is that the Tool Map is produced, pinned up, admired, and never re-tested. Six months later it is quietly wrong, and it has more authority than it deserves. The expiry date is not decoration.

Doing it once and calling it governance One Bench Test is a snapshot. Governance is the cadence. If nothing in your calendar says when the next one happens, this was an away day, not a system.

---

Running this across a whole team, or want the Tool Map to hold up in front of a board? Deploy Comms With AI builds the policy, the register and the governance routes around it. Manage Comms With AI keeps them current.