Creative production planning: benchmark methods and results · 16 October 2026
Models build a six-item production queue from one synthetic brand-cycle pack: four video slots and two static slots. The answer key checks whether the plan follows the performance evidence, rejects misleading gaps and respects production constraints.
Reviewed 16 October 2026 · Tests and editorial direction: Ben Nancarrow · AI-assisted writing and checks.
The pack combines brand context, production constraints, the current creative portfolio, a performance table, a performance-blind diversity report and a codified insight library. It deliberately includes useful targets, contradicted gaps, weak-evidence traps and a data-quality exclusion.
What changes between prompts
Both prompts include the complete pack and JSON output contract. The scaffold adds a method for reconciling apparent gaps with performance and existing learning. It does not add the evaluator's answers.
How the result is scored
A deterministic score combines target recall, decoy rejection, trap handling, insight linkage and ordering. Repeated runs reuse this one brand-cycle case. A 10/10 means the plan satisfied this test's key; the score is not a rating of a finished ad's commercial impact.
All 17 reviewed model results
Answer-key planning correctness / 10
Model
Result / setup
Scored outputs
Configuration / recorded test date
Cost / scored output
Claude Fable 5.1provisional
10.00/10Simple prompt
2
thinking_max1 September 2026
$2.005API-equivalent estimate
Claude Opus 4.8provisional
10.00/10Simple prompt
6
thinking_max5 July 2026
$1.170Recorded spend
Claude Opus 5.5provisional
10.00/10Scaffolded prompt
2
thinking_max23 September 2026
$1.626API-equivalent estimate
Gemini 3.7 Flashprovisional
10.00/10Scaffolded prompt
6
reasoning_high20 August 2026
$0.014Recorded spend
Gemini 3.8 Flashprovisional
10.00/10Scaffolded prompt
2
reasoning_high4 September 2026
$0.049Recorded spend
Google Gemini 4 Argonprovisional
10.00/10Simple prompt
2
high13 October 2026
$0.122Recorded spend
GPT-6 Astraprovisional
10.00/10Simple prompt
2
xhigh5 September 2026
$0.577Recorded spend
GPT-6 Solprovisional
10.00/10Simple prompt
2
xhigh23 September 2026
$0.110Recorded spend
Meta Muse Spark 1.3provisional
10.00/10Scaffolded prompt
2
max8 September 2026
$0.085Recorded spend
Meta Muse Wonderprovisional
10.00/10Simple prompt
2
max13 October 2026
$0.238Recorded spend
Claude Opus 5provisional
9.92/10Combined setups
6
thinking_max15 August 2026
$0.393Recorded spend
Gemini 3.1 Pro Previewprovisional
9.92/10Scaffolded prompt
6
reasoning_high28 August 2026
$0.093Recorded spend
Claude Fable 5provisional
9.83/10Scaffolded prompt
6
thinking_max20 August 2026
$1.080API-equivalent estimate
GPT-5.5provisional
9.83/10Combined setups
6
reasoning_high3 July 2026
$0.154Recorded spend
Gemini 3.6 Flashprovisional
9.75/10Combined setups
6
reasoning_high15 August 2026
$0.027Recorded spend
GPT-5.6 Solprovisional
9.75/10Combined setups
6
xhigh15 August 2026
$0.184Recorded spend
Gemini 3.5 Flashprovisional
9.36/10Simple prompt
6
reasoning_high4 July 2026
$0.015Recorded spend
Recorded output counts can include repeats and both prompt conditions. Configurations and cost bases are retained; joint scores remain joint.
This record preserves the evidence and interpretation reviewed together. The live explorer can advance independently. Public results include provisional evidence; the methods above define what the scores measure.