What this test measures
The model receives spreadsheet data and an analytical brief: identify the main performance finding, explain it and recommend what to change, using CPA and ROAS. The seven datasets are synthetic, with known findings planted in advance. Each is tested with a simple prompt and separately with a general analytical checklist.
These results are provisional. They measure recovery of known findings from supplied data, not the ability to retrieve the right live-account reports, prove incrementality or operate a complete marketing workflow. Small differences do not establish a statistically reliable ordering. The score is not a human-skill rating.
Datasets and prompt setups
The exploratory pack contains five anchor cases and two holdout cases. Findings include a CPA-versus-ROAS trade-off, a placement spike, post-view attribution drift, a delivery anomaly, a day-of-week effect, creative fatigue and a tracking outage. The holdouts test anomaly types the general checklist does not describe.
The same underlying spreadsheet and answer format are used for both prompt setups. The simple prompt asks for the main finding, best entity, supporting metrics and actions. The scaffold adds general checks for aggregation, breakdowns, unreliable data and one-off effects. Neither gives the model the planted finding.
This is an anomaly-rich test pack. Performance on ordinary periods with no anomaly, new accounts and unseen reporting structures still needs separate testing.
Scoring and comparison
Responses are automatically checked against weighted answer-key criteria for the main finding, supporting details and recommended actions. Missing a critical requirement caps that dataset’s score at 5 out of 10. This measures recovery of expected analysis; wording matches and paraphrase rules do not amount to a complete assessment of reasoning or writing quality.
The published task score selects the stronger comparable prompt setup across the whole test. It does not choose a different winning setup for each individual dataset. Paired simple-versus-scaffold results are included in the structured download.
Each leading model in this release has seven paired observations, from seven cases tested in two conditions. Fourteen outputs are not fourteen independent campaigns. Some other models have additional runs, so their recorded sample counts differ. No confidence intervals or significance claims are supplied.
Model names, IDs, configuration labels and test dates are retained from the public benchmark. Provider-internal revisions and complete generation settings are not supplied by that public projection. This record does not claim to be a complete reproduction package.
Costs, provenance and updates
Cost per scored output may pool both prompt setups. Recorded spend, API-equivalent estimates and mixed cost bases retain their labels. API-equivalent estimates are not cash charges, and missing cost information is not treated as zero.
This dated record is a curated copy of the already-public benchmark evidence used for the article. Its permanent address remains available when a later release changes the leader, interpretation or methodology. The article and its reviewed results are published together; the live explorer can contain newer tests.
The CSV contains the model results and recorded sample counts. The JSON also contains paired prompt comparisons, these methodological notes and the source publication identifier. Raw model responses, private evaluation material and personal information are not included.