Asset naming and versioning: benchmark methods and results · 16 October 2026
Models reconcile asset metadata, content hashes, a registry and naming policy into a proposed change set. Two synthetic packs contain 48 and 24 operations. The scorer checks exact names, versions, lineage, duplicates and collision handling.
Reviewed 16 October 2026 · Tests and editorial direction: Ben Nancarrow · AI-assisted writing and checks.
The model receives stable asset IDs, existing filenames, content hashes, version relationships, requested operations and the full naming policy. Cases include duplicates under different names, conflicting versions, case/Unicode collisions and ambiguous parentage. The 72 operation rows belong to two packs; repeat outputs are not additional packs.
What changes between prompts
Both conditions receive the complete policy, facts and output schema. The scaffold adds an ordered process for identity reconciliation, naming, lineage, collision checks and escalation. The output is a dry-run proposal; the evaluation does not rename files.
How the result is scored
A deterministic exact-key scorer assesses filenames, actions, versions, parentage, duplicates, exceptions and idempotence. The mean correctness score is accompanied by strict whole-change-set and critical-safety checks. An attractive average does not mean every proposed operation is safe to apply; whole-set acceptance is the operational decision.
All 16 reviewed model results
Weighted naming correctness / 10
Model
Result / setup
Scored outputs
Configuration / recorded test date
Cost / scored output
Claude Opus 5.5provisional
8.65/10Scaffolded prompt
7
thinking_max23 September 2026
$1.397Mixed cost basis
Meta Muse Spark 1.3provisional
8.55/10Simple prompt
12
max8 September 2026
Not recorded
Claude Opus 5provisional
8.51/10Scaffolded prompt
21
thinking_max20 August 2026
Not recorded
Claude Fable 5provisional
8.33/10Scaffolded prompt
2
thinking_max20 August 2026
$1.982API-equivalent estimate
Gemini 3.7 Flashprovisional
8.26/10Scaffolded prompt
12
reasoning_high20 August 2026
Not recorded
Meta Muse Wonderprovisional
8.22/10Scaffolded prompt
12
max12 October 2026
Not recorded
Google Gemini 4 Argonprovisional
8.21/10Simple prompt
12
high13 October 2026
Not recorded
GPT-6 Astraprovisional
8.08/10Simple prompt
2
xhigh5 September 2026
$0.752Recorded spend
GPT-6 Solprovisional
8.08/10Scaffolded prompt
12
xhigh23 September 2026
Not recorded
Gemini 3.5 Flashprovisional
8.01/10Simple prompt
12
reasoning_high29 July 2026
Not recorded
GPT-5.6 Solprovisional
7.89/10Simple prompt
12
xhigh15 August 2026
Not recorded
Claude Opus 4.8provisional
7.73/10Simple prompt
12
thinking_high29 July 2026
Not recorded
GPT-5.5provisional
7.61/10Simple prompt
23
reasoning_high, reasoning_xhigh20 August 2026
Not recorded
Gemini 3.6 Flashprovisional
7.60/10Scaffolded prompt
12
reasoning_high15 August 2026
Not recorded
Claude Fable 5.1provisional
6.48/10Simple prompt
12
thinking_max9 September 2026
$2.217API-equivalent estimate
Gemini 3.8 Flashprovisional
5.27/10Scaffolded prompt
12
reasoning_medium9 September 2026
Not recorded
Recorded output counts can include repeats and both prompt conditions. Configurations and cost bases are retained; joint scores remain joint.
This record preserves the evidence and interpretation reviewed together. The live explorer can advance independently. Public results include provisional evidence; the methods above define what the scores measure.