{ "schemaVersion": "benmark-task-evidence.v1", "task": { "task_id": "os_ideation_reference_ad_to_concept", "scoring_mode": "hybrid", "score_status": "scored", "model_results": [ { "model_id": "or-google-gemini-4-argon", "model_label": "Google Gemini 4 Argon", "provider": "Google via OpenRouter", "model_profile_ids": [ "high" ], "quality_score_10": 8.33, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.132977+00:00", "evaluated_item_count": 12, "score_count": 12, "scored_run_count": 1, "mean_cost_per_scored_output_usd": 0.031179, "cost_basis": "actual_cash", "costed_scored_output_count": 12 }, { "model_id": "gemini-3.8-flash", "model_label": "Gemini 3.8 Flash", "provider": "Google Gemini", "model_profile_ids": [ "reasoning_high" ], "quality_score_10": 7.89, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.090293+00:00", "evaluated_item_count": 12, "score_count": 12, "scored_run_count": 1, "mean_cost_per_scored_output_usd": 0.008849, "cost_basis": "actual_cash", "costed_scored_output_count": 12 }, { "model_id": "gpt-6-sol", "model_label": "GPT-6 Sol", "provider": "OpenAI", "model_profile_ids": [ "xhigh" ], "quality_score_10": 7.89, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-09-23T11:49:10.201000+00:00", "evaluated_item_count": 12, "score_count": 12, "scored_run_count": 1, "mean_cost_per_scored_output_usd": 0.023519, "cost_basis": "actual_cash", "costed_scored_output_count": 12 }, { "model_id": "gemini-3.1-pro-preview", "model_label": "Gemini 3.1 Pro Preview", "provider": "Google Gemini", "model_profile_ids": [ "reasoning_high" ], "quality_score_10": 7.89, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.056253+00:00", "evaluated_item_count": 12, "score_count": 12, "scored_run_count": 1, "mean_cost_per_scored_output_usd": 0.025988, "cost_basis": "actual_cash", "costed_scored_output_count": 12 }, { "model_id": "claude-opus-5-5", "model_label": "Claude Opus 5.5", "provider": "Anthropic", "model_profile_ids": [ "thinking_max" ], "quality_score_10": 7.72, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.019435+00:00", "evaluated_item_count": 12, "score_count": 12, "scored_run_count": 1, "mean_cost_per_scored_output_usd": 0.771173, "cost_basis": "mixed_actual_cash_and_api_equivalent_estimate", "costed_scored_output_count": 12 }, { "model_id": "gemini-3.5-flash", "model_label": "Gemini 3.5 Flash", "provider": "Google Gemini", "model_profile_ids": [ "reasoning_high" ], "quality_score_10": 7.56, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.219679+00:00", "evaluated_item_count": 36, "score_count": 36, "scored_run_count": 3, "mean_cost_per_scored_output_usd": 0.003593, "cost_basis": "actual_cash", "costed_scored_output_count": 36 }, { "model_id": "claude-fable-5-1", "model_label": "Claude Fable 5.1", "provider": "Anthropic", "model_profile_ids": [ "thinking_max" ], "quality_score_10": 7.5, "selected_prompt_arm": "scaffolded", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.049788+00:00", "evaluated_item_count": 12, "score_count": 12, "scored_run_count": 1, "mean_cost_per_scored_output_usd": 0.448018, "cost_basis": "api_equivalent_estimate", "costed_scored_output_count": 12 }, { "model_id": "claude-fable-5", "model_label": "Claude Fable 5", "provider": "Anthropic", "model_profile_ids": [ "thinking_max" ], "quality_score_10": 7.43, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.043281+00:00", "evaluated_item_count": 36, "score_count": 36, "scored_run_count": 3, "mean_cost_per_scored_output_usd": 0.171511, "cost_basis": "api_equivalent_estimate", "costed_scored_output_count": 36 }, { "model_id": "gpt-5.5", "model_label": "GPT-5.5", "provider": "OpenAI", "model_profile_ids": [ "reasoning_high" ], "quality_score_10": 7.41, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.239254+00:00", "evaluated_item_count": 36, "score_count": 36, "scored_run_count": 3, "mean_cost_per_scored_output_usd": 0.022328, "cost_basis": "actual_cash", "costed_scored_output_count": 36 }, { "model_id": "or-meta-muse-wonder", "model_label": "Meta Muse Wonder", "provider": "Meta via OpenRouter", "model_profile_ids": [ "max" ], "quality_score_10": 7.39, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.139021+00:00", "evaluated_item_count": 12, "score_count": 12, "scored_run_count": 1, "mean_cost_per_scored_output_usd": 0.09729, "cost_basis": "actual_cash", "costed_scored_output_count": 12 }, { "model_id": "or-meta-muse-spark-1-3", "model_label": "Meta Muse Spark 1.3", "provider": "Meta via OpenRouter", "model_profile_ids": [ "high" ], "quality_score_10": 7.33, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.113262+00:00", "evaluated_item_count": 12, "score_count": 12, "scored_run_count": 1, "mean_cost_per_scored_output_usd": 0.019327, "cost_basis": "actual_cash", "costed_scored_output_count": 12 }, { "model_id": "gemini-3.7-flash", "model_label": "Gemini 3.7 Flash", "provider": "Google Gemini", "model_profile_ids": [ "reasoning_high" ], "quality_score_10": 7.3, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.074244+00:00", "evaluated_item_count": 36, "score_count": 36, "scored_run_count": 3, "mean_cost_per_scored_output_usd": 0.003402, "cost_basis": "actual_cash", "costed_scored_output_count": 36 }, { "model_id": "gpt-6-astra", "model_label": "GPT-6 Astra", "provider": "OpenAI", "model_profile_ids": [ "xhigh" ], "quality_score_10": 7.28, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.013175+00:00", "evaluated_item_count": 12, "score_count": 12, "scored_run_count": 1, "mean_cost_per_scored_output_usd": 0.199447, "cost_basis": "actual_cash", "costed_scored_output_count": 12 }, { "model_id": "claude-opus-4-8", "model_label": "Claude Opus 4.8", "provider": "Anthropic", "model_profile_ids": [ "thinking_high" ], "quality_score_10": 7.26, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.199370+00:00", "evaluated_item_count": 36, "score_count": 36, "scored_run_count": 3, "mean_cost_per_scored_output_usd": 0.033643, "cost_basis": "actual_cash", "costed_scored_output_count": 36 }, { "model_id": "gpt-5.6-sol", "model_label": "GPT-5.6 Sol", "provider": "OpenAI", "model_profile_ids": [ "xhigh" ], "quality_score_10": 7.11, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.179375+00:00", "evaluated_item_count": 36, "score_count": 36, "scored_run_count": 3, "mean_cost_per_scored_output_usd": 0.048019, "cost_basis": "actual_cash", "costed_scored_output_count": 36 }, { "model_id": "claude-opus-5", "model_label": "Claude Opus 5", "provider": "Anthropic", "model_profile_ids": [ "thinking_max" ], "quality_score_10": 6.92, "selected_prompt_arm": "baseline", "publication_status": "provisional", "evidence_maturity": "provisional", "last_tested_at": "2026-10-16T00:32:27.126242+00:00", "evaluated_item_count": 24, "score_count": 24, "scored_run_count": 2, "mean_cost_per_scored_output_usd": 0.06755, "cost_basis": "mixed_actual_cash_and_api_equivalent_estimate", "costed_scored_output_count": 24 } ], "scaffold_comparison": { "model_results": [ { "model_id": "claude-fable-5", "model_label": "Claude Fable 5", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 18, "baseline_score_10": 7.4259, "scaffolded_score_10": 6.7223, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "claude-fable-5-1", "model_label": "Claude Fable 5.1", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 6, "baseline_score_10": 7.3887, "scaffolded_score_10": 7.5, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "claude-opus-4-8", "model_label": "Claude Opus 4.8", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 18, "baseline_score_10": 7.2592, "scaffolded_score_10": 6.9073, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "claude-opus-5", "model_label": "Claude Opus 5", "status": "partial", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 12, "baseline_score_10": 6.9167, "scaffolded_score_10": 6.389, "unpaired_baseline_count": 6, "unpaired_scaffolded_count": 6, "unscored_comparison_unit_count": 6, "invalid_comparison_unit_count": 0 }, { "model_id": "claude-opus-5-5", "model_label": "Claude Opus 5.5", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 6, "baseline_score_10": 7.7222, "scaffolded_score_10": 6.9443, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "gemini-3.1-pro-preview", "model_label": "Gemini 3.1 Pro Preview", "status": "partial", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 6, "baseline_score_10": 7.889, "scaffolded_score_10": 7.5, "unpaired_baseline_count": 12, "unpaired_scaffolded_count": 12, "unscored_comparison_unit_count": 12, "invalid_comparison_unit_count": 0 }, { "model_id": "gemini-3.5-flash", "model_label": "Gemini 3.5 Flash", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 18, "baseline_score_10": 7.5556, "scaffolded_score_10": 7.5371, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "gemini-3.7-flash", "model_label": "Gemini 3.7 Flash", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 18, "baseline_score_10": 7.2963, "scaffolded_score_10": 7.1297, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "gemini-3.8-flash", "model_label": "Gemini 3.8 Flash", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 6, "baseline_score_10": 7.889, "scaffolded_score_10": 7.2777, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "gpt-5.5", "model_label": "GPT-5.5", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 18, "baseline_score_10": 7.4074, "scaffolded_score_10": 6.7777, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "gpt-5.6-sol", "model_label": "GPT-5.6 Sol", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 18, "baseline_score_10": 7.1112, "scaffolded_score_10": 6.1852, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "gpt-6-astra", "model_label": "GPT-6 Astra", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 6, "baseline_score_10": 7.2778, "scaffolded_score_10": 5.5, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "gpt-6-sol", "model_label": "GPT-6 Sol", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 6, "baseline_score_10": 7.8888, "scaffolded_score_10": 5.7775, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "gpt-6.1-sol", "model_label": "GPT-6.1 Sol", "status": "awaiting_scores", "publication_status": "withheld", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 0, "baseline_score_10": null, "scaffolded_score_10": null, "unpaired_baseline_count": 6, "unpaired_scaffolded_count": 6, "unscored_comparison_unit_count": 6, "invalid_comparison_unit_count": 0 }, { "model_id": "or-google-gemini-4-argon", "model_label": "Google Gemini 4 Argon", "status": "complete", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 6, "baseline_score_10": 8.3333, "scaffolded_score_10": 7.7222, "unpaired_baseline_count": 0, "unpaired_scaffolded_count": 0, "unscored_comparison_unit_count": 0, "invalid_comparison_unit_count": 0 }, { "model_id": "or-meta-muse-spark-1-3", "model_label": "Meta Muse Spark 1.3", "status": "partial", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 6, "baseline_score_10": 7.3335, "scaffolded_score_10": 7.3333, "unpaired_baseline_count": 3, "unpaired_scaffolded_count": 3, "unscored_comparison_unit_count": 3, "invalid_comparison_unit_count": 0 }, { "model_id": "or-meta-muse-wonder", "model_label": "Meta Muse Wonder", "status": "partial", "publication_status": "provisional", "comparison_metric_key": "quality_score_10", "pairing_basis": "explicit_shared_identity", "comparison_unit_count": 6, "baseline_score_10": 7.3888, "scaffolded_score_10": 6.6667, "unpaired_baseline_count": 4, "unpaired_scaffolded_count": 4, "unscored_comparison_unit_count": 4, "invalid_comparison_unit_count": 0 } ] } }, "path": "/benchmarks/reference-ad-adaptation/", "sourcePublication": "abfb9707d19b695fb05c1694d46222ca", "datePublished": "2026-10-16T08:22:38.946Z", "reviewedAt": "2026-10-16T09:45:42.975Z", "methodology": { "label": "Reference ad adaptation", "summary": "Models turn reference ads into new concepts for the same synthetic brand. The fixed source set contains three images and three videos; the image-only mode uses the three images. The concepts are blind-scored for commercial potential, originality and suitability.", "scoringLabel": "Human-rated concept quality / 10", "scoreUnit": "score", "sections": [ { "heading": "What the model receives", "paragraphs": [ "Each call supplies one source ad and brand context for an invented brand, then asks for one new concept. The aim is to adapt the useful creative mechanism to the new product. The full media mode covers six source ads; output counts can also include repeats and both prompt conditions." ] }, { "heading": "What changes between prompts", "paragraphs": [ "Both conditions receive the same brand context and line-specific inputs, with a 150-word concept limit. The simple prompt states the task. The scaffold adds judging criteria, an ad-quality framework and a hook/narrative/visual/conversion structure. It therefore tests that whole package of guidance, not a single wording change." ] }, { "heading": "How the result is scored", "paragraphs": [ "Ben blind-scores commercial potential, creative innovation, and coherence/suitability on a 0–10 scale. A concept clears the test-worthy threshold only when every dimension reaches 8. The displayed mean summarises quality; it does not measure the ad's eventual conversion rate. Scoring hides model labels, although the scaffold's formatting can reveal the prompt condition." ] } ] }, "content": { "title": "Best LLM for adapting reference ads", "description": "A useful reference gives AI something concrete to work from.", "intro": "A useful reference gives AI something concrete to work from. Benmark tests whether models can carry the selling mechanism into a new concept for a different brand.", "findingTitle": "Give the model a mechanism to adapt", "resultInterpretation": "The strongest result makes reference-led ideation a useful starting point for creative development. The important choice is what to transfer: the way the ad earns attention or makes its case, rather than its surface styling.", "sections": [ { "heading": "Adapt the reason it works", "paragraphs": [ "A reference ad can compress a great deal of creative direction into one input. It shows how the opening earns attention, how the product enters the story and what gives the viewer a reason to care. The model can use those relationships to develop a route for your brand.", "That is a more useful brief than asking for something with the same colour palette or editing style. If the original sells through a demonstration, the adaptation needs a demonstration your product can support. Borrowing the format without that connection produces a familiar-looking ad with little selling logic." ] }, { "heading": "More instructions can weaken the adaptation", "paragraphs": [ "In the reviewed paired results, GPT-6 Sol scores 7.89 with the simple prompt and 5.78 with the tested scaffold. The scaffold adds a full framework and output structure; in this six-observation comparison, that additional direction did not produce stronger concepts.", "The practical lesson is to test your creative framework against a straightforward brief. Keep the same reference and brand context, then compare whether the extra instructions improve the idea. A longer response or a more orderly set of headings is not evidence of a better concept." ] }, { "heading": "Use the result in creative development", "paragraphs": [ "Ask for one clear concept, the mechanism carried over from the reference and the product-specific reason it should work. Compare the concepts on commercial potential, distinctiveness and whether a production team could understand the intended execution.", "For example, transferring a side-by-side demonstration into a product that has no visible before-and-after effect should force a rethink. That is an illustrative decision to build into the brief: the source can suggest a mechanism, but your product has to earn its use." ] } ], "related": [ { "path": "/benchmarks/ad-concept-generation/", "label": "Generating concepts from brand context" }, { "path": "/benchmarks/creative-iteration/", "label": "Developing the next version of an ad" } ] } }