Can LLM assistants turn natural-language requests — “show the heme as orange ball-and-stick”, “superpose the subunits and hide the waters” — into correct molecular visualizations? MolBench measures two distinct skills: authoring a declarative MolViewSpec scene (what the picture is) and controlling a live Mol* viewer through imperative API calls (what the tool does). Both are graded deterministically against frozen, correct-by-construction answers.
Two tracks, two skills. MVS F1 (the primary ranking) scores scene authoring — the declarative MolViewSpec tree a model emits, matched against the reference (parse failures count as 0). API F1 scores tool control — the imperative Mol*/PDBeMolstar calls that drive a live viewer (superpose, hide waters, spin, focus), the operations a static scene cannot express. The leader is highlighted; open-weight models are namespaced with a provider slash.
| # | Model | MVS F1 | Parse % | Cond F1 | API F1 | Cost | Speed |
|---|---|---|---|---|---|---|---|
| 1 | claude-haiku-4-5closedanthropic:claude-haiku-4-5 | 0.869 ±0.11 | 100% | 0.876 | 0.928 ±0.14 | $0.6254 | 1.87s |
| 2 | deepseek/deepseek-v3.2openopenrouter:deepseek/deepseek-v3.2 | 0.847 ±0.18 | 99% | 0.864 | 0.859 ±0.28 | $0.0873 | 8.85s |
| 3 | google/gemma-3-27b-itopenopenrouter:google/gemma-3-27b-it | 0.830 ±0.25 | 94% | 0.895 | 0.834 ±0.15 | $0.0329 | 5.61s |
| 4 | gemini-3.5-flashclosedgemini:gemini-3.5-flash | 0.811 ±0.22 | 96% | 0.855 | 0.951 ±0.10 | $0.2008 | 5.20s |
| 5 | gpt-5.4-nanoclosedopenai:gpt-5.4-nano | 0.681 ±0.22 | 84% | 0.832 | 0.835 ±0.17 | $0.0324 | 2.53s |
| 6 | qwen/qwen3-30b-a3b-instruct-2507openopenrouter:qwen/qwen3-30b-a3b-instruct-2507 | 0.605 ±0.29 | 78% | 0.813 | 0.897 ±0.13 | $0.0199 | 4.35s |
| 7 | openai/gpt-oss-20bopenopenrouter:openai/gpt-oss-20b | 0.529 ±0.27 | 65% | 0.863 | 0.881 ±0.14 | $0.0280 | 7.15s |
| 8 | baseline-rulesclosed | 0.000 | 9% | 0.718 | 0.715 ±0.10 | — | — |
MVS F1: primary scene-authoring score (parse failures = 0). Engine-neutral by design — though Mol* is currently the only viewer that implements MolViewSpec. Parse %: share of samples that emitted valid output. Cond F1: scene quality given valid output — the gap from MVS F1 measures how much a model is dragged down by malformed JSON rather than visualization error. API F1: the tool-control track — imperative PDBeMolstar calls (superposition, component toggles, camera) that specify a sequence of actions on a live viewer rather than a scene. Parse % and Cond F1 reflect the primary MVS run. ± is the spread across tasks; costs use provider list prices.
What the leaderboard actually tells you — read these before the table.
The MVS ranking is largely a JSON-validity ranking. Conditional on producing valid output, every LLM clusters tightly at 0.81–0.90 Cond F1 — yet the unconditional spread is ~0.34, driven almost entirely by Parse %.
gpt-oss-20b is last overall (~0.53 MVS F1) but ~4th (tied for 3rd) by Cond F1 (~0.86): a competent visualizer that is simply format-unreliable.
Open models match the small closed frontier at 6–19× lower cost — Gemma-3-27B at $0.0329, DeepSeek-V3.2 close behind. But “open” spans a wide range: Qwen3-30B and gpt-oss-20b still lag well behind on reliability.
Tasks span translation → structure-grounded → clinical. Harder regimes separate models more — model spread grows from ~0.10 to ~0.24 — exactly the property a benchmark wants. See the per-regime table below.
Tasks bridge clinical genetics and structure: ClinVar pathogenic variants mapped via UniProt + PDBe SIFTS numbering onto the correct residues (e.g. p53 R175H). No other molecular-viz benchmark does this.
Mean MVS F1 by how the answer key is grounded, easy → hard. Harder regimes pull the models apart — the spread you want from a benchmark.
| Regime | claude-haiku-4-5 | deepseek-v3.2 | gemma-3-27b-it | gemini-3.5-flash | gpt-5.4-nano | qwen3-30b-a3b-instruct-2507 | gpt-oss-20b | baseline-rules |
|---|---|---|---|---|---|---|---|---|
| Translation (n=47) | 0.912 | 0.925 | 0.882 | 0.899 | 0.727 | 0.674 | 0.587 | 0.000 |
| Grounded (ligand/SS) (n=8) | 0.870 | 0.704 | 0.745 | 0.666 | 0.662 | 0.613 | 0.167 | 0.000 |
| Clinical (SIFTS/ClinVar) (n=9) | 0.646 | 0.565 | 0.630 | 0.482 | 0.460 | 0.237 | 0.549 | 0.000 |
Mean F1 on tasks tagged with each skill (task count in parentheses). Skills are derived automatically from the reference scene tree.
| Skill | claude-haiku-4-5 | deepseek-v3.2 | gemma-3-27b-it | gemini-3.5-flash | gpt-5.4-nano | qwen3-30b-a3b-instruct-2507 | gpt-oss-20b | baseline-rules |
|---|---|---|---|---|---|---|---|---|
| camera (n=7) | 0.968 | 0.986 | 0.462 | 0.993 | 0.895 | 0.579 | 0.547 | 0.000 |
| color (n=64) | 0.869 | 0.847 | 0.830 | 0.811 | 0.681 | 0.605 | 0.529 | 0.000 |
| load (n=64) | 0.869 | 0.847 | 0.830 | 0.811 | 0.681 | 0.605 | 0.529 | 0.000 |
| multi-component (n=50) | 0.873 | 0.847 | 0.788 | 0.817 | 0.688 | 0.629 | 0.475 | 0.000 |
| representation (n=64) | 0.869 | 0.847 | 0.830 | 0.811 | 0.681 | 0.605 | 0.529 | 0.000 |
| selection (n=30) | 0.830 | 0.767 | 0.793 | 0.733 | 0.589 | 0.546 | 0.381 | 0.000 |
| surface (n=7) | 0.799 | 0.852 | 0.952 | 0.778 | 0.603 | 0.407 | 0.649 | 0.000 |
Strict tree-matching is conservative — it
penalises scenes that are rendered-equivalent but expressed with a
different (sometimes cleaner) tree, e.g. selecting all cysteines with one
label_comp_id CYS rather than enumerating them. The
escalating grader renders each imperfect prediction and escalates only
when needed: tree-match → visual diff → VLM judge. Across
7 models it confirmed 196 of 253
imperfect-tree tasks (77%) as the right scene — so
tree-match under-credits every model by +0.04 to
+0.11 F1.
| Model | Tree F1 | Validated F1 | Δ | Rescued | Diff. |
|---|---|---|---|---|---|
| claude-haiku-4-5closed | 0.853 | 0.945 | +0.092 | 41/51 | 10 |
| deepseek/deepseek-v3.2open | 0.865 | 0.939 | +0.074 | 28/37 | 9 |
| gemini-3.5-flashclosed | 0.798 | 0.907 | +0.109 | 32/39 | 7 |
| google/gemma-3-27b-itopen | 0.858 | 0.903 | +0.045 | 10/18 | 8 |
| gpt-5.4-nanoclosed | 0.690 | 0.781 | +0.091 | 35/41 | 6 |
| qwen/qwen3-30b-a3b-instruct-2507open | 0.620 | 0.704 | +0.084 | 36/46 | 10 |
| openai/gpt-oss-20bopen | 0.423 | 0.461 | +0.038 | 14/21 | 7 |
Validated F1 uses a rescue-only policy: a task scores 1.0 when the cascade confirms it renders the reference scene, otherwise it keeps its tree-match score (the VLM gives a binary same/different, not a graded score). Rescued = render-equivalent or VLM-"same" of the escalated (imperfect-tree) tasks; Diff. = confirmed genuinely different. See the side-by-side evidence in the evaluator gallery →. Judge: claude-opus-4-8.