Can LLM assistants turn natural-language requests — “show the heme as orange ball-and-stick”, “superpose the subunits and hide the waters” — into correct molecular visualizations? MolBench measures two distinct skills: authoring a declarative MolViewSpec scene (what the picture is) and controlling a live Mol* viewer through imperative API calls (what the tool does). Both are graded deterministically against frozen, correct-by-construction answers.
Two tracks, two skills. MVS F1 (the primary ranking) scores scene authoring — the declarative MolViewSpec tree a model emits, matched against the reference by the atoms each selection picks out. A scene scores 0 unless Mol* would show it. API F1 scores tool control — the imperative Mol*/PDBeMolstar calls that drive a live viewer (superpose, hide waters, spin, focus), the operations a static scene cannot express. The leader is highlighted; open-weight models are namespaced with a provider slash.
| # | Model | MVS F1 | Validated F1 | Valid scene | Cond F1 | API F1 | Cost | Speed |
|---|---|---|---|---|---|---|---|---|
| 1 | deepseek/deepseek-v3.2openopenrouter:deepseek/deepseek-v3.2 | 0.903 ±0.18 | 0.921 | 99% | 0.916 | 0.859 ±0.28 | $0.0453 | 8.85s |
| 2 | deepseek/deepseek-v4-flashopenopenrouter:deepseek/deepseek-v4-flash | 0.859 ±0.27 | 0.870 | 93% | 0.925 | 0.930 ±0.10 | — | 9.03s |
| 3 | claude-haiku-4-5closedanthropic:claude-haiku-4-5 | 0.857 ±0.24 | 0.877 | 96% | 0.895 | 0.928 ±0.14 | $0.3073 | 1.87s |
| 4 | gemini-3.5-flash1closedopenrouter:google/gemini-3.5-flash | 0.849 ±0.28 | 0.870 | 94% | 0.900 | 0.951 ±0.10 | $0.2420 | 5.20s |
| 5 | minimax/minimax-m3openopenrouter:minimax/minimax-m3 | 0.817 ±0.31 | 0.825 | 90% | 0.907 | 0.721 ±0.28 | — | 6.31s |
| 6 | qwen/qwen3-next-80b-a3b-instructopenopenrouter:qwen/qwen3-next-80b-a3b-instruct | 0.805 ±0.28 | 0.831 | 91% | 0.881 | 0.871 ±0.16 | — | 1.62s |
| 7 | z-ai/glm-5.2openopenrouter:z-ai/glm-5.2 | 0.805 ±0.31 | 0.832 | 90% | 0.895 | 0.932 ±0.10 | — | 7.22s |
| 8 | meta-llama/llama-4-maverickopenopenrouter:meta-llama/llama-4-maverick | 0.792 ±0.38 | 0.799 | 81% | 0.973 | 0.807 ±0.23 | — | 4.42s |
| 9 | google/gemma-3-27b-itopenopenrouter:google/gemma-3-27b-it | 0.713 ±0.38 | 0.718 | 81% | 0.876 | 0.834 ±0.15 | $0.0167 | 5.61s |
| 10 | moonshotai/kimi-k2.6openopenrouter:moonshotai/kimi-k2.6 | 0.657 ±0.42 | 0.680 | 73% | 0.902 | 0.787 ±0.30 | — | 21.24s |
| 11 | gpt-5.4-nano1closedopenrouter:openai/gpt-5.4-nano | 0.654 ±0.40 | 0.682 | 76% | 0.864 | 0.835 ±0.17 | $0.0156 | 2.53s |
| 12 | openai/gpt-oss-120bopenopenrouter:openai/gpt-oss-120b | 0.561 ±0.45 | 0.574 | 63% | 0.893 | 0.894 ±0.15 | — | 7.56s |
| 13 | qwen/qwen3-30b-a3b-instruct-2507openopenrouter:qwen/qwen3-30b-a3b-instruct-2507 | 0.519 ±0.47 | 0.530 | 56% | 0.932 | 0.897 ±0.13 | $0.0099 | 4.35s |
| 14 | openai/gpt-oss-20bopenopenrouter:openai/gpt-oss-20b | 0.252 ±0.41 | 0.256 | 29% | 0.883 | 0.881 ±0.14 | $0.0131 | 7.15s |
| 15 | nvidia/nemotron-3-super-120b-a12bopenopenrouter:nvidia/nemotron-3-super-120b-a12b | 0.078 ±0.26 | 0.079 | 9% | 0.910 | 0.860 ±0.17 | — | 8.27s |
| 16 | baseline-rulesclosed | 0.000 | — | 0% | 0.000 | 0.715 ±0.10 | — | — |
| — | mistralai/mistral-small-26032openopenrouter:mistralai/mistral-small-2603 | — | — | — | — | 0.817 ±0.17 | — | 1.39s |
MVS F1: primary scene-authoring score, 70 tasks, 1 sample each, run 2026-10-18. Engine-neutral by design — though Mol* is currently the only viewer that implements MolViewSpec. Validated F1: the same scenes after the render-and-judge cascade (see Tiered grading). Valid scene: share of samples Mol* accepts and renders; the rest are invalid JSON, invalid MVS or failed to load, and score 0 — hover for the breakdown, click for every failing sample. Cond F1: scene quality over valid scenes only — the gap from MVS F1 is how much a model loses to broken output rather than to visualization error. API F1: the tool-control track, not re-run in October: these are the 2026-07-04 results (20 tasks, ×3 samples, same grader). Cost: the October MVS run at provider list prices (— = not reported). Speed: seconds per call from the July run (October timings include rendering). ± is the spread across tasks.
1 Ran via OpenRouter in October (the July API-track result used the provider's native API). Treated as the same model. 2 Not re-run in October: OpenRouter's upstream (Mistral) rate-limited every attempt (11 tries). MVS columns blank; API track from July.
What the leaderboard actually tells you — read these before the table.
The MVS ranking is largely a valid-output ranking. Over valid scenes, every LLM clusters at 0.86–0.97 Cond F1 — yet the unconditional spread is ~0.83. Valid-scene rate alone correlates with MVS F1 at r = 0.996. Of the 266 failed samples, 227 are invalid JSON and 36 are JSON that Mol* rejects as MVS (3 more failed to load) — details.
nvidia/nemotron-3-super-120b-a12b is last among the LLMs (~0.08 MVS F1) yet scores ~0.91 Cond F1 — competent when its scene is valid, but it usually isn’t (valid scene 9%; 64 of 70 samples are invalid JSON).
The top open model, deepseek-v3.2 (0.90), beats the best closed model, claude-haiku-4-5 (0.86) at 7× lower cost ($0.0453 vs $0.3073 for the run). But “open” spans a wide range: gpt-oss-20b (0.25), nemotron-3-super-120b-a12b (0.08) sit at the bottom, mostly on invalid output.
Tasks span translation → structure-grounded → clinical. Across regimes the model spread runs ~0.67–1.00; in the expanded panel it tracks output validity — a few models rarely emit a valid scene — more than task difficulty. See the per-regime table below.
Tasks bridge clinical genetics and structure: ClinVar pathogenic variants mapped via UniProt + PDBe SIFTS numbering onto the correct residues (e.g. p53 R175H). No other molecular-viz benchmark does this.
Adding a frozen fact sheet about each structure (chains, ligands, numbering) changes MVS F1 by -0.041 on average across 15 models. On the 647 task×model pairs whose scene is valid in both runs the effect is -0.004. The drop comes from more invalid output on the longer prompt, worst for qwen3-30b-a3b-instruct-2507 (valid scenes 56% → 31%). The 2×2 ↓
With only the output contract (“bare”), 0 of 420 scenes are valid across 3 models, with or without a dossier. Mostly the JSON parses but Mol* rejects it — parameters written straight onto the node instead of under params. Models don’t know MVS from pretraining.
Every model above ran with the spec prompt: the output contract plus a neutral MolViewSpec reference. We also ran it with a dossier — a grounding pack of facts about the task’s structure (chains, residue ranges, ligand codes and atom names, SIFTS/UniProt numbering, ClinVar positions), built offline and frozen with the task. And a bare prompt, the output contract only, on three models. 70 MVS tasks, 1 sample, run 2026-10-18.
| Model | Spec F1 | +Dossier F1 | Δ | Valid (spec) | Valid (+dossier) | Δ valid in both | Δ lookup tasks |
|---|---|---|---|---|---|---|---|
| deepseek-v3.2 | 0.903 | 0.888 | -0.016 | 99% | 97% | -0.013 (n=67) | -0.152 |
| deepseek-v4-flash | 0.859 | 0.778 | -0.081 | 93% | 84% | -0.011 (n=55) | -0.073 |
| claude-haiku-4-5 | 0.857 | 0.842 | -0.015 | 96% | 93% | -0.002 (n=65) | -0.051 |
| gemini-3.5-flash | 0.849 | 0.816 | -0.033 | 94% | 90% | -0.013 (n=62) | -0.130 |
| minimax-m3 | 0.817 | 0.819 | +0.002 | 90% | 91% | -0.010 (n=59) | -0.037 |
| qwen3-next-80b-a3b-instruct | 0.805 | 0.830 | +0.025 | 91% | 91% | +0.020 (n=60) | +0.040 |
| glm-5.2 | 0.805 | 0.783 | -0.022 | 90% | 86% | +0.005 (n=56) | -0.124 |
| llama-4-maverick | 0.792 | 0.793 | +0.001 | 81% | 83% | -0.000 (n=52) | +0.127 |
| gemma-3-27b-it | 0.713 | 0.654 | -0.059 | 81% | 70% | +0.028 (n=47) | -0.159 |
| kimi-k2.6 | 0.657 | 0.568 | -0.089 | 73% | 64% | -0.008 (n=36) | -0.027 |
| gpt-5.4-nano | 0.654 | 0.605 | -0.049 | 76% | 76% | -0.047 (n=40) | +0.085 |
| gpt-oss-120b | 0.561 | 0.582 | +0.020 | 63% | 66% | -0.011 (n=26) | +0.160 |
| qwen3-30b-a3b-instruct-2507 | 0.519 | 0.266 | -0.254 | 56% | 31% | -0.013 (n=16) | +0.042 |
| gpt-oss-20b | 0.252 | 0.207 | -0.046 | 29% | 24% | +0.033 (n=6) | +0.012 |
| nemotron-3-super-120b-a12b | 0.078 | 0.081 | +0.003 | 9% | 9% | — (n=0) | -0.035 |
Δ = dossier − spec, MVS F1 (tree match, no rendering). Δ valid in both: mean paired change over the tasks whose scene Mol* accepts in both runs (n tasks). Δ lookup tasks: all 15 clinical and resolve-by-name tasks, valid or not.
| Model | Valid (spec) | F1 (spec) | Valid (bare) | F1 (bare) | Valid (bare+dossier) | F1 (bare+dossier) |
|---|---|---|---|---|---|---|
| deepseek-v3.2 | 99% | 0.903 | 0% | 0.000 | 0% | 0.000 |
| claude-haiku-4-5 | 96% | 0.857 | 0% | 0.000 | 0% | 0.000 |
| qwen3-30b-a3b-instruct-2507 | 56% | 0.519 | 0% | 0.000 | 0% | 0.000 |
Bare scenes mostly parse as JSON but are invalid MVS — see the Mol* messages. Without the reference, models don’t know the MVS tree format.
Mean MVS F1 by how the answer key is grounded, easy → hard, left to right. Best per column highlighted. Columns marked other test a different skill and are not ordered by difficulty.
| Model | Translation n=41 | Grounded (ligand/SS) n=8 | Clinical (SIFTS/ClinVar) n=9 | Resolution (by name) n=6 (other) | Interactions (atoms) n=6 (other) |
|---|---|---|---|---|---|
| deepseek-v3.2 | 0.977 | 0.973 | 0.731 | 0.673 | 0.798 |
| deepseek-v4-flash | 0.940 | 0.875 | 0.634 | 0.685 | 0.795 |
| claude-haiku-4-5 | 0.976 | 0.963 | 0.650 | 0.238 | 0.830 |
| gemini-3.5-flash | 0.976 | 1.000 | 0.719 | 0.426 | 0.395 |
| minimax-m3 | 0.927 | 0.986 | 0.630 | 0.367 | 0.569 |
| qwen3-next-80b-a3b-instruct | 0.864 | 0.824 | 0.576 | 0.704 | 0.823 |
| glm-5.2 | 0.940 | 0.748 | 0.675 | 0.475 | 0.488 |
| llama-4-maverick | 0.967 | 0.991 | 0.358 | 0.000 | 0.773 |
| gemma-3-27b-it | 0.786 | 0.863 | 0.593 | 0.472 | 0.438 |
| kimi-k2.6 | 0.761 | 0.622 | 0.502 | 0.281 | 0.602 |
| gpt-5.4-nano | 0.779 | 0.855 | 0.362 | 0.111 | 0.515 |
| gpt-oss-120b | 0.659 | 0.856 | 0.311 | 0.179 | 0.258 |
| qwen3-30b-a3b-instruct-2507 | 0.730 | 0.375 | 0.222 | 0.000 | 0.235 |
| gpt-oss-20b | 0.317 | 0.000 | 0.222 | 0.278 | 0.167 |
| nemotron-3-super-120b-a12b | 0.098 | 0.118 | 0.058 | 0.000 | 0.000 |
| baseline-rules | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
Mean MVS F1 on tasks tagged with each skill (task count under the name). Skills are derived automatically from the reference scene tree.
| Model | camera n=5 | color n=63 | interactions n=2 | load n=70 | measurement n=3 | multi-component n=50 | representation n=70 | resolve n=6 | selection n=35 | surface n=8 | transparency n=1 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| deepseek-v3.2 | 1.000 | 0.924 | 0.500 | 0.903 | 0.929 | 0.904 | 0.903 | 0.673 | 0.863 | 1.000 | 1.000 |
| deepseek-v4-flash | 0.781 | 0.874 | 0.900 | 0.859 | 0.667 | 0.861 | 0.859 | 0.685 | 0.833 | 0.996 | 0.969 |
| claude-haiku-4-5 | 1.000 | 0.919 | 1.000 | 0.857 | 0.763 | 0.891 | 0.857 | 0.238 | 0.853 | 0.962 | 0.692 |
| gemini-3.5-flash | 1.000 | 0.886 | 0.686 | 0.849 | 0.000 | 0.857 | 0.849 | 0.426 | 0.796 | 1.000 | 1.000 |
| minimax-m3 | 0.781 | 0.861 | 1.000 | 0.817 | 0.233 | 0.819 | 0.817 | 0.367 | 0.810 | 0.964 | 0.711 |
| qwen3-next-80b-a3b-instruct | 0.486 | 0.817 | 0.900 | 0.805 | 0.809 | 0.763 | 0.805 | 0.704 | 0.800 | 0.964 | 0.711 |
| glm-5.2 | 0.781 | 0.838 | 0.333 | 0.805 | 0.523 | 0.810 | 0.805 | 0.475 | 0.769 | 0.837 | 0.692 |
| llama-4-maverick | 1.000 | 0.869 | 0.500 | 0.792 | 0.976 | 0.849 | 0.792 | 0.000 | 0.792 | 0.839 | 0.711 |
| gemma-3-27b-it | 0.000 | 0.732 | 0.000 | 0.713 | 0.542 | 0.662 | 0.713 | 0.472 | 0.717 | 1.000 | 1.000 |
| kimi-k2.6 | 0.600 | 0.687 | 0.843 | 0.657 | 0.309 | 0.613 | 0.657 | 0.281 | 0.532 | 0.958 | 1.000 |
| gpt-5.4-nano | 0.912 | 0.706 | 0.461 | 0.654 | 0.500 | 0.690 | 0.654 | 0.111 | 0.599 | 0.799 | 0.667 |
| gpt-oss-120b | 0.400 | 0.596 | 0.428 | 0.561 | 0.000 | 0.644 | 0.561 | 0.179 | 0.616 | 0.462 | 0.692 |
| qwen3-30b-a3b-instruct-2507 | 0.562 | 0.566 | 0.000 | 0.519 | 0.233 | 0.487 | 0.519 | 0.000 | 0.371 | 0.714 | 0.711 |
| gpt-oss-20b | 0.000 | 0.238 | 0.000 | 0.252 | 0.000 | 0.160 | 0.252 | 0.278 | 0.114 | 0.500 | 1.000 |
| nemotron-3-super-120b-a12b | 0.000 | 0.087 | 0.000 | 0.078 | 0.000 | 0.109 | 0.078 | 0.000 | 0.128 | 0.000 | 0.000 |
| baseline-rules | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
Tree-matching can under-credit a scene that renders the same through a different tree. The escalating grader renders each valid but imperfect scene and escalates only when needed: tree-match → visual diff → VLM judge. The visual diff rescues only near-identical renders (≤ 2 pixels changing by > 48 in any channel, at 256 px), because a one-residue highlight changes just a few pixels. Across 15 models it confirmed 84 of 294 imperfect scenes (29%) as the right scene — 25 rendering the same (≤ 2 pixels differ), 59 judged the same by the VLM. That moves validated F1 by only +0.00 to +0.03: grading selections by the atoms they pick out already credits most equivalent namings.
| Model | Tree F1 | Validated F1 | Δ | Perfect | Escalated | Pixel same | VLM same | Different | Failed |
|---|---|---|---|---|---|---|---|---|---|
| deepseek-v3.2open | 0.903 | 0.921 | +0.018 | 45 | 24 | 0 | 6 | 18 | 1 |
| claude-haiku-4-5closed | 0.857 | 0.877 | +0.020 | 42 | 25 | 0 | 3 | 22 | 3 |
| deepseek-v4-flashopen | 0.859 | 0.870 | +0.011 | 44 | 21 | 4 | 3 | 14 | 5 |
| gemini-3.5-flashclosed | 0.849 | 0.870 | +0.021 | 49 | 17 | 0 | 4 | 13 | 4 |
| glm-5.2open | 0.805 | 0.832 | +0.027 | 40 | 23 | 0 | 7 | 16 | 7 |
| qwen3-next-80b-a3b-instructopen | 0.805 | 0.831 | +0.026 | 32 | 32 | 0 | 8 | 24 | 6 |
| minimax-m3open | 0.817 | 0.825 | +0.008 | 42 | 21 | 0 | 3 | 18 | 7 |
| llama-4-maverickopen | 0.792 | 0.799 | +0.007 | 49 | 8 | 1 | 2 | 5 | 13 |
| gemma-3-27b-itopen | 0.713 | 0.718 | +0.005 | 34 | 23 | 0 | 3 | 20 | 13 |
| gpt-5.4-nanoclosed | 0.654 | 0.682 | +0.028 | 11 | 42 | 18 | 4 | 20 | 17 |
| kimi-k2.6open | 0.657 | 0.680 | +0.023 | 33 | 18 | 2 | 5 | 11 | 19 |
| gpt-oss-120bopen | 0.561 | 0.574 | +0.013 | 25 | 19 | 0 | 5 | 14 | 26 |
| qwen3-30b-a3b-instruct-2507open | 0.519 | 0.530 | +0.011 | 26 | 13 | 0 | 4 | 9 | 31 |
| gpt-oss-20bopen | 0.252 | 0.256 | +0.004 | 14 | 6 | 0 | 1 | 5 | 50 |
| nemotron-3-super-120b-a12bopen | 0.078 | 0.079 | +0.001 | 4 | 2 | 0 | 1 | 1 | 64 |
Validated F1 uses a rescue-only policy: a task scores 1.0 when the cascade confirms it renders the reference scene, otherwise it keeps its tree-match score (the VLM gives a binary same/different, not a graded score). Perfect: tree match already 1.0, not rendered. Escalated: valid, imperfect scenes that were rendered. Pixel same / VLM same: rescued at the visual-diff / VLM tier. Different: the VLM saw a different scene. Failed: no valid scene, scores 0 (see failures). The VLM verdicts are not yet checked against human raters — see the side-by-side evidence in the evaluator gallery →. Judge: claude-opus-4-8.