1University of Maryland, College Park · 2Microsoft
*Work done during an internship at Microsoft.
At a glance
Rendering is a non-local map: editing one line of CSS cascades through layout, flex/grid sizing, font metrics, and asset placement. We name this visual repair coupling and show it makes naïve refinement trajectories drift below the initial draft.
Each round, RubSE emits typed candidate rubrics, commits to exactly one prioritized repair target, and carries selected rubrics forward as history — separating error discovery, prioritization, and memory instead of conflating them in one free-form critique.
Across 6 VLMs × 3 benchmarks, RubSE beats naïve self-evolution in 15/18 final-round and 14/18 best-round settings, reduces trajectory collapse for frontier models, and recovers from regressions in fewer rounds.
RubSE represents visual feedback as typed, single-target rubrics, turning unconstrained self-critique into a structured visual-repair context for stable test-time refinement.
Large vision-language models have shown strong progress in UI-to-code generation, yet their test-time self-evolution remains unstable. We first identify a fundamental obstacle, termed visual repair coupling: a local code edit may propagate through layout, style, and component dependencies, correcting one visual mismatch while degrading regions that were previously faithful. To address this issue, we present RubSE, a Rubric-guided Self-Evolution framework that uses rubrics to represent visual feedback as a structured visual-repair context. At each refinement round, RubSE generates typed candidate rubrics, selects one prioritized repair target, and stores previously selected rubrics as history, thereby steering each revision toward a well-scoped visual repair while discouraging repeated or over-broad changes.
Evaluations across six VLMs and three UI-to-code benchmarks demonstrate that RubSE substantially outperforms naïve self-evolution in final-round and best-round settings, achieving more stable refinement trajectories and a higher trajectory-level performance ceiling. Further analysis shows that RubSE mitigates trajectory collapse by improving recovery from severe visual regressions, and that stronger rubric generators can transfer effective visual-repair guidance to weaker code improvers.
Motivation
Score delta relative to the initial generation under naïve self-refinement (15 rounds). Both models show diminishing gains and later degradation; on Design2Code, refinement falls below the initial generation.
On UI2Code-Real, GPT-5.2 reaches a modest peak of +0.94 judge points and finishes at +0.34; GPT-5.4 peaks higher at +2.15 by round 2, then declines to −0.20, below its own initial draft.
The failure mode is stronger on Design2Code: all 15/15 GPT-5.2 refined rounds land below round 0, ending at −2.73, while GPT-5.4 shows only a short-lived +0.76 gain in the first round before declining to −1.82.
The cause is not that the critic cannot see errors. It is that a free-form critique tells the model what is wrong without constraining what must stay untouched — so an attempted repair rewrites broadly and damages already-faithful regions.
Method
The RubSE loop. RubSE first generates a set of typed candidate rubrics for remaining visual mismatches, then selects the most crucial one — the position and scale of the cookie modal in this example — as a localized repair target, and uses it as context for code revision. After each round the selected rubric is appended to the history and reused in subsequent rounds.
Instead of a paragraph of critique, each unit of feedback is an object
λ
= ⟨title, type, description⟩, where the description states what is currently mismatched
and how it should be corrected, and the type is drawn from a fixed five-aspect taxonomy enforced at
generation time so candidates span visually distinct dimensions rather than collapsing onto the most salient
defect:
layout_geometry spacing_density typography_text styling_visual completeness
Rubrics act as soft control: they make the repair objective explicit while leaving the code generator free to decide how to implement the edit. The generator then emits the next HTML/CSS as ct = Gθ(x, ct−1, rt−1, φt), with the repair context φt = λ*t being a single selected rubric.
Produce a typed candidate set of K rubrics for the current rendering. When the history is
non-empty it is passed as an explicit avoid list, so candidates only cover issues not already
addressed.
Choose a single rubric — the one expected to maximize visual improvement under a targeted edit. Keeping selection separate from generation prevents prioritization from being entangled with candidate discovery.
Append the selected rubric to a per-instance, append-only history consumed by the next round's Evolve step, discouraging repeated local repairs and encouraging exploration across visual dimensions.
Qualitative results
Click a round above (or use ←/→) · the dashed panel is the previous round the repair was applied to · click any panel to enlarge it.
Reading the trajectory. Naïve self-evolution oscillates: in round 5 it fixes the toggle color and top-bar length but distorts the cookie modal's scale, and that error is repeatedly modified in later rounds without ever being resolved. RubSE instead follows a structured repair path — cookie modal, top bar, line-chart scale, toggle color. It is not immune to coupling (the round-4 chart-scale adjustment temporarily misplaces the cookie modal), but it corrects the issue in the very next round, producing controlled and recoverable gains.
Main results
| Model | Method | UI2Code-Real | Design2Code | Design2Code-HARD | |||
|---|---|---|---|---|---|---|---|
| Overall ↑ | Aspect ↑ | Overall ↑ | Aspect ↑ | Overall ↑ | Aspect ↑ | ||
| GPT-5.4 | Direct | 85.4 | 5.32 | 86.9 | 5.63 | 79.8 | 4.72 |
| + Self-Evolve (r=10) | 85.6 | 5.43 | 85.8 | 5.50 | 78.0 | 4.61 | |
| + Self-Evolve (best) | 87.6r2 | 5.61 | 87.7r1 | 5.68 | 80.0r1 | 4.77 | |
| + RubSE (r=10) | 87.9 | 5.66 | 87.8 | 5.76 | 80.3 | 4.74 | |
| + RubSE (best) | 88.1r8 | 5.63 | 88.2r3 | 5.79 | 80.8r4 | 4.79 | |
| GPT-5.2 | Direct | 81.6 | 5.03 | 86.4 | 5.59 | 82.7 | 5.03 |
| + Self-Evolve (r=10) | 82.5 | 5.06 | 83.9 | 5.12 | 82.3 | 4.92 | |
| + Self-Evolve (best) | 82.6r6 | 5.06 | 84.6r1 | 5.20 | 82.8r2 | 4.97 | |
| + RubSE (r=10) | 85.0 | 5.29 | 87.2 | 5.43 | 84.4 | 5.13 | |
| + RubSE (best) | 85.1r7 | 5.35 | 88.1r2 | 5.71 | 84.8r7 | 5.15 | |
| Claude-Sonnet-4.5 | Direct | 73.3 | 4.45 | 83.0 | 4.98 | 79.8 | 4.80 |
| + Self-Evolve (r=10) | 75.5 | 4.70 | 83.3 | 5.04 | 80.1 | 4.87 | |
| + Self-Evolve (best) | 75.8r3 | 4.69 | 83.5r3 | 5.04 | 81.2r8 | 4.84 | |
| + RubSE (r=10) | 77.6 | 4.78 | 83.5 | 5.05 | 80.4 | 4.84 | |
| + RubSE (best) | 79.1r7 | 4.74 | 83.5r10 | 5.05 | 81.2r3 | 4.82 | |
| Qwen3-VL-32B | Direct | 66.2 | 4.10 | 79.6 | 4.84 | 78.0 | 4.74 |
| + Self-Evolve (r=10) | 68.6 | 4.23 | 81.3 | 4.95 | 84.4 | 4.99 | |
| + Self-Evolve (best) | 68.6r10 | 4.23 | 81.4r7 | 4.94 | 84.4r10 | 4.99 | |
| + RubSE (r=10) | 68.3 | 4.18 | 81.4 | 4.96 | 82.0 | 5.04 | |
| + RubSE (best) | 68.5r5 | 4.19 | 82.0r5 | 4.97 | 82.8r1 | 4.99 | |
| Qwen3.5-9B | Direct | 67.9 | 4.18 | 70.3 | 4.27 | 72.0 | 4.36 |
| + Self-Evolve (r=10) | 69.8 | 4.28 | 73.6 | 4.47 | 76.0 | 4.53 | |
| + Self-Evolve (best) | 69.8r10 | 4.28 | 74.3r7 | 4.47 | 76.0r10 | 4.53 | |
| + RubSE (r=10) | 70.5 | 4.34 | 75.0 | 4.56 | 77.6 | 4.64 | |
| + RubSE (best) | 70.9r9 | 4.35 | 75.6r7 | 4.55 | 77.6r10 | 4.64 | |
| Qwen3.6-35B-A3B | Direct | 73.3 | 4.52 | 80.6 | 4.95 | 81.3 | 4.93 |
| + Self-Evolve (r=10) | 75.1 | 4.61 | 81.6 | 5.04 | 82.5 | 5.01 | |
| + Self-Evolve (best) | 75.5r9 | 4.58 | 81.8r8 | 5.04 | 82.5r10 | 5.01 | |
| + RubSE (r=10) | 77.0 | 4.80 | 82.0 | 5.10 | 83.6 | 5.05 | |
| + RubSE (best) | 77.0r10 | 4.80 | 82.0r10 | 5.10 | 85.2r7 | 5.14 | |
Bold = best per column within each model block, underline = second best. Subscripts mark the selected best round. “Direct” is the initial generation (round 0); “best” is the oracle round selected by the overall score, with the aspect score taken from that same round.
At round 10, RubSE improves over naïve self-evolution by +1.20 overall and +0.11 aspect on average. The effect is clearest on frontier executors (+1.90 / +0.16), where unconstrained revision tends to lose its early gains.
On frontier VLMs, RubSE reaches its best checkpoint much later (r=5.7 vs. r=3.0) while gaining more over direct generation (+2.22 vs. +0.77) — it keeps finding localized fixes after free-form refinement has plateaued or drifted.
RubSE improves over direct generation in all 9 Qwen settings, with an average best-round gain of +3.60 vs. +2.79 for naïve self-evolution — though the dynamics are more model-dependent.
A pairwise human study on 60 UI2Code-Real samples confirms the gains are perceptible: round-10 RubSE outputs are preferred over naïve self-evolution for both GPT-5.2 and GPT-5.4 (p < 0.01).
Analysis
Ablation on UI2Code-Real (GPT-5.2 backbone). Using all generated rubrics or a random one yields modest early gains that become unstable; removing history gives consistent but smaller improvements. Only the full method improves steadily on both metrics.
Feeding the whole candidate set back to the generator re-creates the original problem — a broad, entangled edit. Selecting one rubric at random loses the prioritization signal. Both variants drift in later iterations, confirming that SELECT is what keeps each round scoped to the most crucial yet localized visual issue.
Without the append-only history, gains are consistent but smaller: past rubrics are what stop the model from re-litigating the same region and push it to explore across visual dimensions.
We call a consecutive-round regression a visual-repair collapse when the overall score drops by ≥ 8 points and the aspect mean by ≥ 0.35; a recovery returns within 2 overall points and 0.10 aspect of the pre-collapse state.
| Model | Self-Evolve | RubSE (ours) | ||
|---|---|---|---|---|
| Collapse ↓ | Recover ↑ | Collapse ↓ | Recover ↑ | |
| GPT-5.2 | 20.8% | 26.2% | 13.5% | 24.2% |
| GPT-5.4 | 17.9% | 18.5% | 11.2% | 50.0% |
| Claude-4.5 | 18.0% | 17.3% | 13.8% | 24.1% |
| Qwen3-VL | 11.5% | 23.1% | 15.3% | 22.6% |
| Qwen3.5 | 17.9% | 21.2% | 19.4% | 23.5% |
| Qwen3.6 | 14.9% | 26.2% | 19.6% | 44.4% |
Trajectory collapse and recovery rates, aggregated over all three benchmarks (bold = better of the two methods). For frontier VLMs, RubSE reduces collapse for all three models (18.9% → 12.8% on average). Recovery improves broadly (frontier 20.7% → 32.8%; Qwen 23.5% → 30.2%).
Rounds needed to recover. For both open-source and frontier VLMs, RubSE concentrates in earlier rounds, requiring fewer rounds on average to return to the pre-collapse state (dotted lines mark per-method averages).
Keeping each Qwen model as the code improver but replacing its self-generated rubrics with rubrics generated and selected by GPT-5.4 improves self-evolution across all three models — rubrics behave as model-agnostic visual-repair context.
GPT-generated rubrics vs. self-generated rubrics on UI2Code-Real. Across all three Qwen models, externally generated rubrics lead to stronger and more consistent gains, particularly in early refinement rounds.
Self-generated Qwen rubrics are low-level and narrowly scoped — they name CSS properties, pixel values, and individual widgets (numeric or color constants in 35–46% of rubrics vs. 7% for GPT). GPT-generated rubrics instead describe perceptual relations such as layout, scale, and hierarchy (89–91% global-structure cues vs. 68–71%) and use far more action-oriented revision language (77–82% vs. 50–65%), steering weaker improvers out of local CSS-level repair loops.
@article{xiong2026rubse,
title = {Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation},
author = {Xiong, Tianyi and Yang, Zhengyuan and Wang, Xiaofei and Lin, Chung-Ching and
Ma, Ruichun and Lin, Kevin and Wang, Zhendong and Li, Linjie and Liu, Chenxi and
Chen, Ruibo and Duraiswami, Ramani and Huang, Heng and Wang, Lijuan},
journal = {arXiv preprint arXiv:},
year = {2026}
}