Results

The eval asks one question: on the same budget of evaluations, does an LLM steering the exploration do better than plain optimisers with no LLM? The CORDIC design space is small enough (635,040 designs) to score every design, so every run is graded against the exact answer, always with the same (M1) cost model so the milestones are comparable.

Milestone 1 found a split: the agent picked better designs than NSGA-II and random search, but mapped the trade-off curve far worse, because it dived at the spec's corner and stopped early. Milestone 2 set out to close that gap without losing the selection lead: a map_front decision, a 40% front-mapping reserve that code spends before a run stops, and five seeds instead of three. It ran three models through OpenRouter, Sonnet 5.5, DeepSeek V4.1 Flash and Qwen3.8-27B with reasoning off: 60 runs, 232 LLM calls, $1.65 of provider-reported usage. Every table below is mean ± population standard deviation over the seeds.

Milestone 3 then put the design inside its system and tried a free-form agent on top of the structured one. Milestone 4, the last, added a second target (an ASIC on the open sky130 PDK, post-synthesis), a fleet of structured graphs orchestrated by code, and feedback from L2 to L1, and closed the project with a write-up. Milestone 4's results come first, losses first; milestones 3, 2 and 1 follow, as they were. Before them, the newest data: the local Vivado runs that settled the two FPGA questions milestone 4 left open.

After M4: the local Vivado runs (2026-10-11)

Vivado 2025.2 on the same part, run on the maintainer's machine: high_precision's two candidates synthesised and routed again at the spec's own 20 ns clock, and report_power on the 14 measured designs, the first measured FPGA power. No LLM call and no eval re-run; the default cost model and every ground truth are unchanged. The power is Vivado's own vectorless estimate (12.5% default toggle rate, confidence Low), not a board measurement.

Where the models were wrong.

• The better-fitting timing model got the one boundary wrong. A refit with one path factor per family class cut the Vivado Fmax leave-one-out error from 12.5% to 7.4%, but predicted high_precision's m=8 at 52.51 MSPS (feasible); Vivado measures 49.88. The rule, written before the fit, required both, so it was rejected.
• No model sees routing. m=6 goes from 65.39 MSPS post-synthesis to 53.96 post-route, a real margin of +7.9% over 50 MSPS where the models' unrouted figures suggest +25.0% to +27.5%. Across the Vivado designs, routed Fmax is 0.675–0.709× the synthesised figure on plain pipelined and 0.980–1.056× on unrolled_k: every model throughput on this site is an unrouted figure.
• The power index is off by 20.6% RMS after synthesis and 36.3% after routing. It reads unrolled_k low (0.61–0.97× post-synthesis, 0.48–0.85× post-route) and plain pipelined high (1.07–1.18× and 1.14–1.27×).

What held.

• m=8 misses 50 MSPS at its own clock in both views (49.88 post-synthesis, 48.31 post-route) and m=6 meets in both. The M1 model and the refit had both called m=8 infeasible, so high_precision's winner stays m=6. Relaxing the clock from 10 to 20 ns did not move either design's post-synthesis figure.
• The refit, counted once, is the same fit. The 20 ns post-synthesis rows duplicate the 10 ns ones, so vivado-2025.2-m4 keeps one row per design (4 dropped) and equals vivado-2025.2: 51 points, Vivado leave-one-out 15.4% LUT, 3.7% FF, 12.5% Fmax. No spec's winner changes; M1 stays the default.
• The power index ranks the designs well: Spearman 0.982 post-synthesis and 0.969 post-route. A per-family activity factor (4 parameters) cuts the error to 10.2% in-sample and 13.9% leave-one-out; it is kept as a separate calibration, not the default.

How the decision was recorded, in the repository's words: the first run of this decision printed ADOPTED. That was a bug, not the rule: load_calibration caches by path, and both candidates were written to one path, so the second candidate was scored with the first one's constants (identical m=8/m=6 throughputs with and without the factor). The bug was confirmed and fixed (recalibrate.refit clears the cache after writing; a test covers it), and the rule text was not changed. A later revision counts the 20 ns rows once, since they duplicate the 10 ns rows; the outcome did not change.

iterative

0.915–0.946 (mean 0.932, 3 designs)

pipelined

0.675–0.709 (mean 0.687, 3 designs)

pipelined_m

0.786–0.969 (mean 0.900, 6 designs)

unrolled_k

0.980–1.056 (mean 1.013, 4 designs)

0.60.70.80.91.01.1

iterative W12

0.95 / 1.04

iterative W16

1.00 / 1.00

iterative W24

1.07 / 0.93

pipelined W12

1.16 / 1.27

pipelined W16

1.18 / 1.16

pipelined W24

1.07 / 1.14

pipelined_m W16 m=4

0.97 / 0.87

pipelined_m W16 m=2

1.02 / 0.98

pipelined_m W26 N22 m=8

0.95 / 0.79

pipelined_m W26 N22 m=6

0.97 / 0.79

unrolled_k W16 k=2

0.97 / 0.85

unrolled_k W16 k=3

0.71 / 0.58

unrolled_k W16 k=4

0.61 / 0.48

unrolled_k W16 k=8

0.83 / 0.67

0.511.5

post-synthesis (first number)post-route (second)iterativepipelinedpipelined_munrolled_k

Two designs tie at 22.0 mW post-synthesis. The repository's Spearman ranks the tie by order of appearance (0.982); averaging the tied ranks, the usual convention, gives 0.988. Both are computed at export.

Milestone 4 at a glance

Where it lost.

• The fleet mapped the trade-off curve worse than one graph, at a higher cost. On the same five specs, seeds and 400 evaluations, the fleet had the higher hypervolume in only 18 of 55 pairs (sign test p = 0.014); significant for DeepSeek alone (6 of 25, p = 0.015), not for Qwen or Sonnet. It cost 1.41–1.60× as much per spec. The review found the fleet's shared Store was never read back and phase 2 started cold; the fix (warm starts from the Store) did not change the loss: the fixed Qwen fleet was higher in 8 of 25 (p = 0.11).
• The ASIC model's winner misses its deadline when measured. On asic_multiaxis_control the ground truth's winner meets the 0.5 µs batch deadline at its estimated 82.5 MHz; synthesised on sky130 it runs at 72.2 MHz, and its simulated p99 is then 0.567 µs, 13.5% over.
• The ASIC model's Fmax is optimistic. On the 9 held-out designs it never saw, area is within 5% on all of them, but Fmax is off by -14.9% to +39.6% and optimistic on 5 of 9 (by +8.8% or more).

What held.

• The structured graph carries to the ASIC target. Every selected design met its spec (45/45), and on the power-driven spec every model's regret was a fraction of NSGA-II's (+3.5% to +15.6% against +54.0%).
• With Qwen, the fixed fleet picked better designs: regret +6.3% against the single graph's +14.8%, lower in 22 of 25 pairs (p = 0.0002). The fixed fleet ran hours after the single arm, so provider drift is not ruled out; it is one model, and DeepSeek and Sonnet show no regret difference.
• The family ranking is the same on both targets; the ASIC wants shallower pipelining (m=3 where the FPGA wants m=7; m=2 where it wants m=5).
• L2 → L1 feedback works where it is needed, which on the committed specs is never: it fired in 0 of 92 recorded and 0 of 60 live searches; on a constructed spec it recovered 3 of 3 against 1 of 3 without the correction.
• Milestone 4 cost $4.15 by the key, under the $5.00 cap, and left the FPGA path's record untouched: the 60 M2 and 30 M3 structured runs still replay identically.

The fleet A/B: one graph against a fleet of graphs

Five specs (the three ASIC specs, low_area_control and multiaxis_control), seeds 0–4, the same 400 L1 evaluations for every arm, and every run started only after the previous one finished, so its wall-clock is its own. Sonnet's fleet arm covers one spec, asic_multiaxis_control, by the owner's choice (the spend cap); a full Sonnet arm would mostly have repeated a result two models already show. The first fleet is kept as before the fix; the fixed fleet was re-run for Qwen only, on the same specs and seeds.

The arms are paired (same spec and seed), so the tests are paired: the exact sign test and the paired t-test, both two-sided, recomputed by this site from the run summaries and checked against the repository's table.

Hypervolume: the single graph maps more of the front. Pooled over the first fleets the loss is significant (p = 0.014), and for DeepSeek alone; for Qwen (10 of 25, p = 0.42) and Sonnet (5 pairs) it is not. The fixed fleet against the first: 12 higher, 13 lower. The cold restart did not explain the loss. The repository's reading of the records, not a tested cause: a fixed split by contract class allocates worse than the architect's own early choice of families, and smaller per-member studies cost front coverage.

Regret: a gain with Qwen. The first Qwen fleet picked the better design in 18 of 24 untied pairs (p = 0.023), the fixed one in 22 of 25 (p = 0.0002, mean -8.6 points). Caveat: the fixed fleet ran hours after the single arm it is paired with, and the repository measured large provider drift for Qwen elsewhere, so drift is not ruled out; its regret is lower on all five specs, which a uniform drift would not explain on its own.

Throughput, measured, and what it projects to. Designs evaluated per hour of wall-clock, one run at a time; the three 24-hour columns are projections: the measured rates multiplied by 24 hours of back-to-back runs on one machine, assuming nothing changes (provider latency, rate limits, the spec mix). They were not run.

ASIC specs: the structured graph against the baselines

Five seeds per model on the three ASIC specs, against NSGA-II and random search with the same budget, all scored on the ASIC ground truth (the system spec on simulated feasibility):

Loss: on asic_multiaxis_control nobody gets close: NSGA-II and the three models all land +17.9% to +25.1% above the true winner, whose trick (two more iterations for a narrower angle path and a faster clock) sits in a corner the L1 bound hides. Hypervolume on the power spec is below NSGA-II's for every model (0.754–0.794 against 0.812). And these winners are scored on the model's estimates: the held-out check shows the multiaxis winner itself misses the deadline when measured.

The ASIC target: ground truth on both targets

On asic_low_area all four families are feasible and rank the same on both targets; pipelined costs 4.17× the winner's area on sky130 and 4.99× its LUT+FF on the FPGA. The FPGA-or-ASIC animation steps through the depth sweeps, and the calibration, the held-out check and the deadline miss are there too.

L2 → L1 feedback

The constructed spec l2_feedback_bursty: bursts of 32 requests at 130 MS/s, a sustained throughput floor of 130 MS/s, p99 latency within 0.4 µs. 37,882 designs pass the L1 bound and 447 the simulation. Fifteen searches of 400 evaluations:

The correction recovered a passing design in 3 of 3 (regret +17.3% against the true winner); the same re-exploration budget screened with the uncorrected bound recovered 1. Three cases is thin evidence. The first version of this spec had a 1 MS/s throughput floor, a strawman against a 130 MS/s stream; with it the feedback fired in 6 of 15. The review caught it and the repository kept both results.

What M4 cost

The key's usage rose by $4.15 over milestone 4 ($3.83 for the first pass and $0.32 for the fixed-fleet re-run), under the $5.00 cap measured. That reading is the authoritative one: calls that fail inside the client report no usage. The provider-reported ledger has 147 entries, $3.84; the archived traces hold 941 calls in 314 run directories (one per structured run and one per fleet member's run). The models were checked as listed, with tools and structured outputs and no substitution, before the first call.

Milestone 3 at a glance

• A free-form campaign agent gained nothing over the structured graph, at a higher cost. On the same specs, seeds and budget, its hypervolume and regret, averaged over the specs, were slightly worse for every model, it cost 1.44–2.25× as much per run, Qwen looped in 4 of its 30 campaigns, and in 52 of 65 campaigns its plan came down to one call to the structured graph. It stays experimental, off by default.
• Cross-run memory showed no benefit in the one model and three seeds it was tested on.
• System-level simulation changes the winner where the system is the hard part. On multiaxis_control only the simulation finds the right design (m=4); on bursty_offload a back-of-envelope peak-rate floor finds the same winner, and on dds_sfdr nothing changes, a negative result the repository keeps.
• The structured graph keeps its M2 pattern on the new specs: hypervolume at or above NSGA-II's (one exception, by 0.002) and much lower regret.

The A/B: structured graph against the campaign agent

Same specs, same seeds, same 400 evaluations per spec. The structured arm is the M2 runs on the four M2 specs (replayed through the M3 graph: 60 of 60 identical, so they can be reused) and fresh runs on the two system specs; the campaign arm is one single-spec campaign per spec and seed, memory off, with the same model as campaign agent and inner architect. The Sonnet campaign arm covers one spec, multiaxis_control, by the owner's choice (the full arm would have cost about 6.9 dollars against the 5-dollar cap); DeepSeek and Qwen cover all six.

Wall-clock is not controlled: in that session three or four runs (structured, campaign and memory-on) ran at a time, and the structured runs on the M2 specs come from the M2 session. Hypervolume and regret are the mean of the per-spec means over the feasible specs; the infeasible spec was called correctly by every run of both arms, the campaigns after spending their whole budget re-checking it.

No gain, higher cost. In 52 of the 65 campaigns the agent spent its whole budget on one run_dse(400), so its result is the structured graph's plus sampling noise, and then walked the ladder. Where it deviated, it did not help: on low_area_control seed 0 its notes to the inner architect narrowed the first box, and the run ended at hypervolume 0.097 and +224.5% regret (Qwen3.8-27B, reasoning off). Qwen failed 4 campaigns by re-running verify, synthesize and back-annotate until the 40-call cap (each had already used its 400 evaluations, so each is still scored). The campaigns read 3.5×–8.9× as many input tokens per run. And the ladder's tools rarely had anything to say: most selections have W ≥ 15, which has no recorded RTL or synthesis data and is too large for a fresh exhaustive run inside a campaign.

Per spec, the same comparison:

Memory on vs off

The campaign agent can carry lessons from one spec to the next: code writes them after every spec (per family, per spec class, calibration history) and the agent can add notes of its own. The test: Qwen3.8-27B, reasoning off on a fixed sequence of four specs, 3 seeds, memory on against the campaign arm's memory-off runs.

No benefit is visible. The first spec starts with an empty store in both conditions, so its large gap (0.564 → 0.915) is run-to-run noise, which also shows how wide the noise is at three seeds. Later in the sequence memory-on is worse on hypervolume for dds_250msps and multiaxis_control. The lessons themselves were sensible ("unrolled_k: explored in 3 specs, on the front in 0"). The agent's own notes also copied numbers from tool outputs, once inconsistently; since the review, every number in LLM-written text is masked before it is stored or passed on (the live runs predate that). One model, three seeds: weak evidence either way.

System specs: what the simulation changes

Three new specs constrain the system, not just the unit: a batch deadline, a p99 latency under bursts, a DDS that must never stall with a spurious-free dynamic range of at least 90 dBc. Each has an exhaustive ground truth, simulated for every one of the 635,040 designs, and three more views of it: what the L1 bound sees, the spec read in milestone-2 terms (a throughput floor at the system's average rate), and, recomputed for this site, a floor at the peak rate the system needs.

• multiaxis_control: the simulation is needed. The average-rate view picks m=7, the L1 bound and a peak-rate floor both pick m=5, and m=5 misses the simulated 0.44 µs deadline by 1.5%, because a tick lands between clock edges and up to a cycle passes before the first accept. Only the simulation gets to m=4 (51,244 designs pass it, 53,667 pass the bound). The replay on How it works shows it request by request.
• bursty_offload: average-rate sizing is wrong, but simulation is not needed. In milestone-2 terms the winner is the small FSM design that wins low_area_control (251 LUT+FF), but it takes one request every 15 cycles, so the eighth request of a burst waits for the other seven: its simulated p99 is 1.18 µs, 2× even its bound. The winner is a pipelined_m at 3× the area. But sizing for the burst (8 requests in 0.4 µs, 20 MSPS) gives the same winner and the same 151,528 designs.
• dds_sfdr: a negative result. For this design space the DDS's system constraints reduce to the throughput-only view: the throughput-only winner already has 100.3 dBc spurious-free dynamic range, and "no stall" is "throughput ≥ the sample rate". The repository keeps it as evidence and does not evaluate agents on it.

The structured graph on the system specs

Five fresh runs per model on the two evaluated system specs. The baselines get the same L1 screening and the same L2 step at the end, and every run is scored on simulated feasibility against the exhaustive ground truth.

Every selected design met the spec in simulation. The agents' hypervolume is 0.838–0.869 against NSGA-II's 0.806–0.840 (the one cell below NSGA-II: Qwen on bursty_offload), and their regret is +3.2% to +15.8% against NSGA-II's +16.0% and +26.7%. L2 changed the L1 selection in 0 of the 30 structured runs: their L1 picks already passed the simulation. It did change it in 4 campaign runs, all on multiaxis_control, where the L1 pick was the m=5 design.

The map_front fix (offline)

Milestone 2's worst run (Qwen, high_precision, one seed) came from a blind spot: front-mapping only revisited families already on the front. Milestone 3 gives a family that was explored without a feasible design a share of the mapping budget, gated on its own designs having met the structural constraints. Tested offline on the 60 recorded M2 decisions:

The unconditional version spends budget on families that can never reach 250 MSPS. The gated version's high_precision gain (+8.0% → +4.0% regret) comes from that single run, and it lowers hypervolume on four low_area_control replays without changing a selection; the gate was also refined on these same replays. By the owner's decision it is the default only for the system specs, so the M2 results stand as recorded.

What M3 cost

The key's usage rose by $3.58 over the M3 session, under the $5.00 cap; that reading is the authoritative one, because calls that fail inside the client report no usage. The provider-reported ledger has 114 entries, $3.49: structured $1.19, campaign $2.30 including the memory runs and the pilots. The models were checked as listed, with tools and structured outputs and no substitution, at the start of the session.

M1 → M2

Loading the animation…

The front-mapping gap is closed. Every M2 model now covers 0.96–1.09× NSGA-II's hypervolume on every feasible spec, where in M1 the agents reached 0.18–0.87× on the two specs they lost. And the selection lead mostly survives: mean regret is below NSGA-II's in 8 of the 9 model-and-spec cells.

Where it got worse

Plainly, first.

• Qwen on high_precision lost to NSGA-II on selection: +17.5% mean regret (± 27.2%) against NSGA-II's +10.3%, up from +7.5% in M1. One seed accounts for it: its first box for pipelined_m was too narrow to reach the accuracy target, the front became a single pipelined design, and front-mapping only revisits the families on the front, so the winner's family never came back. That seed selected a design +71.7% from the optimum; the median of its five seeds is +5.6%. The same seed pulls its high_precision hypervolume below NSGA-II's (the one cell under 1.0 above). Milestone 3 built a fix and tested it offline on these recorded runs (above); by the owner's decision it is the default only for the new system specs, so this result stands as recorded.
• DeepSeek's selection got worse: +2.2% → +7.6% on dds_250msps and +1.6% → +5.1% on low_area_control. It chose map_front in 13 of its 15 feasible-spec runs (Qwen 7, Sonnet 4), spending its rounds on coverage rather than on the corner. It still beats NSGA-II on both.
• The agent now always uses the whole budget on feasible specs, 400 evaluations like the baselines, by design: the reserve spends what is left on mapping. M1's "fewer evaluations" came with its narrow fronts.
• Five seeds is still a small sample: several cells overlap within one standard deviation (DeepSeek on low_area_control, for one).

Where it held

The infeasible spec is still caught every time: every model on every seed declared it infeasible itself (15 of 15 runs), naming throughput as the binding constraint, usually after 100–200 evaluations. The baselines can only report that they found nothing.

Pareto replay

Pick a run, M2 or M1, and watch it explore round by round: what the LLM asked for, what code found, the front so far against the true front, and the LLM's verdict on each round in its own words. M2's front-mapping rounds are marked; the last one is code's alone.

Loading the animation…

The hypervolume race

How quickly each method covers the true front, evaluation by evaluation. Switch to M1 to see the agents' lines go flat when their runs stopped; in M2 they keep climbing to the end of the budget.

Loading the animation…

Every spec, every method (M2)

The selected design of every seed, its regret, the front coverage, and what the LLM cost on that spec: tokens, dollars per run and dollars per evaluated design. The baselines use no LLM.

dds_250msps: NCO / DDS sin-cos generator for a digital up-converter. One sample per clock at >= 250 MSPS, max error <= 2^-13. Minimise LUTs. True optimum on luts: pipelined W=18 N=15.

low_area_control: Field-oriented motor-control loop needing sin/cos of the rotor angle at >= 1 MSPS with max error <= 2^-10. Area is everything: minimise LUTs + FFs. True optimum on luts_plus_ffs: iterative W=15 N=12.

high_precision: Coherent demodulator for a precision instrument: max error <= 2^-20 at >= 50 MSPS. Minimise area (LUTs + FFs) and the relative power index. True optimum on luts_plus_ffs: pipelined_m W=26 N=22 m=6.

infeasible_dds_400msps: Wideband DDS needing >= 400 MSPS single-lane sin/cos with max error <= 2^-12. Minimise LUTs. No design meets it: the fastest of 635,040 reaches 292 MSPSestimate.

What the LLM cost, by model (M2)

A complete exploration of one spec cost between $0.0073 and $0.064 per run, and five seeds of one spec between $0.036 (Qwen, reasoning off) and $0.32 (Sonnet). For M2, three ways of counting the spend agree to within a cent; M1's provider-reported figure undercounted the key:

Cost and labour

The architecture lets several agents run round the clock, each evaluating a real design decision. What that costs is a projection, and the calculator builds it only from the measured M2 runs above. The engineer-time side is yours to fill in: there are no salary assumptions here.

Loading the calculator…

Milestone 1, archived

M1's runs are kept as they were: three seeds, four model configurations (Qwen3.8-27B also with its default reasoning), 48 runs, $1.74 provider-reported. NSGA-II mapped the front better on two of the three feasible specs (hypervolume 0.800 against the agents' 0.256–0.695 on dds_250msps, 0.905 against 0.164–0.238 on low_area_control), while 11 of the 12 model-and-spec pairs beat NSGA-II's mean regret. With reasoning off, Qwen was about 7.8× cheaper per run and needed 15× less wall-clock; with its default reasoning it failed 18 of its 45 calls, 17 of them by reaching the output-token limit. M2 ran Qwen with reasoning off only.

dds_250msps: NCO / DDS sin-cos generator for a digital up-converter. One sample per clock at >= 250 MSPS, max error <= 2^-13. Minimise LUTs. True optimum on luts: pipelined W=18 N=15.

low_area_control: Field-oriented motor-control loop needing sin/cos of the rotor angle at >= 1 MSPS with max error <= 2^-10. Area is everything: minimise LUTs + FFs. True optimum on luts_plus_ffs: iterative W=15 N=12.

high_precision: Coherent demodulator for a precision instrument: max error <= 2^-20 at >= 50 MSPS. Minimise area (LUTs + FFs) and the relative power index. True optimum on luts_plus_ffs: pipelined_m W=26 N=22 m=6.

infeasible_dds_400msps: Wideband DDS needing >= 400 MSPS single-lane sin/cos with max error <= 2^-12. Minimise LUTs. No design meets it: the fastest of 635,040 reaches 292 MSPSestimate.

The repository's own tables are in eval/results.md; every row this page shows is checked against it when the site's data is built.