HW Design Space Agent: final technical report (M1–M4)

Can a language model make hardware design-space exploration better, if it is never allowed to produce a number? The running problem is a CORDIC sin/cos unit with four architecture families (iterative, unrolled_k, pipelined, pipelined_m) and numeric knobs (data width, iterations, angle-path width, guard bits, rounding, and the unroll or pipelining factor): 635,040 designs, small enough to enumerate, so every method can be scored against the exact answer. A spec gives constraints (throughput, accuracy, from M3 system timing), one to three objectives, a selection rule and a budget of 400 evaluations.

The rule held from the first commit to the last: the LLM is never the optimiser and never produces a PPA, accuracy or system-level number. It plays the architect: it reads the spec, picks families and ranges from a fixed registry, reads code-written summaries, and takes one of a few typed decisions (refine, widen, add a family, map the front, declare infeasible, stop). Optuna's NSGA-II searches inside its boxes; a bit-accurate golden model gives exact accuracy; cost models give estimates; synthesis gives measurements; simulation gives system metrics. Every number carries its provenance: exact, estimate, measured (tool, version) or simulated (model, version). The structured-output schemas have no field where a number about a design could go.

Two scores recur. HV fraction: the hypervolume of the feasible designs a run found over the true hypervolume of the exhaustive grid. Selection regret: how much worse the design the run finally selects is than the true optimum on the spec's selection metric. Baselines are NSGA-II and random search over the whole space with the same budget and the same evaluation function.

Each rung is code; the LLM sits only in the L1 loop. The ladder was built bottom-up in evidence: the golden model is bit-exact against the reference RTL on all 65,536 input angles of the reference configuration; the generated RTL matched the golden model in 47 configurations across two simulators (94 runs, 0 mismatches) and at gate level (22 runs, 0 mismatches), with 7 of 7 formal proofs; L4 has 39 open-source FPGA points, 14 Vivado designs and, in M4, 57 ASIC calibration points plus 9 held-out ones.

Loss first. The M1 agent mapped the Pareto front far worse than plain NSGA-II on two of three feasible specs: its HV fraction relative to NSGA-II was 0.32, 0.87 and 0.46 on dds_250msps and 0.23, 0.22 and 0.18 on low_area_control for Sonnet 5.5, DeepSeek V4.1 Flash and Qwen3.8-27B (reasoning off). It dived at the corner the selection rule picks and the 1% hypervolume-gain rule then stopped it with a narrow front.

What worked. It selected better. Mean selection regret was below NSGA-II's on every model and spec except one (Qwen on low_area_control, +12.2% against NSGA-II's +8.0%), and every model declared the infeasible 400 MSPS spec infeasible itself in every run, using 100–200 evaluations where the baselines spend all 400 and simply find nothing. The cost model behind all of this was weak by construction: five constants solved from two Vivado anchors.

The levers. M2 added deterministic whole-curve levers: a 40% reserve the LLM's own rounds may not spend once something is feasible, a front-mapping round that code runs with whatever is left, and a map_front decision the LLM may choose. They were tuned offline by replaying the recorded M1 decisions, without spending anything. Live, the front-mapping gap closed: agent HV relative to NSGA-II went to 1.02, 1.08 and 1.09 on dds_250msps, 0.97, 1.02 and 0.99 on low_area_control and 1.04, 1.05 and 0.96 on high_precision.

Losses. The selection lead mostly survived but not everywhere: DeepSeek's regret rose on dds_250msps (+2.2% to +7.6%) and low_area_control (+1.6% to +5.1%), and Qwen on high_precision reached +17.5% ± 27.2 on the strength of one seed whose first box was too narrow to reach the accuracy target; the front-mapping round then never revisited that family.

The measurements. The open-source flow (Yosys 0.68 + nextpnr-xilinx on the anchors' part) and, later, Vivado 2025.2 on 14 generated designs showed what the two-anchor model gets wrong. Its known biases against Vivado are pipelined Fmax low by 15–17%, iterative Fmax high by 14–16%, and unrolled_k LUTs low by 37–40% at k = 3–4. A refit on 51 points was no better on unseen Vivado points than the untouched M1 model, so the default never changed, and no spec's ground-truth winner moved under the refit.

L2. A cycle-accurate model of each family's interface contract matched the generated RTL on 48/48 bursty traces (28,424 clock edges). SimPy models put the unit in a DDS, a control loop and a bursty stream. On multiaxis_control only the simulation finds the winner: 53,667 designs pass the L1 bound, 51,244 the simulation, and the bound's winner misses the batch deadline. On bursty_offload a peak-rate floor would have done as well, and the DDS spec reduced to the throughput-only view. In the live runs L2 changed the L1 selection in 0 of 30 structured runs and in 4 of 65 campaign runs, all on multiaxis_control.

The campaign agent (loss). An outer agent on LangChain Deep Agents could call the structured graph, single NSGA-II studies, the L2–L5 rungs and a memory. In 52 of 65 campaigns its plan came down to one 400-evaluation call to the structured graph, and it cost 1.44 (Qwen), 1.62 (DeepSeek) and 2.25 (Sonnet) times as much per spec. Qwen failed 4 of 30 campaigns by looping on the ladder tools. Its free-text notes to the inner architect narrowed the first box in the worst run. Memory on vs off showed no benefit at three seeds. The campaign layer stayed experimental and off by default.

An ASIC target on sky130

The flow. The same generated RTL goes through Yosys 0.33 and OpenSTA 3.1.0 against the sky130_fd_sc_hd library at its typical corner (25 °C, 1.80 V), with a 10 ns clock: cell area, flip-flops, the register-to-register path with the library's wire-load model, and vectorless power. There is no placement: these are post-synthesis numbers, the ASIC counterpart of the Vivado anchors, not of nextpnr's routed timing. OpenROAD was not installed: it has no apt package for this Ubuntu, its conda build installs only through a commercial channel, there is no Docker daemon for the container, and it would need a floorplan and power grid per design, so place and route stayed out of scope.

The model. The FPGA model's structural counts, re-weighted for standard cells and fitted to 57 synthesised designs. Leave-one-out RMS error: area 8.7%, flip-flops 4.6%, Fmax 10.6%, power against OpenSTA 5.8%. Per family, the worst area error is unrolled_k's (12.8%) and the worst Fmax error pipelined's (12.9%, with unrolled_k at 12.5%). One adder constant for every family fitted badly (14.8% on area); the FSM families needed their own. The fitted adder delay is linear in width, about 0.29 ns per bit: Yosys's default adder mapping ripples, which makes chained rotations expensive on this target.

Does the family ranking differ from the FPGA? No, on thin evidence. Three ASIC specs, each with an FPGA twin (the same constraints with area in LUTs + FFs, on the FPGA ground truth), rank the families in the same order on both targets. Only asic_low_area ranks all four families; on the other two only the pipelined families meet the rate, so each is a single pairwise comparison (pipelined_m ahead of pipelined). What moves is the best configuration inside a family. At 50 MSPS the ASIC winner is pipelined_m with m=3 where the FPGA twin's is m=7; under the 0.5 µs batch deadline of asic_multiaxis_control it is m=2 where the FPGA's is m=5. The slow ripple adders force shallower pipelining. The gaps shrink too: pipelined costs 4.17× the iterative winner's area on asic_low_area and 4.99× its LUT+FF on the FPGA twin. Measured directly, pipelined is 4.77× iterative in cell area at the anchors' W=16, N=14.

L2 carried over unchanged (it reads only the contract and Fmax) and on asic_multiaxis_control it changes the winner: 8,533 designs pass the bound, 7,898 the simulation.

The held-out check: where the ASIC model is wrong

Loss first. Nine ground-truth designs that the fit never saw (each spec's per-family optima, the FPGA twins' optima, the L1-bound pick) were synthesised. Area is within 5% on all nine; Fmax is not, from −14.9% to +39.6%, the worst an unrolled_k design. Two consequences. The FPGA twin's m=7 winner for asic_power_50msps runs at 26.7 MHz on sky130 and cannot make 50 MSPS. And the model's winner for asic_multiaxis_control measures 72.2 MHz against an estimate of 82.5, and at that clock its simulated p99 batch latency is 0.567 µs against the 0.5 µs deadline; of the nine, only plain pipelined W=17, N=14 (119.1 MHz) meets that spec with measured numbers. The ASIC back-annotation node flags exactly this case, but it does not re-explore: there is no ASIC L5 loop yet, and the nine held-out designs were kept out of the fit so that they remain a test.

ASIC specs: the structured graph against the baselines

Five seeds of the structured graph per model on the three ASIC specs, against NSGA-II and random search with the same budget (all scored on the ASIC ground truth, the system spec on simulated feasibility). Every selected design met its spec (15/15 per model).

The M2 pattern carries to the new target where the selection rule has a corner to aim at: on the power-driven spec every model's regret is a fraction of NSGA-II's (+54.0% ± 36.1): the architects narrow onto the pipelining depth that the selection rule rewards, where NSGA-II's union search keeps sampling every family. HV is lower than NSGA-II's on that spec (0.754–0.794 against 0.812), level with it on asic_low_area for Sonnet and DeepSeek (0.925 and 0.923 against 0.922, Qwen 0.859) and between 0.820 and 0.864 against 0.837 on asic_multiaxis_control. Loss: on asic_multiaxis_control nobody gets close: NSGA-II and the three models all land 17.9–25.1% above the true winner, whose trick (two more iterations for a narrower angle path and a faster clock) sits in a corner the bound hides. L2 changed the L1 pick in 2 Sonnet runs.

The fleet

M3's lesson was that an LLM in the outer loop adds cost and loops without adding search. M4 tried the opposite: several structured graphs in parallel with code deciding everything between them. Two members, one per contract class (FSM families and pipelined families), each an unmodified structured graph restricted to its families. Half the budget is split equally; code computes per-family facts from the evaluations the members write to a shared LangGraph Store and gives the other half to the members in proportion to their share of the merged Pareto front; code merges, selects and runs L2 once. No LLM-written text reaches the store or another member. No LLM coordinator was built: the coordinator's decisions are arithmetic over numbers code already has, and M3 showed what an LLM in that seat costs.

A fix after review. In the first A/B the Store was only written: the coordinator read it, but the phase-2 members started cold, and a member budget of 10 could not be fully spent (one run used 396 of 400 evaluations). The fleet now seeds every phase-2 study with the fleet's stored front designs of its family, read back from the Store, and a member gets at least 20 evaluations. The first fleet's results are kept as before_fix; the fixed fleet was re-run for Qwen only (the spend cap), on the same specs and seeds.

The A/B (same five specs, three ASIC and two FPGA, seeds 0–4, 400 L1 evaluations for every arm, every run started alone so its wall-clock is its own; Sonnet's fleet arm on asic_multiaxis_control only, for the spend cap):

The arms are paired (same spec and seed), so the tests are paired: the exact sign test and the paired t-test, both two-sided.

HV: the single graph maps more of the front; significant for DeepSeek and pooled, not for Qwen or Sonnet. Pooled over the three models, the first fleet had the higher HV in 18 of 55 pairs (sign test p = 0.014). DeepSeek alone: 6 of 25 (p = 0.015; paired t = −3.03, p = 0.006). Qwen alone: 10 of 25 (p = 0.42), not significant; Sonnet, 2 of 5 pairs, cannot tell. The fixed fleet did not change this: 8 of 25 against the single graph (p = 0.11), and against the first fleet 12 higher, 13 lower (p = 1.0). The cold restart did not explain the HV loss.

Regret: a gain for the fleet with Qwen. Qwen's first fleet selected the better design in 18 of 24 untied pairs (p = 0.023); the fixed fleet in 22 of 25 (p = 0.0002), with a mean paired difference of −8.6 points. Against the first fleet the fix changed regret by −5.4 points on average, not significantly (p = 0.21). DeepSeek and Sonnet show no regret difference.

Cost and time. The fleet costs 1.41 (Qwen), 1.54 (DeepSeek) and 1.60 (Sonnet) times as much per spec. Wall-clock is noisy (provider latency dominates; the arms ran at different times); the first fleet was slower for every model, the fixed one less so. No run in any arm failed to produce a spec-meeting design; the first fleet had more failed (retried) LLM calls, 17 against 3 over the Qwen and DeepSeek arms, and the fixed Qwen fleet had 0 (against 7 before the fix).

A confound. The fixed fleet ran hours after the single arm it is paired with, so provider drift (see below: large for Qwen) may affect the quality comparison as well as the wall-clock. Its regret is lower than the single graph's on every one of the five specs, which a uniform drift would not explain on its own, but the comparison is not same-session.

What the records show, and what they do not. The fleet spends more of its budget on the FSM families than a single graph does: on average 180 of 400 evaluations for the Qwen and DeepSeek fleets against 128 (Qwen) and 166 (DeepSeek) for their single graphs, and 100 against 18 for Sonnet on asic_multiaxis_control, where no FSM design can meet the deadline. Its FSM member often proves that the hard way (Qwen's declared its families infeasible 16 times in 25 runs), and phase 2 then went entirely to the pipelined member in 14 (Qwen), 15 (DeepSeek) and 5 (Sonnet) runs, and 15 for the fixed Qwen fleet. The HV loss is largest on the system specs (multiaxis_control, Qwen: 0.840 single, 0.734 before the fix, 0.745 after). One reading is that a fixed split by contract class allocates worse than the architect's own early choice of families plus one front-mapping round over all of them, and that smaller studies per member (fewer NSGA-II generations each) cost front coverage; the regret gain would fit a pipelined member that searches only the families a selection rule usually picks from. These are readings of the records, not tested causes.

Throughput, measured (designs evaluated per hour of wall-clock, one run at a time): single 25,212 (Qwen), 9,410 (DeepSeek), 35,002 (Sonnet, one spec); first fleet 8,400, 6,292 and 29,256; fixed Qwen fleet 16,094. Projected, not run: one such machine running Qwen single-graph runs back to back would evaluate 605,084 designs and close 1513 specs a day for $13.64, if provider latency, rate limits and the spec mix stayed as measured.

L2 → L1 feedback

When L2 rejects every L1-feasible design of a run, the run spent its budget where the bound was wrong. The feedback measures the bound's error on every simulated design, fits a monotone correction in the device's utilisation per contract class (still optimistic at every design L2 has seen), and re-explores with a separate budget of 3 × 60 evaluations under the corrected bound.

It never fires on the committed specs: 0 of 92 recorded searches (M3 structured runs rebuilt from their archived evaluations, M3 campaign pools, NSGA-II and random on every system spec) and 0 of 60 live M4 runs found an L1-feasible set that L2 rejected whole. A spec was constructed where it does: bursts of 32 requests at 130 MS/s, a sustained throughput floor of 130 MS/s and p99 latency ≤ 0.4 µs. 37,882 designs pass the bound and 447 pass the simulation, all plain pipelined; the cheaper pipelined_m designs the area objective prefers pass the bound and queue in simulation. Of 15 searches of 400 evaluations (5 each of NSGA-II, random and the rule-based architect) it fired in 3, all random search: NSGA-II and the architect found one of the few passing designs on their own. The correction recovered a passing design in 3 of 3 (regret +17.3% ± 11.0 against the true winner); the ablation, the same re-exploration budget screened with the uncorrected bound, recovered in 1 of 3. That is evidence for the correction over the extra evaluations, on three cases.

(The first version of this spec had a throughput floor of 1 MS/s; against a 130 MS/s stream that was a strawman, and with it the feedback fired in 6 of 15 searches.)

Across four milestones the gains in front mapping came from code: the levers, the L2 shortlist, the correction of the bound. The LLM's contribution that held up in every A/B is the early, typed judgement of which families can meet a spec and where its selection corner is. Giving the LLM a bigger role (the campaign agent) lost. Replacing that judgement with a fixed split by contract class (the fleet) lost front coverage; with Qwen it selected better, a result that deserves a second model before anyone builds on it.

Provider drift is real and large. Qwen on low_area_control, through a graph that replays its M2 runs identically, had +3.7% mean regret in M2 and +22.8% in this session's single arm; comparisons are therefore made only between arms run in the same session.

• Every ground truth is a model's. The FPGA one is the two-anchor M1 model, kept fixed so that four milestones are scored on the same answer; the ASIC one is the fitted sky130 model. The measurements say how far each is off (FPGA Fmax by up to 17% against Vivado; ASIC Fmax 10.6% leave-one-out and up to +39.6% on a held-out design), and specs whose winner sits closer to a constraint than that are only as settled as a measurement makes them.
• Post-synthesis timing on the ASIC, one corner, a wire-load model, no buffering or sizing, no clock tree, no place and route; ripple adders from Yosys's default mapping. ABC maps to a relaxed 10 ns delay target (abc -D 10000) with area recovery, so the Fmax figures are those of an area-oriented netlist, not the fastest one the library allows. OpenSTA reports no clock power (there is no clock tree to power), so the power numbers miss the clock network. A commercial flow would change the absolute numbers and could change the within-family optimum again.
• Power is a toggle-rate model everywhere. On the FPGA the index is now checked against Vivado's report_power on 14 designs, itself vectorless: it ranks them well (Spearman 0.982) but is off by 20.6% RMS post-synthesis, and a per-family activity factor (13.9% leave-one-out) is kept as a separate, non-default calibration. On the ASIC the index reproduces OpenSTA's vectorless power. Both assume uniform activity; neither has seen real switching.
• Small samples. Five seeds per model and spec, two or three models, a handful of specs. Day-to-day provider drift is visible in the data (above), so arms are compared only within one session.
• Single-lane designs only; the 400 MSPS spec is infeasible by construction of the registry.
• The L2 → L1 feedback never fired on the committed specs; its evidence is one constructed spec and offline searches.

1. A routed timing scale and real switching activity on the FPGA. The local Vivado runs are done: high_precision misses 50 MSPS at its own 20 ns clock with m=8 (49.88 MSPS post-synthesis, 48.31 post-route), so the per-family-class path factor, which predicted it feasible (52.5), was rejected by its pre-set rule even though it cut Vivado Fmax leave-one-out from 12.5% to 7.4%. The winner m=6 loses 17% to routing (65.39 → 53.96 MSPS), which no model here sees, and the measured power is vectorless. Next: a post-route timing calibration of its own, and power from simulated switching activity.
2. Place and route on sky130 (OpenROAD, from its prebuilt release packages pinned by URL and SHA-256, or a container image pinned by digest) for the held-out designs, to see whether the ASIC model's Fmax error is the wire-load model or the structure.
3. An ASIC measure-and-re-explore loop: synthesise the selected designs, re-select under the measured numbers, as the FPGA path's L5 loop does. Then decide whether to refit with an unrolled_k chain term, disclosed, keeping the held-out points held out.
4. Multi-lane and polyphase families, so that the 400 MSPS spec has an answer.
5. Treat provider drift as a first-class measurement: pin model snapshots where the provider allows it, and re-run a fixed reference set each session before comparing arms.