Project record: how it was built and why

The project is a LangGraph agent for hardware design-space exploration. Given a hardware spec (function, throughput, accuracy, area/power budget), it explores architecture families and parameters and converges on a Pareto-optimal design, going down a ladder of fidelity: spec → analytical model → system-level and cycle-level simulation → RTL and verification → synthesis → back-annotation.

The first design space is a CORDIC sin/cos unit on an Artix-7 FPGA, chosen because it's small enough to search exhaustively. That gives a true answer to score the agent against, which most agent demos lack.

Where it stands on 2026-10-11:

• The idea. Hardware architecture is trade-off exploration: area against throughput against accuracy against power. An AI agent is the perfect environment for trade-off exploration: it can read a spec, decide which families and ranges are worth searching, interpret results and decide what to do next, around the clock, while deterministic tools do the measuring. It's a way to avoid premature optimisation: explore first, then spend effort on the critical 3% that matters (after Knuth, "Premature optimization is the root of all evil", ACM Computing Surveys 6(4), 1974).
• The showcase. It demonstrates both production software practice (LangGraph, Python, TypeScript, tests, CI, eval harnesses, a deployed site) and deep hardware expertise (bit-exact models, RTL, formal verification, synthesis, calibration against real tool results).

1. The LLM is never the optimiser and never produces a power/performance/area, accuracy or system-level number. It does the architect's job: choose families and ranges, interpret summaries, decide to refine, widen, map the front, declare the spec infeasible, or stop. Code produces every number: a bit-exact model, cost models, Optuna NSGA-II, Pareto/hypervolume maths, simulators and synthesis tools.
2. Every number carries its provenance: exact (bit-accurate simulation), estimate (cost model, naming its calibration, or an L1 analytic bound), measured (tool, version) or simulated (the L2 system model, since M3).
3. Scored against ground truth. An exhaustive grid of 635,040 designs gives the true front and the true optimum per spec. The agent is compared with NSGA-II and random search at the same evaluation budget.
4. Results come only from real LLM runs, with model IDs, tokens and cost recorded. A scripted fake LLM is used only for tests, and the eval harness refuses it.
5. Losses are reported first. The site's Results page opens with where the agent lost.
6. Reproducible. Every run's trace is committed; the results file regenerates from the committed data with no diff; the site vendors the repo at a pinned commit, with hashes, and its TypeScript ports of the models match the Python fixtures exactly.

The project itself is built with AI coding agents (Claude Code), under a deliberate process:

• A planner session reads the state, makes or asks for decisions, writes self-contained briefs, and independently verifies each result before anything merges.
• Executor sessions do the work from a brief:
  • cloud sessions for milestones that need live LLM calls (an OpenRouter key held as an environment variable, never printed, logged, traced or committed);
  • local sub-agents for the site and for Vivado, which is installed locally. Executors open PRs; the planner merges after verification.
• Verification gates before a merge: CI green at the PR head; tests re-run locally; a scan of the full git history and every trace for key material; the results file regenerated with no diff; the traces' cost sum matched to the results; the key's usage checked against the spend cap; numbers in prose spot-checked against the data files; for M2, an independent code review posted on the PR and a Vivado spot-check.
• Spend cap: $5 of LLM usage per milestone.
• Small machine discipline: one heavy job at a time under a memory cap; heavy synthesis and evals run in the cloud where possible.

M1 (merged 0082ccd; 48 live runs, 3 seeds)

• Selection regret (how far the chosen design is from the true optimum on the spec's metric): the agent beat both baselines on every spec (best +1.6% / +3.1% / +1.6% against NSGA-II's +12.3% / +12.4% / +8.0%).
• Hypervolume: NSGA-II won on two of three specs (0.800 and 0.905, against the agents' 0.16–0.70). The agents dived at the spec's corner and a 1% hypervolume-gain stopping rule ended the run early.
• Infeasible spec: every model, on every seed, declared it infeasible and named throughput as the binding constraint (the true maximum is 291.5 MSPS against 400 asked for).
• Cost: $1.74 provider-reported; about $2.50 by the key's usage.

M2 (merged 9060f8f; 60 live runs, 5 seeds)

• L3 verification: generated RTL for all four families matched the bit-exact golden model in 47 configurations × 2 simulators (Verilator and Icarus) with zero mismatches, exhaustive up to 16 bits. 7/7 formal proofs at 8 bits. 22 gate-level simulations of the synthesised netlist, zero mismatches.
• L4 synthesis: 39 points through Yosys + nextpnr-xilinx, every one labelled with tool versions and command lines.
• Hypervolume relative to NSGA-II: 0.97–1.09 on the two specs where M1 lost (M1: 0.18–0.87).
• Selection: the agent still beat NSGA-II in 8 of 9 model/spec cells. It lost on Qwen high_precision (+17.5% ± 27.2 against +10.3%, one bad seed).
• What it cost: DeepSeek's selection got worse (2.2% → 7.6% and 1.6% → 5.1%), and the agent now always uses the full 400-evaluation budget, so M1's "fewer evaluations" advantage is gone.
• Spend: the key's usage rose by $1.66 between the before and after readings (provider-reported $1.67 including the pilot; the two readings' timing cannot fully separate the pilot).

M3 (merged a92fe65; structured arm 30 new runs, campaign arm 65 runs, 5 seeds)

• L2 vs RTL: the cycle-accurate model matched the RTL on 48/48 traces (28,424 clock edges, 8 designs × 3 seeds, Verilator and Icarus).
• M2 comparability: all 60 M2 runs replay identically under M3 code, down to byte-identical architect prompts, so the A/B reuses them.
• System specs (ground truth simulated, exhaustive): on multiaxis_control the system winner is pipelined_m m=4 (1,072 LUT+FF), where the MSPS-only view picks m=7 and even peak-rate sizing picks m=5. On bursty_offload the simulated winner differs from an average-rate view but equals peak-rate sizing, so L2 adds nothing there. A third spec (dds_sfdr) changes nothing and is kept as a negative result.
• Structured agent on the system specs: HV 0.84–0.87 against NSGA-II's 0.81–0.84; regret +3.2% to +15.8% against NSGA-II's +16.0% / +26.7%.
• A/B, structured → campaign (pooled): DeepSeek HV 0.905 → 0.893, regret +7.3% → +7.7%, cost 1.62×; Qwen HV 0.872 → 0.819, regret +9.2% → +19.0%, cost 1.44×, 4/30 campaigns failed by looping; Sonnet (one spec) HV 0.839 → 0.833, regret +3.2% → +5.7%, cost 2.25×. In 52 of 65 runs the campaign agent's plan came down to a single 400-evaluation call to the structured graph.
• Memory on vs off (Qwen, 3 seeds): no benefit; the run-to-run noise is as large as any effect.
• Spend: $3.58 by the key's usage (ledger $3.49), under the $5 cap; the before reading preceded the first run.

M4 (merged b119c8a; ASIC eval 45 runs, fleet A/B 55 + 55 paired runs + 25 re-run, 5 seeds)

• ASIC flow (measured): Yosys 0.33 + OpenSTA 3.1.0 on sky130_fd_sc_hd (TT, 25 °C, 1.80 V, 10 ns), 57 fitted points across all four families plus 9 held out; CI re-measures all 57 identically.
• ASIC model (estimate): leave-one-out error 8.7% area, 4.6% FFs, 10.6% Fmax, 5.8% power index. Held out: area within 5% on 9/9, Fmax −14.9% to +39.6% (optimistic by more than 1% on 5 of 9).
• ASIC specs (exhaustive ground truth): the family ranking matches the FPGA's on all three, but pipelining is shallower (m=3 vs m=7; m=2 vs m=5). On asic_multiaxis_control L2 changes the winner (m=2 W17 N16) — and that winner misses its deadline at its measured 72.2 MHz (p99 0.567 µs against 0.5), which the model did not predict.
• ASIC eval (regret, low area / power / multiaxis): Sonnet +0.8 / +3.5 / +17.9%; NSGA-II +6.1 / +54.0 / +23.4%; 45/45 selections meet spec.
• Fleet vs single graph (same specs, seeds, 400 evaluations; runs not overlapping): the fleet maps the front worse (pooled HV loss, sign test p = 0.014; significant for DeepSeek). After the store and budget fixes, Qwen's fleet picks better designs: regret +6.3% against +14.8%, lower in 22 of 25 pairs (p = 0.0002), with HV still no better, so the cold restart did not explain the HV loss. Caveat: the re-run was hours after the single arm.
• Throughput (measured; 24 h figures labelled projections): 16,094 designs/hour for the fixed Qwen fleet against 25,212 for one graph.
• L2→L1 feedback: never fires on the committed specs (0/92 recorded, 0/60 live); on a constructed spec it fires in 3 of 15 searches and recovers 3 of 3, against 1 of 3 without it.
• FPGA path untouched: all 60 M2 and 30 M3 runs replay identically.
• Spend: $4.15 by the key ($3.83 first pass + $0.32 fleet re-run), under the $5 cap.

Vivado M4 runs + refit (merged 80415c6, PR #7; local Vivado 2025.2, no LLM spend)

• high_precision at its own 20 ns clock (measured): m=8 49.88 MSPS post-synthesis (WNS −0.05 ns), 48.31 post-route; m=6 65.39 and 53.96. m=8 misses 50 MSPS in both views; the winner stays m=6.
• Path factor (rejected by the pre-set rule): one factor per family class would cut Vivado Fmax leave-one-out from 12.5% to 7.4%, but it predicts m=8 at 52.51 MSPS (+5.3%), i.e. feasible, which the measurement contradicts.
• Refit: counted once per design it is the vivado-2025.2 fit (51 points; Vivado leave-one-out LUT 15.4%, FF 3.7%, Fmax 12.5%; M1 15.4 / 3.1 / 11.7%). No spec's winner changes; M1 stays the default.
• First measured FPGA power (Vivado report_power, vectorless, Low confidence): the power index ranks the 14 designs well (Spearman 0.982 post-synthesis, 0.969 post-route) with 20.6% / 36.3% log-RMS error; it reads unrolled_k low (0.61–0.97× post-synthesis, 0.48–0.85× post-route). A per-family activity factor gets 10.2% in-sample, 13.9% leave-one-out (non-default).
• Routed vs synthesised Fmax: 0.675× (pipelined) to 1.056× (unrolled_k); every model throughput is an unrouted figure.

An independent review (no high-severity bugs; 3 medium, 5 low; posted on the PR) found:

• the final front-mapping round can exceed the spec's per-round and round-count limits (the 400 total is respected);
• the per-family recalibration overfits (leave-one-out error 12.9% against 9.0% in-sample on LUTs), and its unweighted fit lets 37 open-source points outvote the 2 Vivado anchors;
• a Vivado spot-check of the generated pipelined design: 720 LUT / 751 FF / 327.4 MHz against the reference RTL's 745 / 784 / 272.9. Generated and reference RTL agree on area but not on timing, which the Vivado anchor work must account for;
• prose claims to correct: two marked wrong (the dense-sweep angle count above W = 16, and an unrolled_k conclusion drawn from one comparison) and three to soften.

All were fixed on the PR. M2 itself was merged before the planner's re-check of the fixes (a check after the merge found nothing wrong); from the Vivado refit on, every PR has been verified before merge.

Two planner review rounds before merge, both posted on the PR (round 1, round 2). Round 1: 1 blocking, 6 should-fix, 12 nits.

• Blocking: L2 shortlisted only from the L1 front, so it could not reach the multiaxis_control winner; its test passed only because it hand-built a front. Fixed (all L1-feasible designs), with a regression test on a real front.
• Campaign tests were skipped in CI (dependency missing); now required on Python 3.11+.
• Overclaims corrected: L2 did re-select in 4/65 campaign runs; the campaign's "whole ladder" mostly had no RTL or synthesis data; bursty_offload's winner change holds only against an average-rate view.
• LLM-written memory notes carried self-computed numbers; now masked.
• The cost table compared different spec sets for Sonnet; fixed.

Round 2 found everything fixed and 2 optional nits, which the executor also fixed before merge.

Two planner review rounds before merge, both posted on the PR (round 1, round 2). Round 1: no blocking, 7 should-fix, 15 nits; every data claim reproduced.

• The write-up's number test let 59% of deliberately altered numbers pass; it now binds every number in order to a named fact, and all 658 mutants fail. The tighter test found two real errors, both fixed.
• The A/B wording overclaimed the HV loss and called Qwen's regret gain "mixed"; paired sign and t tests now set it.
• The fleet's shared store was never read and a 10-evaluation member budget could not be spent; fixed and re-run.
• The Vivado power parser double-counted core power; the constructed feedback spec used a 1 MS/s floor against a 130 MS/s stream; three doc slips. All fixed.

Round 2: one doc fix (a ruled-out refit still listed as planned), fixed before merge.

Two planner rounds, both posted on PR #7. Round 1: no blocking, 1 should-fix, 3 nits; every output regenerated byte-identically.

• The two 20 ns post-synthesis rows duplicated the 10 ns rows and escaped the duplicate filter by their tool label, so the refit counted two designs twice and its leave-one-out looked better than it was. Now counted once, in code, with a test.
• A cached calibration made the first decision run score a refit with stale constants and print "adopted"; the cache is cleared after a refit, the test now fails on the bug itself, and the history is recorded.

Round 2: verified, all items fixed. The path factor's rejection held under both counts.

• One design space (CORDIC); one FPGA family and one ASIC library (sky130, typical corner, post-synthesis only — no place and route).
• The default cost model is still M1 (two Vivado anchors), by choice: the Vivado-weighted refit on 14 measured points did not predict unseen points better, so it is reported, not adopted. Its error is about 15% RMS on LUTs and 12% on Fmax.
• Vivado post-synthesis Fmax is the cost model's timing scale; routed Fmax is lower, by up to about a third on the short-path pipelined designs.
• The power figure is a relative index, not watts. It is now checked against Vivado's report_power, which is itself a vectorless estimate, not a measurement of real switching.
• Natural-language spec intake has only been tested with a scripted model.
• 5 seeds is still a small sample; some cells overlap within one standard deviation.
• The A/Bs are incomplete for the frontier model: Sonnet's campaign arm (M3) and fleet arm (M4) each cover one spec.
• The ASIC model's Fmax is optimistic by more than 1% on 5 of 9 held-out points (by 0.7% on a sixth), and the asic_multiaxis_control winner fails its deadline at its measured clock.
• The fixed fleet ran hours after the single-graph arm it is compared with, so provider drift is not ruled out.
• M3's wall-clock figures were measured with 3–4 runs in parallel (M4's were not).
• System-level metrics come from a SimPy model, not from hardware: its interface contract is checked against the RTL cycle by cycle, but it runs each design at its estimated Fmax, and it aligns requests to the next clock edge with no synchroniser (a real clock-domain crossing would add about two cycles).
• Only one of the two evaluated system specs needs the system model (multiaxis_control, by a 1.5% margin); on bursty_offload a peak-rate throughput floor gives the same winner.

1. M2 done (9060f8f); Vivado anchors and the L5 refit done (bbbf71e, PR #4): 14 generated designs in Vivado 2025.2, no spec's winner changes, M1 stays the default cost model.
2. Site part C1 live 2026-10-09 (ad6101a): L3, L4, gate level and L5 animated from real data; this record.
3. M3 done (a92fe65): SimPy system-level and cycle-level simulation; the campaign agent and its A/B; cross-run memory.
4. Site part C2 live 2026-10-10 (d195189): L2 and the A/B on the showcase site; this record synced.
5. M4 done (b119c8a): ASIC cost model on sky130, a code-orchestrated fleet and its A/B, L2→L1 feedback, the write-up (docs/WRITEUP.md).
6. Site part C3 live 2026-10-11 (82e2a33): the ASIC target, the fleet A/B and the write-up on the showcase site.
7. Local Vivado run done (80415c6, PR #7): m=8 misses at 20 ns, the class path factor rejected, first measured FPGA power.
8. Site part C4 (this version): a "For a hardware team" page (/adoption: pros and cons against general-purpose agents, a proposed deployment, improvements) and the PR #7 numbers; this record synced.
9. After M4: an ASIC measure-and-re-explore loop; OpenROAD place and route from pinned binaries.