Attending Slush 2026 in Helsinki?Join our Frontier AI for Real-World Engineering event

Making AI possible for real-world engineering work

ConstraintBench releasesRelease date
[2]v1.2 · 50 tasks, 11 modelsNewNov 2026
[1]v1.1 · 11 models, FW-01 grader fixOct 2026
[0]v1.0 · results, limited previewOct 2026

Today’s frontier models build web apps in seconds, but struggle in safety-critical embedded environments

About us

We build custom evals, data sets, and refine data for the embedded engineering domain.

100,000+

Domain engineers registered

Our background

We come from the automotive industry, where we have helped the world’s largest carmakers build low-level chip integrations & implement AI in engineering processes.

Selected clients
Member of

Data and evaluations for frontier labs & enterprises

[ 01 ] Custom evals and data

Evals and data built to your capability target

TASK · EXPERT-WRITTENSet how long the firmware waits for the regulator on this board to settle.GOLDEN ANSWER · HIDDEN
GRADERRESULT
Uses the board’s own test data
Wait within 5% of the golden answer
Startup order kept
Code builds and runs
Reasoning explains the choice
4 OF 5 CRITERIA MET
  • Tasks written by domain experts
  • Golden answers and deterministic graders
  • Quality review on every task
  • Delivered in your format
[ 02 ] Off-the-shelf data

Ready-to-use data sets and reinforcement learning environments

AGENT
ENVIRONMENT
GRADER
REWARD
REWARD BY TRAINING ROUND
BOARDTOOLCHAINGRADER
  • Environments with graders built in
  • Areas picked from our model failure research
  • Licensed and ready to train on
  • Improved after delivery
[ 03 ] Operational engineering data

Real engineering work from engineering firms

DESIGN REVIEW · PAST PROJECTReviewed with Anna Virtanen for Client X. The brake signal arrives late on prototype P-17, so the timeout moves from 20 ms to 35 ms. Option B was dropped because the bus load passed 70% on test track 3.ANONYMISED
REQUIREMENTBrake signal timeout
DECISION20 ms to 35 ms
REASONLate signal on a prototype
REJECTED OPTIONBus load over 70%
  • Specifications, tickets and source code
  • Design reviews and the reasons behind them
  • Rebuilt and anonymised
  • Work models have never seen
[ 04 ] Computer-use dataNEW

Engineers and agents working in real engineering software

PCB Editor
STEPSACTIONSKEYSVOICE
  • Screen video at 30–60 fps
  • Keys, clicks and window events, on one clock
  • Reasoning, spoken or from the agent
  • Task files and graded outcomes
[ 05 ] Harness engineering

Guardrails for the AI inside your engineering toolchain

ENGINEER“Set the radar speed scaling from the roller check”
1# radar speed scaling
2check = 101.2 # true 100.0
3scale = 0.01 # sheet value
3scale = 0.01 * 100.0 / 101.2
checker.pyrunning ···fail · 1.2% speed errorrunning ···pass · all rules
BACK TO THE ENGINEERChecked and fixed, ready to review✓ all rules pass
  • Private evals of models in your environment
  • Rules and checkers in your editor and pipeline
  • Fewer mistakes in AI-written code
  • Your own post-trained model

Select a product

Where real-world engineering fails

Today's frontier models are able to code almost any system, but fail on basic real-world engineering tasks, or produce plausible but fake answers.

Timing

Example: The response to an interrupt must land within 10 µs. The model’s code answers late.

Timing: the response must land within ten microseconds of the interruptThe interrupt arrives at 2 microseconds. The frontier model responds at 15.6, which is 3.6 late. The engineer responds at 8.4.05101520 µs[0, 10] µsInterruptModelEngineer15.6 µs late8.4 µsTiming: the response must land within ten microseconds of the interruptThe interrupt arrives at 2 microseconds. The frontier model responds at 15.6, which is 3.6 late. The engineer responds at 8.4.01020 µs[0, 10] µsInterruptModelEngineer15.6 µs late8.4 µs

Memory

Example: The build must fit 128 KB of RAM. The model’s build runs over.

Memory: the build must fit 128 kilobytes of RAMThe frontier model needs 151 kilobytes, 23 over the limit. The engineer needs 118.064128 KB[0, 128] KBModelEngineer151 KB, 23 over118 KBMemory: the build must fit 128 kilobytes of RAMThe frontier model needs 151 kilobytes, 23 over the limit. The engineer needs 118.064128 KB[0, 128] KBModelEngineer151 KB,23 over118 KB

Budgets

Example: Average current must stay within 2.5 mA. The model’s code keeps the chip awake between wake-ups.

Power: average current must stay within 2.5 milliamperesThe frontier model keeps the chip awake and averages 3.85 milliamperes. The engineer lets it sleep between wake-ups and averages 0.95.0250500 ms[0, 2.5] mA averageModelEngineer3.85 mA over0.95 mAPower: average current must stay within 2.5 milliamperesThe frontier model keeps the chip awake and averages 3.85 milliamperes. The engineer lets it sleep between wake-ups and averages 0.95.0250500 ms[0, 2.5] mA averageModelEngineer3.85 mA over0.95 mA

ConstraintBench: A benchmark for real-world constraint-based coding

Measures how AI models handle real engineering work in chip design, RTL, CAD and embedded coding. Domain engineers write every task, and scripts grade each answer against the design’s real constraints. New models are added as they are released.

Tasks50

Chip design, RTL, CAD and embedded coding

Environments3

Each graded by one family of checker scripts

Pass rate · ConstraintBench v1.2
1GPT-6.1 Sol65.8%[50, 82]
2Claude Opus 5.560.8%[47, 74]
3Claude Sonnet 5.546.7%[33, 60]
4Gemini 4 Argon39.6%[27, 53]
5Muse Spark 1.329.2%[17, 43]
6Grok 4.726.3%[14, 39]
7Kimi K37.1%[3, 13]
8Qwen3.8 Max5.4%[1, 11]
9GLM 5.33.8%[0, 8]
10Nemotron 3 Ultra1.7%[0, 5]
11Nemotron 3.5 Super VL0.0%[0, 0]

11 models, 50 tasks, 8 runs per task. Brackets mark the 95% confidence interval.

Access

Partnering with a small number of frontier labs

Contact us for samples, data cards and pricing.

Step 1 of 2

I’m interested in…

Experts High-paid engineering work

Other questions? Contact calle@hashlist.com