Attending Slush 2026 in Helsinki?Join our Frontier AI for Real-World Engineering event during Slush

Register on Luma

Making AI possible for real-world engineering work

ConstraintBench releasesRelease date
[2]v1.2 · 50 tasks, 11 modelsNewNov 2026
[1]v1.1 · 11 models, FW-01 grader fixOct 2026
[0]v1.0 · results, limited previewOct 2026

Today’s frontier models build web apps in seconds, but struggle in safety-critical embedded environments

About us

We build custom evals, data sets, and refine data for the embedded engineering domain.

100,000+

Domain engineers registered

Our background

We come from the automotive industry, where we have helped the world’s largest carmakers build low-level chip integrations & implement AI in engineering processes.

Selected clients
Member of

Data and evaluations for frontier labs & enterprises

[ 01 ] Custom evals and data

Evals and data built to your capability target

TASK · EXPERT-WRITTENSet how long the firmware waits for the regulator on this board to settle.GOLDEN ANSWER · HIDDEN
GRADERRESULT
Uses the board’s own test data
Wait within 5% of the golden answer
Startup order kept
Code builds and runs
Reasoning explains the choice
4 OF 5 CRITERIA MET
  • Tasks written by domain experts
  • Golden answers and deterministic graders
  • Quality review on every task
  • Delivered in your format
[ 02 ] Off-the-shelf data

Ready-to-use data sets and reinforcement learning environments

AGENT
ENVIRONMENT
GRADER
REWARD
REWARD BY TRAINING ROUND
BOARDTOOLCHAINGRADER
  • Environments with graders built in
  • Areas picked from our model failure research
  • Licensed and ready to train on
  • Improved after delivery
[ 03 ] Operational engineering data

Real engineering work from engineering firms

DESIGN REVIEW · PAST PROJECTReviewed with Anna Virtanen for Client X. The brake signal arrives late on prototype P-17, so the timeout moves from 20 ms to 35 ms. Option B was dropped because the bus load passed 70% on test track 3.ANONYMISED
REQUIREMENTBrake signal timeout
DECISION20 ms to 35 ms
REASONLate signal on a prototype
REJECTED OPTIONBus load over 70%
  • Specifications, tickets and source code
  • Design reviews and the reasons behind them
  • Rebuilt and anonymised
  • Work models have never seen
[ 04 ] Computer-use dataNEW

Engineers and agents working in real engineering software

PCB Editor
STEPSACTIONSKEYSVOICE
  • Screen video at 30–60 fps
  • Keys, clicks and window events, on one clock
  • Reasoning, spoken or from the agent
  • Task files and graded outcomes
[ 05 ] Your own engineering model

Your own AI model for your engineering work

BENCHMARK
DATA
TRAINING
YOUR MODEL
CORRECT ON YOUR TASKS, BY ROUND
YOUR SERVERSYOUR WEIGHTSYOUR DATA
  • Benchmark of today’s models on your tasks
  • Training data from your engineers’ work
  • Open model post-trained with fine-tuning and RL
  • Runs on your servers, you own the weights
[ 06 ] Forward deployed domain experts

Domain experts working alongside your research team

OUR NETWORK
100,000+
ACTIVE CONTRIBUTORS
YOUR RESEARCH TEAM
RESEARCHERSFDES
YOUR RESEARCH PROBLEM
  • Chip design, RTL, CAD and embedded coding
  • Work in your repos, compute and channels
  • Matched to the domain your model is learning
  • From 100,000+ active contributors

Select a product

ConstraintBench: A benchmark for real-world constraint-based coding

Measures how AI models handle real engineering work in chip design, RTL, CAD and embedded coding. Domain engineers write every task, and scripts grade each answer against the design’s real constraints. New models are added as they are released.

Tasks50

Chip design, RTL, CAD and embedded coding

Environments3

Each graded by one family of checker scripts

Pass rate · ConstraintBench v1.2
1GPT-6.1 Sol65.8%[50, 82]
2Claude Opus 5.560.8%[47, 74]
3Claude Sonnet 5.546.7%[33, 60]
4Gemini 4 Argon39.6%[27, 53]
5Muse Spark 1.329.2%[17, 43]
6Grok 4.726.3%[14, 39]
7Kimi K37.1%[3, 13]
8Qwen3.8 Max5.4%[1, 11]
9GLM 5.33.8%[0, 8]
10Nemotron 3 Ultra1.7%[0, 5]
11Nemotron 3.5 Super VL0.0%[0, 0]

11 models, 50 tasks, 8 runs per task. Brackets mark the 95% confidence interval.

Access

Partnering with a small number of frontier labs

Contact us for samples, data cards and pricing.

Step 1 of 2

I’m interested in…

Experts High-paid engineering work

Other questions? Contact calle@hashlist.com