Making AI possible for real-world engineering work.

ConstraintBench releasesRelease date
[1]v0.2 · public releaseUpcomingNov 2026
[0]v0.1 · limited preview, 30 tasksOct 2026

Today’s frontier models build web apps in seconds, but struggle in safety-critical embedded environments.

About us

We build custom evals, data sets, and refine data for the embedded engineering domain.

100,000+

Domain engineers registered

Our background

Our background comes from the vehicle industry, where we helped the world’s biggest automotive OEMs build low-level chip and sensor integrations.

Selected clients

Data and evaluations for frontier labs & enterprises.

01

Custom Evals & Data

Evals & data sets tailored to help you reach specific capability targets. Built with our world-class pool of domain experts.

02

Off the Shelf Data

Instantly available datasets & RL environments within high-demand capability areas, directly from our internal research.

03

Operational Engineering Data

Engineering data, documentation, codebases, decisions, and more from our engineering firm partnerships.

04

Harness Engineering

Make AI agents work inside enterprise environments through rules, checkers, and model fine-tuning.

Where real-world engineering fails.

Today's frontier models are able to code almost any system, but fail on basic real-world engineering tasks, or produce plausible but fake answers.

Timing

Example: The response to an interrupt must land within 10 µs. The model’s code answers late.

Timing: the response must land within ten microseconds of the interruptThe interrupt arrives at 2 microseconds. The frontier model responds at 15.6, which is 3.6 late. The engineer responds at 8.4.05101520 µs[0, 10] µsInterruptModelEngineer15.6 µs late8.4 µsTiming: the response must land within ten microseconds of the interruptThe interrupt arrives at 2 microseconds. The frontier model responds at 15.6, which is 3.6 late. The engineer responds at 8.4.01020 µs[0, 10] µsInterruptModelEngineer15.6 µs late8.4 µs

Memory

Example: The build must fit 128 KB of RAM. The model’s build runs over.

Memory: the build must fit 128 kilobytes of RAMThe frontier model needs 151 kilobytes, 23 over the limit. The engineer needs 118.064128 KB[0, 128] KBModelEngineer151 KB, 23 over118 KBMemory: the build must fit 128 kilobytes of RAMThe frontier model needs 151 kilobytes, 23 over the limit. The engineer needs 118.064128 KB[0, 128] KBModelEngineer151 KB,23 over118 KB

Budgets

Example: Average current must stay within 2.5 mA. The model’s code keeps the chip awake between wake-ups.

Power: average current must stay within 2.5 milliamperesThe frontier model keeps the chip awake and averages 3.85 milliamperes. The engineer lets it sleep between wake-ups and averages 0.95.0250500 ms[0, 2.5] mA averageModelEngineer3.85 mA over0.95 mAPower: average current must stay within 2.5 milliamperesThe frontier model keeps the chip awake and averages 3.85 milliamperes. The engineer lets it sleep between wake-ups and averages 0.95.0250500 ms[0, 2.5] mA averageModelEngineer3.85 mA over0.95 mA

ConstraintBench: A benchmark for real-world constraint-based coding

A benchmark that measures how well models think and operate in real-world engineering systems. Written by top domain engineers. Continuously updated on new model releases.

Tasks30

Low-level programming with real constraints

Environments3

Each with one family of checkers

Why attempts failed
43%
26%
20%
11%
Wrong engineeringarchitecture, traps, arithmetic
No verificationskipped the calculation
Incompletecriteria short
No deliverystopped

ConstraintBench v0.1 limited preview

Access

Partnering with a small number of frontier labs.

Contact us for samples, data cards and pricing.

Request access
team@hashlist.com