Why we execute candidate code instead of scoring it by eye
A model asked whether code looks correct will say yes to code that does not compile. We run submissions against test cases in a sandbox and score what passes, because plausible and working are different properties.
The cheapest way to build a coding assessment is to ask a language model whether the submitted code solves the problem. It is also the least reliable. Models are trained to produce plausible text, and plausible code is exactly what a candidate under time pressure produces. The failure mode is not random — it is biased towards passing confident-looking wrong answers.
So we execute instead. Every submission runs against the problem's test cases in an isolated sandbox, across 71 languages, and the score is the proportion of cases that pass. A solution that does not compile scores zero regardless of how well-structured it reads.
This has a cost we accept: writing good test cases is harder than writing a prompt, and a badly specified problem produces unfair results faster than a subjective review would. That is why the assessment planner drafts test cases alongside the problem and shows them to you before anything is sent.
It also changes what the score means. Aptitude counts 40% and executed code 60%, weighted that way because multiple choice can be guessed and a passing test suite cannot.