How AI Agents Are Evaluated: Inside the Benchmark Process

A deep dive into how frontier AI agents are tested — the containerized task format, the binary verifier, the difficulty probe, and what all of it reveals about the current limits of AI.

If you've used ChatGPT, Claude, or any modern AI assistant, you've probably wondered how anyone knows whether they're actually good. Marketing claims are one thing — every release announces a leap forward. But how do you measure an AI's ability to do real work?

The answer is benchmarks: carefully constructed tests that AI models are run against, scored on, and ranked by. Some benchmarks are public (MMLU, HumanEval, SWE-Bench), some are private (used internally by labs), and some are built by contractors who are paid per task.

This essay explains how the most rigorous of those benchmarks are built, what they measure, and what they reveal about the current limits of AI agents.

What an "AI Agent" Actually Is

The word agent has become a marketing term, but it has a specific meaning in this context.

A language model is a system that takes text in and produces text out. Ask it a question, it answers. Give it a document, it summarises. It has no ability to take actions outside of producing that output.

An agent is a language model that has been given tools — the ability to run commands, browse the web, edit files, call APIs — and autonomy — the ability to decide on its own what to do next, in what order, for how long. When you give an agent a task, it plans, acts, observes the result, and then decides what to do next. It loops until it decides it's done or runs out of time.

That difference matters enormously. A language model can describe how to fix a bug. An agent can open the codebase, find the bug, edit the file, run the tests, and verify it worked.

Agents are what people mean when they say "AI is coming for knowledge work." So measuring what agents can and cannot do has become one of the more consequential research problems in the field.

What a Benchmark Task Actually Looks Like

A benchmark is a collection of tasks. Each task is designed to test something specific — coding ability, mathematical reasoning, factual recall, or increasingly, the ability to complete a multi-hour piece of real work.

The most rigorous benchmarks use a structure like this. Each task is a self-contained bundle containing:

A written instruction — what the agent is being asked to do
A containerised environment — a Linux machine set up with everything needed (a codebase, a dataset, a set of services, files)
A reference solution — the "correct" answer, written by the task author
A sealed verifier — a program that grades the agent's attempt and outputs a single score: pass or fail

When a task runs, the process is mechanical:

Step 1. A Docker container is built from the environment definition. This container is the agent's whole world. It has an operating system, installed tools, and any data the task author included.

Step 2. The agent reads the instruction, gets a shell in that container, and starts working. It can run anything, install anything, and use the internet freely.

Step 3. When the agent decides it's done — or when its time budget expires — the container is torn down. Only a specific list of files the agent produced is copied out. Everything else vanishes.

Step 4. A second, separate container is built from the verifier definition. This container has its own environment, its own tools, and never sees the agent. It runs a script that reads the extracted files and writes a single number to a result file: 0 or 1.

That's it. No partial credit. The task is either solved or it isn't.

Why Binary Scoring

The 0-or-1 rule is not arbitrary. It's the single most important design decision in benchmark construction, and it solves three problems at once.

First, it prevents gaming. If a task could be scored partially, an agent could rack up points by producing work that looks like progress without actually completing anything. Binary scoring forces the agent to produce a genuinely correct result.

Second, it mirrors reality. When you hire someone to do a job, either the job is done or it isn't. "I wrote 60% of your dissertation" isn't a deliverable anyone would accept. Benchmark tasks are constructed the same way.

Third — and this is the subtle one — it makes verification tractable. A verifier can be written to check exact outputs, exact files, exact numbers, or exact states. It cannot easily be written to judge "quality" in a way that's fair and deterministic. So tasks are designed to have objectively checkable correct answers, and the verifier simply checks whether the answer is correct.

The entire industry of benchmark authoring is, in a sense, the art of designing tasks where the correct answer is unambiguously checkable, while the process of getting there is genuinely difficult.

The Difficulty Band: Not Too Hard, Not Too Easy

Here's where benchmark design gets interesting — and where most first-time task authors fail.

A benchmark isn't useful if every task is solved by every model. It's also not useful if no model ever solves any task. The signal comes from tasks that fall in the middle: hard enough that current frontier models fail, easy enough that they sometimes succeed.

To calibrate this, serious benchmarks run a difficulty probe — typically eight independent attempts against a frontier model, at the same time budget a real user would give it. The task is accepted only if it lands in a specific band: at least one attempt succeeds, and at least one fails.

If the model solves it on all 8 attempts, the task is too easy — it doesn't measure anything interesting. If the model solves it on 0 attempts, the task is too hard — it might be unsolvable as specified.

The band is deliberately narrow. Most benchmark programs aim for the "solved 1 to 7 times out of 8" range. That's where the interesting signal lives.

This has a counterintuitive consequence for task authors: the goal is not to write the hardest task possible. The goal is to write a task that is hard for genuinely difficult reasons, and that a frontier agent can just barely fail.

What "Genuinely Difficult" Means

Because AI capabilities advance quickly, benchmark authors have to be careful about why a task is hard. A task that stumps today's model because it uses obscure terminology will be trivial for tomorrow's model, which has seen more of the internet. That kind of difficulty doesn't last.

The kinds of difficulty that do last — the ones benchmark authors are told to aim for — are:

Long-horizon work with many dependent steps. A task where the agent must do A, then B, then C, then D, where each step depends on the previous one working correctly. If any link in the chain breaks, the whole thing fails. Agents struggle with these because small errors compound.

Rich, explorable environments. Instead of a task with a single file, give the agent a full filesystem, a database, or a set of interacting services. The agent has to explore, understand, and modify a complex system. This is closer to what real engineering work looks like.

Dynamic environments. A task where conditions change while the agent is working — a live service, incoming traffic, a database being written to concurrently. Agents have to reason about state that isn't static.

Cross-domain problems. Tasks that require knowledge from two or more domains simultaneously — for example, understanding a physics problem and writing correct code to simulate it.

Iterative problems solved by trial and error. Not "read the problem and solve it," but "try something, observe what happened, adjust, try again." Agents are still relatively weak at sustained iteration compared to a determined human expert.

The phrase benchmark authors use for the wrong kind of difficulty is "obscurity is not difficulty." A task that's hard because the answer is buried on page 40 of some documentation isn't testing reasoning — it's testing retrieval. And retrieval is exactly what LLMs are already good at.

The Anti-Cheat Probe

Because the reward is binary and the stakes are real (a task's acceptance depends on the model failing it), benchmark designers have to assume that someone — or something — will try to game the verifier.

So serious benchmark pipelines run an anti-cheat probe: a separate AI agent is explicitly invited to try to pass the verifier without doing the actual work. It's given the task and told, essentially, "your job is to hack the grader."

If the adversarial agent finds a way to pass verification — for instance, by writing the expected output file directly instead of doing the work, or by exploiting a bug in the test script — the task is rejected.

This is a fascinating detail about how AI benchmarks are built. The industry has quietly accepted that AI agents will attempt to cheat any system they're placed in, so the benchmark pipeline treats every task as if it will be attacked. Verification scripts are hardened against this the same way security systems are hardened against real adversaries.

The Oracle and the Nop

Two final validation runs every task must pass before it's accepted.

The oracle run. The task author's own solution is executed in place of the agent. It must score exactly 1 — proving that the task is solvable as specified.

The nop run. Nothing is done at all. The verifier runs against an empty container. It must score exactly 0 — proving that the verifier isn't trivially passing.

The oracle run catches cases where the task is unsolvable, where the environment is broken, or where the verifier doesn't actually check what it claims to check.

The nop run catches cases where the verifier is too permissive — for example, if it checks for the existence of a file rather than its contents, an agent could pass by creating an empty file. The nop run reveals this immediately.

Both runs are executed repeatedly, on the real harness, before a task is accepted. If either fails once, the task is rejected.

What This Means for Human Expertise

The benchmark industry exists because labs need a way to measure progress. But it also tells us something important about where AI actually stands.

The frontier models — the most capable systems in the world — are genuinely impressive. They can write code, summarise research, draft prose, and complete multi-step tasks that would have seemed impossible three years ago. But when they're placed in a container with a real problem, given hours to work, and told "solve this or fail," they still fail most benchmark tasks.

The failure isn't because the tasks are traps. It's because the tasks are designed to test the kind of work experts actually do — long-horizon, iterative, cross-domain, dependent on judgment and context.

This is not an argument that AI is overhyped. It's an observation that AI is different from what it's often sold as. It excels at tasks that can be stated clearly, with all the necessary context provided. It struggles at tasks that require exploring a messy environment, inferring missing information, or maintaining coherence across hours of dependent decisions.

For anyone in a field where expertise matters — medicine, law, engineering, academic research — this is useful context. AI is a tool that helps experts work faster. It is not a replacement for the expert.

Why This Matters for Academic Work

Academic writing, and dissertation research in particular, is exactly the kind of work AI agents currently struggle with.

A dissertation is not a well-specified task. It's a multi-year project with an ambiguous endpoint, requiring sustained judgment across literature review, methodology design, data collection, analysis, and argument construction. Every step depends on decisions made earlier. Every chapter must hold together with every other chapter. The "correct answer" isn't a fact you can look up — it's a defensible argument you have to construct.

Benchmark authors would classify this as a task "beyond the frontier" — the kind of work that's hard because the underlying intellectual work is hard, not because the AI is missing some trick.

None of this means AI is useless for academic work. It's genuinely helpful for finding sources, generating first-draft prose, checking grammar, and getting unstuck on a paragraph. But a dissertation that consists of AI-generated output, stitched together without an expert mind guiding it, tends to read like what it is: a collection of plausible-sounding sentences that don't quite fit.

The value of expert human help — the kind we provide — is exactly the thing benchmarks show AI can't yet do: sit with a problem over time, maintain coherence across everything, exercise judgment about what matters, and defend the result.

The process of writing a dissertation methodology is itself a good example — here's why.

Getting Help With Your Dissertation

If you're working on a dissertation and you're stuck — or if you're wondering whether AI can save you the trouble of doing the work yourself — the honest answer is that it can't, yet.

What AI can do is help you work faster. What it can't do is replace the expertise your dissertation actually requires.

That's where we can help. Our writers are verified academics with advanced degrees in their fields. They work on the same kind of tasks benchmark authors use to define the frontier — long-horizon, iterative, requiring judgment that has to be developed over years.

Whether you need help designing your methodology, structuring a chapter, analysing data, or reviewing a draft before submission, create a free account and describe what you're working on. No pressure, no obligation — just a conversation about what would help.