The researchers formalize all of this as a finite-horizon POMDP, short for Partially Observable Markov Decision Process. In plain terms, that's a model for making decisions when you can't see the true state of the world and only get partial, incomplete observations of it. The "finite-horizon" part just means every episode has to end within a fixed number of turns. So why reach for that framing?

Because the agent genuinely can't see most of what matters. The latent state has four pieces: the backend state, whatever the customer hasn't said yet, the evaluator's event log, and the episode status. What the agent actually observes is much thinner: customer messages and tool results. It can peek at the database through read tools, and anything the customer is holding back only surfaces if the agent asks.

One more design condition matters as much as the grading: isolation between attempts. If two attempts shared database rows or cached tool state, pass@k would stop meaning anything, so every attempt gets a deterministic reset to the initial state.

If the sandbox is the machinery, the tasks are what actually measure the models. There are 507 of them across five domains: 98 in retail and e-commerce, 104 in travel and hospitality, 100 in auto insurance, 104 in neobank internal IT, and 101 in consulting IT/HR. For 477 of them, the verdict comes from backend state alone. The remaining 30 also check the final response, because some requirements leave no trace in a database. Whether an agent leaked a hotel's confidential internal classification, for instance, is invisible in the backend, so those tasks get a response check on top.

How the tasks are written matters as much as what gets checked. The customer here isn't a person; it's a separate LLM, and the researchers specify what that model knows and how it's allowed to speak, task by task. That specification comes in two parts. The first is the opening request, which isn't generated at all. It's replayed verbatim from the task file. The second is a block called USER CONTEXT, holding the behavioral rules plus the facts the customer is permitted to disclose. Those facts are deliberately withheld from the opening request, so the agent only gets them by asking. Asking the right clarifying question isn't a nicety here; it's a requirement. Here's what that block looks like for a retail membership-upgrade task. 🛍

That's the entire hand the simulated customer is holding. Even the email address needed to look up the account stays hidden unless the agent asks for it.

The rules binding the customer are tight in other ways too. Entities like IDs, names, dates, and numbers have to be copied verbatim from the allowed information, never invented. If the agent asks about something the customer doesn't know, the answer is a flat "I don't know" rather than a plausible guess. The customer answers only the question in front of it, never volunteering the rest of the spec, and if the agent tries to hand its own work back ("draft this for me," "walk me through your reasoning"), the customer pushes it right back: that's your job, you're the assistant. Cooperative when asked, silent when not, which means the agent has to notice what's missing on its own before anything moves forward.

Its view is narrowed too, not just its voice. The transcript passed to the simulated customer keeps only what the customer and agent said out loud, with tool calls, tool results, and reasoning traces stripped out. The customer therefore knows less than the agent does. It can't see a tool throw an error, so it can't rescue an agent that mishandles one, and it can't recite backend state the agent never surfaced in conversation.

The team put 12 models through the same sandbox, with 20 independent runs on each of the 507 tasks. Every run resets the backend, so attempts can't contaminate each other, and each model accumulates 10,140 trials. Run a task once and all you learn is pass or fail. Run it twenty times and you find out whether the model got lucky or actually knows how to do the job. 🎯

Three metrics, before the numbers:

• pass@1: average success rate on a single attempt
• pass@20: share of tasks solved at least once across twenty attempts
• pass^20: share of tasks solved on all twenty attempts

GPT-5.4, the top scorer, landed at 65.36%, 91.12%, and 25.25% respectively. Give it twenty shots and it finds a working trajectory on 91% of the 507 tasks, but it nails all twenty on only 25%. The roughly 66-point spread in between is the zone where a model can solve the task but can't be counted on to.

Executable checks tell you that a trial failed, not why. So the researchers walked each failed trial and assigned it a single dominant failure signature. An average of 77.5% of failures came down to the agent failing to recover after a tool returned an error. For GPT-5.4 it was 89.6% of failures; for GLM-5.1, 88.1%. What does that look like in practice?

Often the agent simply proceeds as if the failed action had worked. One trace in the appendix: in a neobank internal IT task, the agent calls the access-provisioning tool and gets back an error saying the employee already has access. Instead of retrying with an update operation or opening the required ticket, it tells the user "I've updated your access accordingly" and ends the conversation. The checker caught both gaps: no access row in the golden state, and one fewer ticket than required.

The remaining categories are much smaller: wrong state update at 12.1%, incomplete user resolution at 7.9%, and no state-changing action at all at 2.5%. Wrong state update is the interesting one, because the tool executes cleanly and the result is still wrong. In one insurance task, the agent pulled up an account that had already used both payment arrangements allowed in a twelve-month window, then granted a third. 💸

Here's the part that should worry anyone shipping agents: these failures don't look like failures. The researchers re-graded all 79,853 failed trials on surface signals alone, ignoring the database. 80.88% ended cleanly, with no dangling questions, and called at least one state-changing tool. 67.24% also finished with no error in the final tool response. Judge on fluent responses and successful tool calls, and most of these would have passed.

The database says otherwise. Wrong field values showed up in 77.61% of failures, unintended extra side effects in 43.30%, and missing state changes in 25.36%. (A trial can trip multiple checks, so those don't sum to 100%.) Note which side is bigger: a refund issued twice or a booking created on the wrong account is more dangerous than something that simply didn't happen.

The simulated customer is not like a real one. It never misremembers a fact, never changes its goal midway, and never withholds cooperation no matter how many times the agent fumbles. Failures that come from mishandling a difficult person don't show up in these numbers at all. There's one more wrinkle the authors flag themselves: the model playing that customer is GPT-5.4-mini, the same family as the top-scoring agent, so a same-family interaction advantage can't be ruled out.

Still, the gap the paper surfaces is real. Fluent tool use and reliable task completion are not the same capability, and the distance between them only becomes visible when you open up the final state. An agent that gets it right once out of twenty isn't one you hand the refund button to. We're waiting on the agents that get it right every time. 💫