Your Agent Aced the Task. Will It Do It Again?
2026-09-15 · Hugging Face
Your Agent Aced the Task. Will It Do It Again?
The Reliability Crisis Hidden by Average Success Rates
When an agent performs well in rehearsal but fails the same task during a live demo or in production, it becomes a severe reliability problem. For mission-critical workflows like reconciling financial transactions, a single success does not guarantee future reliability.
Most standard benchmarks report Mean@k, which averages the pass rate over k runs. However, this fails to answer the real user's question: will it succeed if I ask the exact same question again? To answer this, we need Pass^k, the fraction of tasks where the agent succeeds on all k runs. Unlike the optimistic Pass@k (at least one success), Pass^k is pessimistic, requiring every attempt to succeed. Thus, Pass^k ≤ Mean@k ≤ Pass@k.
A ReAct agent backed by GPT-4.1 posted a Mean@5 of 77.4% on AppWorld. Yet, its Pass^5 was only 53.0%, revealing a 24.4-point consistency gap. Nearly a quarter of the tasks were inconsistently solved. This is not a capability issue fixed by a larger model; an agent can be capable and inconsistent simultaneously.
Why Agents Flip: Sharp vs. Flat Distributions
Every time an LLM agent makes a decision, the outcome comes from a probability distribution over next tokens. The shape of this distribution dictates reliability:
- Sharp distributions: Place most mass on a single token, making the same choice resilient to minor platform-side effects like GPU floating-point non-associativity.
- Flat distributions: Spread comparable mass across near-tied tokens, making the outcome as unpredictable as a coin flip.
Flat distributions are vulnerable to tiny perturbations. Because a trajectory chains dozens of decisions, a small per-step chance of flipping compounds into a large probability that a run goes differently. This explains the 24-point gap.
This problem persists despite decoding settings. Greedy decoding and fixed seeds govern how a distribution turns into a token, not the distribution itself. On hosted endpoints, probabilities shift slightly between runs, meaning the same prompt at temperature zero can resolve differently over time.
Diagnose and Fix: Consistency Analyzer and Guidelines
To address this, the team introduced consistency guidelines into the ALTK-Evolve system, driven by a diagnostic tool called the Consistency Analyzer.
Diagnose: The Consistency Analyzer
Given a single recorded trajectory, the analyzer replays each decision step. It requires no ground truth and no end-to-end re-runs. Instead, it resamples each decision point in the trace with a single call requesting k completions (default k=5) to identify flip-prone steps where the model was one token away from a different action.
Fix: Turning Diagnosis into Guidelines
By converting this diagnosis into consistency guidelines and injecting them back at inference time, the system halves the consistency gap from 24.4 percentage points to 12.0 percentage points. This yielded a 16.0-point improvement for same-task Pass^5 and 13.0 points for similar tasks, with zero cost to average accuracy.