OORT基准测试：生产环境中的代理性能

Most AI benchmarks measure the wrong thing. They compare models on curated datasets, with optimized prompts, in environments without latency or failures. The results are impressive — and completely irrelevant for anyone who needs to operate agents in real production.

In production, data arrives dirty. External APIs fail. Load varies. Business contexts are ambiguous. Gartner reports that AI agent precision drops between 15% and 40% when moved from controlled environments to real operations. MLCommons identified that average latency in production is 2.4 times higher than in the lab.

The OORT Benchmark was created to measure what truly matters: how agents perform when facing a company’s operational reality. Not in idealized scenarios, but in real workflows, with real data, under real load conditions.

AI benchmarks were designed to compare models, not to validate operations. MMLU measures general knowledge. HumanEval measures code generation. HELM measures a broader spectrum. None of them answers the question that matters for a company: will this agent work in my workflow, with my data, at my volume?

The distance between an academic benchmark and real operations is structural. In the lab, the prompt is perfect. In production, the user types with errors, omits context, sends data in unexpected formats. In the lab, the response takes 800ms. In production, the agent needs to query three external APIs, process a 40-page document, and validate against a rule base — and the response can take 12 seconds.

Stanford reported that LLM performance on complex reasoning tasks can drop up to 39% when the question format changes slightly. If a model is sensitive to prompt format, imagine what happens when input comes from a legacy system that formats data inconsistently.

The OORT Benchmark measures performance across four complementary dimensions. Each dimension reveals a different aspect of an agent’s operational health. Optimizing only one — for example, precision — without considering the others produces slow, expensive agents or ones that escalate to humans at every decision.

End-to-end latency — the total time from when the agent receives a task until delivering the final result. Includes LLM calls, database queries, external API integrations, and orchestration overhead. It’s not model latency. It’s operation latency.

Complex task precision — accuracy rate evaluated by human review through statistical sampling of real executions. Unlike academic benchmarks, the evaluation considers business context: a technically correct answer that doesn’t solve the user’s problem is counted as an error.

Cost per operation — how much it costs to execute each task, including LLM tokens, API calls, retries after failures, and infrastructure overhead. Measuring cost per operation (not cost per token) reveals invisible inefficiencies: an agent that succeeds on one attempt costs half of one that needs three.

Human fallback rate — percentage of tasks where the agent couldn’t complete the operation alone and triggered human review. A high fallback rate indicates the agent is operating outside its competence envelope — or that confidence thresholds are calibrated too conservatively.

Performance degradation between lab and production isn’t random. It follows predictable patterns that the OORT Benchmark identifies and quantifies.

Imperfect data. In the lab, test data is clean and standardized. In production, 27% of executions involve data with missing fields, inconsistent formats, or ambiguous information. IBM estimates that data quality problems cost US$ 3.1 trillion per year to the American economy. Agents not tested against imperfect data fail silently — producing plausible but incorrect responses.

Compound latency. Each external API call adds latency. An agent that queries a CRM, checks a knowledge base, and validates against compliance rules can accumulate 8-15 seconds of latency — even if each individual call takes less than 3 seconds. Timeouts and retries multiply this effect.

Cumulative edge cases. Any test dataset captures only a fraction of real scenarios. In operations with thousands of daily executions, edge cases representing 0.1% of volume become dozens of failures per day. The OORT Benchmark catalogs these cases and incorporates them into the improvement cycle.

The OORT Benchmark isn’t a one-time evaluation. It’s a continuous monitoring system that directly feeds the agent optimization cycle. Each production execution generates data that refines the next operation.

The compound effect is significant. In practice, we observe that agents optimized with OORT Benchmark data improve between 7 and 9 percentage points of precision in the first 90 days, while the human fallback rate drops by half. This pattern is consistent because improvements are driven by real operational data, not intuition.

Benchmarks without transparent methodology are marketing, not engineering. The OORT Benchmark publishes test conditions, sample size, and confidence intervals for each reported metric.

Measurements are conducted on real production executions, not simulations. Data is anonymized, but conditions are preserved: load volume, task complexity, input quality, and external integration state. This ensures the numbers reflect operational reality, not an optimistic scenario.

Precision evaluation combines automated validation (verifiable business rules) with human review through statistical sampling. The sample is sized for 95% confidence with ±3% margin of error. Ambiguous results are classified by domain specialists, not by textual similarity metrics.

Most companies choose AI agents based on impressive demos and lab benchmarks. Then they discover that production performance doesn’t resemble what was presented. The gap between expectation and reality is predictable — and avoidable.

The OORT Benchmark exists because we believe an agent’s performance is defined by what it does in production, not by what it does on a slide. Measuring rigorously is the first step to operating with confidence.

Agents that improve continuously need data about how they’re performing continuously. Without a production benchmark, optimization is guesswork. With one, it’s engineering.