Home › Week 27 · 337 Scenarios From Tool Specs Alone: Where Generated Agent Evals Break
Week 27 · August 2026

337 Scenarios From Tool Specs Alone: Where Generated Agent Evals Break

August 30, 2026 · by Satish K C 9 min read
MCP Agents LLMs Evaluation
Built by the Author Kravhal - autonomous agents that run your business workflows end-to-end. Pay per outcome.
Get Early Access

The Paper

"Agent Seer: Synthesizing Scenarios from Specification Understanding" was published in June 2026 by Harish Karumuri, Mahesh Vemula, and David Lopes Pegna of Apple. The central claim is that a tool specification on its own - function names, natural-language descriptions, and typed parameter schemas - already encodes enough semantic information to synthesize a complete evaluation harness, with no seed examples, no live tool execution, and no domain-specific tuning. Running the pipeline over seven public Model Context Protocol specifications produced 337 scenarios that score a mean 0.911 on tool-calling correctness and 0.855 on conversational coherence, with complete tool coverage on every specification up to 56 tools.

Read the Paper on arXiv →

The Problem Before This Paper

Agent evaluation has good benchmarks and no way to get one for the tool suite you actually run. GAIA2 (Froger et al., ICLR 2026), AgentBench (Liu et al., 2023), WorkArena (Drouin et al., ICML 2024), and the MCP-native suites that followed - MCPVerse, MCP-AgentBench (Guo et al., AAAI 2026), Toolathlon - are all built by manual curation or require live tool access, and all of them freeze the moment they are released. That produces three specific gaps the paper names. The curation bottleneck: writing a realistic scenario means connecting a user goal to a tool sequence, filling every parameter with a plausible value, and knowing how practitioners actually chain the calls, which is domain expertise that does not scale across the combinatorial space of tool pathways. The static benchmark problem: an agent scoring well against a snapshot may be graded against tool descriptions that no longer match production. The multi-turn gap: follow-up turns that react to specific tool outputs require real tool responses to write against. Synthetic data work mostly assumes execution - APIGen verifies through real calls, TOUCAN (Xu et al., 2025) scaled to 1.5M trajectories by driving live MCP servers - while simulated environments such as tau-bench (Yao et al., 2025), tau2-bench (Barres et al., 2025), and ToolSandbox (Lu et al., 2025) replace the tools but still need someone to build the environment. The residual is what the authors call the cold-start evaluation problem: producing realistic evaluation data for a tool suite that has none, which is the normal condition for new, private, or fast-moving internal APIs.

What They Built

Agent Seer is a four-stage pipeline where each stage consumes validated structured output from the previous one, so a schema violation is caught at the boundary instead of propagating. Stage one, tool interpretation, prompts an LLM for four semantic fields per tool: a functional description, required parameters with semantic roles, a primary use case, and organizational context. Stage two, scenario generation, emits enterprise workflow scenarios at two complexity levels, each carrying a title, a user-facing instruction, an ordered list of expected tool calls with concrete parameter values, and a follow-up question. The output schema forces reasoning fields into the generation itself: every call carries a quick_explanation for why it fires, and every scenario a novelty_reason for its evaluation value. Stage three synthesizes a mock JSON response per call, tagged with a grounding tier of high, medium, or low that records whether reference material was available; supplying real example outputs is optional, which makes the pipeline a spectrum rather than a binary. Stage four expands a scenario into multi-turn dialogue, splitting at natural phase boundaries so the resulting turns preserve the two patterns BFCL v3 formalizes - multi-step, where each call depends on the previous result, and multi-hop, where independent calls gather information that must be synthesized. Follow-up turns quote concrete values from the mock outputs, entity names and counts and status codes, rather than restating the task. When expansion yields a single turn the scenario is discarded on the assumption it lacked the substance to split. The product is a self-contained harness: a prompt, an ordered list of expected calls held out as the oracle, mock outputs to feed as tool responses, and the conversation. Any MCP-compatible framework can run an agent against it without touching the real tools.

The scoring is where the paper earns its result. Rather than a pass/fail on the call, an LLM judge scores four dimensions on 0-10, normalized to 0-1, with argument correctness decomposed into six sub-dimensions and cascading penalties enforced through the judge prompt:

TC_turn = mean(usage, selection, ordering*, arguments)
arguments = mean(completeness, name, value,
               type, format, relevancy)
wrong name or missing required -> value = type = format = 0
wrong value                -> type = format = relevancy = 0
* ordering is dropped when the turn calls one tool

A single critical error therefore collapses the argument mean rather than shaving a few points off it, which is the mechanism that makes argument failures visible at all. Usage correctness captures necessity, whether a tool was warranted; overuse is recorded as a diagnostic but excluded from the aggregate. Selection averages correctness, specificity, and completeness of the tools chosen. Ordering averages sequence logic, dependency handling, and execution efficiency. Coherence is scored separately and never folded in:

Coh = mean(logical flow, completeness, conciseness,
          topic relevance, context retention)
each sub-aspect on 1-3, normalized to 0-1

Generation ran on Gemini 2.5 Flash Lite in structured-output mode at temperature 0.7, with up to three retries on validation failure before the record was discarded. Judging ran on Gemini 2.5 Flash at temperature 0. The seven specifications span Illustrator (64 tools, nested objects), Selenium (56, flat but stateful), Redis (47, flat key-value), Git (33, deeply nested with 11.2 parameters per tool), Elasticsearch (20, nested query DSL), Slack (16, mixed), and Filesystem (14, flat).

Key Findings

Results

Mean unsupervised tool-calling across the corpus is 0.911 (95% bootstrap CI [0.897, 0.925], median 0.979) and mean coherence is 0.855 ([0.838, 0.872], median 0.933), from 337 scenarios yielding 391 evaluation records. The distribution is top-heavy: 31.7% of records score a perfect 1.0 on tool calling and only 2.3% fall below 0.5. Per specification, Redis leads at 0.966 tool calling and 0.902 coherence, followed by Selenium at 0.935, Elasticsearch at 0.930, Illustrator at 0.898, Slack at 0.886, Filesystem at 0.876, and Git last at 0.857 with the worst coherence at 0.757. Slack scores the highest coherence at 0.938 despite mid-pack tool calling, and the two dimensions stay weakly correlated at the record level (r = +0.23, n = 381) and the tool level (r = +0.16, n = 222), so they carry largely independent diagnostic signal. Complex scenarios cost 7.3 points of tool calling (0.949 to 0.877) and 5.3 points of coherence (0.883 to 0.830), with non-overlapping confidence intervals in both dimensions; Elasticsearch degrades hardest at 12.2 points and Git at 11.1 (0.910 to 0.799), while Selenium is flat at 0.932 to 0.935. Re-scoring the entire corpus with an out-of-family judge, Qwen3.5-122B-A10B-FP8, leaves tool calling essentially unchanged - no mean shift, 95% CI [-0.009, +0.008], paired r = 0.79 over 384 records, MCP ranking preserved at rho = 0.86 - and reproduces argument value accuracy as the dominant sub-failure by a 4-5x margin under both judges. Coherence does not replicate as cleanly: the second judge is systematically stricter by about 0.16, paired correlation drops to 0.42, rank preservation to rho = 0.46, and the worst-coherence specification changes identity between judges. The authors report absolute coherence levels as judge-dependent rather than defending them.

Why This Matters for AI and Automation

My Take

The contribution the authors claim is the four-stage structure with validated boundaries, and the contribution I think actually matters is the argument sub-dimension decomposition with cascading penalties, because it turns a 42%-perfect argument rate into something diagnosable rather than a number to feel bad about. The correlation result is the practically useful half: parameter density, not tool count, is what degrades quality, and it holds at both grains with 222 tools behind the tool-level version, which is a large enough sample to design against. What the paper does not do, and what I kept waiting for, is the downstream experiment - nobody runs a real agent against these harnesses and shows that the scores separate a strong model from a weak one. Everything reported is the quality of the generated data as graded by a judge, so the harness is validated as data and not yet as an instrument, and "0.911 tool-calling correctness" is a statement about the generator, not about anything you would learn from running it. The circularity concern is handled about as well as it can be without humans: an out-of-family judge replicates tool calling at paired r = 0.79 and reproduces the failure taxonomy, which is real evidence, but coherence moving 0.16 under a judge swap says the coherence numbers are a property of Gemini's taste. The limitation I would fix first is cross-call referential integrity, since mock outputs for dependent calls are generated independently and IDs may not line up across a chain, which quietly undercuts exactly the multi-step workflows the harness exists to test; the authors name the fix, a shared state dictionary across the workflow, and it is not hard. The coverage ceiling at 64 tools is the other thing to watch, because enterprise MCP servers do not stop at 64, and coverage-aware sampling is the difference between a harness that exercises your tool suite and one that exercises the third of it the generator finds interesting.

Discussion question: The pipeline picks the right tool 77% of the time and the right function name almost always, but fills arguments perfectly only 42% of the time, with wrong values outnumbering wrong parameter names 223 to 11. If argument values are the hard part for a generator reading your spec, what evidence do you have that they are not also the hard part for the agent calling it in production, and does your evaluation score arguments below the level of the function name?

← Back to all papers
Share