SOURCE-LINKED INTELLIGENCE

Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation

arXiv · AI, language, vision and robotics · article · Sep 22, 2026 · UTC

A coding agent must emit a valid tool call--a parseable invocation of a tool in the provided schema--before the harness can execute its chosen action. We study how local serving stacks affect this protocol step and show that measured outcomes can depend on the serving layer rather than model behavior alone. In Ollama, the default tools= request is gated per model by a static template flag: some models are accepted and return calls as text, some return native tool_calls, while Phi-3 and Gemma-3 are rejected before inference. In our harness, rejection and retry exhaustion are not preserved as st

Read original source ↗ Open in workspace

recordType

paper

region

Global