SOURCE-LINKED INTELLIGENCE

TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving

arXiv · AI, language, vision and robotics · article · Aug 26, 2026 · UTC

Prefix caching introduces a fundamental tradeoff in multi-agent large language model (LLM) serving: retaining a long system-prompt key-value (KV) cache for an agent accelerates future calls, yet it reduces the GPU memory available for batching concurrent requests. In multi-stage workflows, existing schedulers tend to prioritize either immediate prefix locality or overall workflow progress. However, under a shared KV cache budget, optimizing either objective in isolation can prolong tasklevel job completion time (JCT) through downstream delays or frequent prefix replacement. To strike a balance

Read original source ↗ Open in workspace

recordType

paper

region

Global