SOURCE-LINKED INTELLIGENCE

Substrate-Portable Execution for Production LLM Workflows

arXiv · AI, language, vision and robotics · article · Sep 5, 2026 · UTC

Production LLM agents execute tool-calling loops, retrieval chains, and compositional workflows in multiple modes, yet execution semantics are often coupled to one runtime. We encountered this portability problem in Rufus, a conversational AI assistant with a large tool catalog that serves millions of Amazon customers. Rufus supports real-time serving, asynchronous background tasks, and high-volume batch workloads such as evaluation and content pregeneration. Each mode has distinct service-level objectives and typically uses a separate runtime. Reusing streaming orchestration makes asynchronou

Read original source ↗ Open in workspace

recordType

paper

region

Global