SOURCE-LINKED INTELLIGENCE

ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

Powerful code agents can execute scripts, call tools, and manage files, yet many important applications remain accessible primarily through graphical user interfaces. We argue that screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning. We introduce ASIL (Agent-Software Interaction Layer), an agent-native interface that exposes software through structured JSON observations and code-executable semantic actions, realized through the deepest feasib

Read original source ↗ Open in workspace

recordType

paper

region

Global