SOURCE-LINKED INTELLIGENCE

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

arXiv · AI, language, vision and robotics · article · Sep 18, 2026 · UTC

We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce.

Read original source ↗ Open in workspace

recordType

paper

region

Global