SOURCE-LINKED INTELLIGENCE

Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

In many low-resource settings, even just eliciting speech for data collection is difficult. One promising approach has been to ask speakers to describe images. But how do we build models from such visually grounded speech data? Given a dataset of images with Hindi spoken captions, we consider how we can map a written English keyword to spoken realisations of that word in Hindi. Previous work trained end-to-end multimodal neural models. Instead, we explore a simpler alignment-based approach built on self-supervised speech representations. Written English tags are automatically obtained from ima

Read original source ↗ Open in workspace

recordType

paper

region

Global