SOURCE-LINKED INTELLIGENCE

Pretraining of Medical Visual Encoders Toward Multi-modal Large Language Models

arXiv · AI, language, vision and robotics · article · Sep 20, 2026 · UTC

Multimodal Large Language Models (MLLMs) commonly reuse visual encoders pretrained with CLIP, although the features of these ViTs are ultimately consumed by autoregressive LLMs. We refer to this mismatch as the semantic-interface gap and introduce MedMLIP, a framework that pretrains the visual encoder through report generation with a frozen LLM, while employing Local Relational Distillation (LRD) to preserve relationships among visual patches to avoid visual collapse. We pretrain MedMLIP on IU-Xray and Open-PMC-300K and evaluate the resulting encoders on VQA-RAD and SLAKE. Only the ViT is tran

Read original source ↗ Open in workspace

recordType

paper

region

Global