SOURCE-LINKED INTELLIGENCE

Communication-Efficient LLM Adaptation over Decentralized GPU Meshes

arXiv · AI, language, vision and robotics · article · Sep 13, 2026 · UTC

Decentralized training enables large-model training over low-end GPUs and internet-grade connections, but communication along both data-parallel and pipeline-parallel axes becomes the primary bottleneck. We study post-pretraining adaptation in this setting. We propose an asynchronous two-circuit system: a fast compressed training circuit drives throughput using activation masking for pipeline-parallel (PP) transfer and compressed data-parallel (DP) synchronization, while a slow anchor circuit runs occasional unmasked forward--backward passes off the critical path. Then, we introduce a spectral

Read original source ↗ Open in workspace

recordType

paper

region

Global