SOURCE-LINKED INTELLIGENCE

Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

arXiv · AI, language, vision and robotics · article · Sep 22, 2026 · UTC

Large language model (LLM) outputs are expected to be reproducible under greedy decoding, yet in practice the same model, prompt, and software stack produce different outputs on different GPUs. The root cause is floating-point non-associativity combined with hardware-dependent kernel selection. Inference frameworks select different matrix-multiplication kernels on each architecture, with different parallel reduction orders and unspecified tensor-core arithmetic, and the resulting rounding differences can flip output tokens. Existing solutions have imperfect cross-architecture reproducibility a

Read original source ↗ Open in workspace

recordType

paper

region

Global