SOURCE-LINKED INTELLIGENCE

Do Large Language Models Capture the Diversity in their Training Data?

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

Large language models are trained to model conditional distributions over text, yet it remains inadequately understood whether they capture the full diversity of plausible outputs present in their training data. We study this question through an information-theoretic lens by comparing the conditional entropy of model-generated outputs with that of the corresponding training data. Given paired input-output samples, we use conditional entropy and its matrix-based analogue based on von Neumann entropy to measure output variability beyond what is explained by the conditioning input, without requir

Read original source ↗ Open in workspace

recordType

paper

region

Global