SOURCE-LINKED INTELLIGENCE

An Analysis of Self-supervised Pre-training with Dependent Samples

arXiv · AI, language, vision and robotics · article · Sep 4, 2026 · UTC

Self-supervised learning relies on so-called data augmentations $φ(x)$ of unlabeled datapoints $x$ --- for example, masking random pixels in an image $x$ --- that should leave the label of $x$ invariant and are often used to learn a lower-complexity invariant subspace $\cal V$ for downstream tasks. In practice, such augmentations $\{ φ_l(x_i) \}$ are pooled together to learn $\cal V$, despite obvious inter-dependencies between different augmentations $φ_l(x), φ_k(x)$ of the same datapoint $x$. However, theoretical works on the subject typically consider procedures that avoid such dependencies,

Read original source ↗ Open in workspace

recordType

paper

region

Global