SOURCE-LINKED INTELLIGENCE

Mind the Approximation: Fisher-Weighted SVD Compression for ViTs

arXiv · AI, language, vision and robotics · article · Sep 7, 2026 · UTC

Model compression is key to mitigate deployment challenges of ever growing machine learning models. In this area of research, singular value decomposition (SVD)-based compression offers a compelling trade-off between computational efficiency and model accuracy. Fisher-weighted SVD in particular provides principled, loss-aware compression. However, we find that improving the fidelity of Fisher approximation used in the compression is poorly predictive of post-compression accuracy for Vision Transformers (ViTs). Motivated by this observation, we propose FACTS, a structured Fisher Approximation t

Read original source ↗ Open in workspace

recordType

paper

region

Global