We investigated the theoretical tradeoff between the compression ratio and output perturbation of neural network models and found out that the rate-distortion theoretic formulation introduces a theoretical foundation for pruning. 2/6
1
0
3
Replies
Excited to share our
#AISTATS2022
paper titled "An Information-Theoretic Justification for Model Pruning":
Come say hi at the conference during our poster session on Wednesday, March 30th, 8:30-10 am PST.
1/6
3
5
51
We first derived an information-theoretical limit of model compression and then showed that this limit can only be achieved when the reconstructed model is sparse (pruned). 3/6
1
0
1
This essentially implies that pruning, implicitly or explicitly, must be a part of a good compression algorithm, providing insight into the empirical success of model pruning. 4/6
1
0
1
We also developed a novel model compression method (called SuRP), guided by this information-theoretic formulation, which indeed outputs a sparse model without an explicit pruning step. 5/6
1
0
2
Our results, including the algorithm, are also applicable to gradient compression for communication-efficient federated learning. 6/6
0
0
3