We show that neural networks have a remarkable preference for low complexity which overlaps strongly with real-world data across modalities. PAC-Bayes proves that such models generalize, explaining why NNs are almost universally effective. Tweet added by Micah Goldblum @micahgoldblum

Micah Goldblum

10 months

We show that neural networks have a remarkable preference for low complexity which overlaps strongly with real-world data across modalities. PAC-Bayes proves that such models generalize, explaining why NNs are almost universally effective.

The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of...

No free lunch theorems for supervised learning state that no learner can solve all problems or that all learners achieve exactly the same accuracy on average over a uniform distribution on...

arxiv.org

Jim Fan

@DrJimFan

10 months

There're few who can deliver both great AI research and charismatic talks. OpenAI Chief Scientist @ilyasut is one of them. I watched Ilya's lecture at Simons Institute, where he delved into why unsupervised learning works through the lens of compression. Sharing my notes: -…

53

434

3K

1

26

196

Peter Conlon

@phc27x

10 months

@micahgoldblum Sounds right, isnt it what @guillefix and Ard Louis proposed here

Deep learning generalizes because the parameter-function map is...

Deep neural networks (DNNs) generalize remarkably well without explicit regularization even in the strongly over-parametrized regime where classical learning theory would instead predict that they...

arxiv.org

2

0

10

Micah Goldblum

@micahgoldblum

10 months

@phc27x @guillefix Yeah, this field of research has a long history, and we actually cited one of Ard's related papers. Interestingly, we found that neural networks (e.g. CNNs) can compress data from domains they were never designed for (e.g. tabular data).

1

Replies