The Yaser Abu-Mostafa Insight on the ICLR 2017 Best Paper: Why Do Deep Neural Networks Generalize Better?

Introduction

In the realm of machine learning (ML), one intriguing aspect of deep neural networks (DNNs) is their ability to generalize far better than expected theoretically. Yaser Abu-Mostafa, the renowned professor of machine learning at Caltech, sheds light on this phenomenon in his discussion of the International Conference on Learning Representations (ICLR) 2017 best paper, specifically under the context of his 'Learning from Data' CaltechX course on the edX platform.

Empirical Evidence vs. Theoretical Predictions

According to Abu-Mostafa, the empirical evidence about the generalization capability of DNNs with a large number of weights is that they perform better than what the theory predicts. This empirical observation extends beyond the mere loose bounds described by existing theories. It points to a difference between theory and practice—a difference that challenges our understanding of why these networks exhibit such robust generalization.

Historical Perspectives on Overfitting and Generalization

Abu-Mostafa references another important historical instance of this phenomenon: boosting. In ML, boosting techniques were known to generalize better than expected, showcasing a similar gap between empirical results and theoretical predictions. For instance, when the complexity of models increased, one would expect overfitting to occur, but in the case of boosting, this did not happen. A theoretical explanation based on a cost function other than in-sample error ((E_{in})) was proposed, but it did not withstand rigorous scrutiny. Specifically, directly minimizing the new cost function resulted in overfitting again.

Lack of Conclusive Explanation

Despite the success of boosting in avoiding overfitting, there was no definitive explanation for how it achieved this. Experts provided various intuitions, but these were often a mix of explanation and rationalization rather than a solid theoretical foundation. This echoes the current state of understanding regarding deep neural networks (DNNs).

Exploring DNN Generalization

Abu-Mostafa notes that while DNNs have also been studied in a variety of ways, the explanations often rely on alternative approaches to generalization. These approaches, such as understanding DNNs based on the stability of weights, do not lead to significant changes in the generalization performance. Instead, they focus on stability as a criterion for good generalization, akin to a form of regularization.

Potential Theoretical Breakthroughs

Given the vast number of parameters in DNNs, Abu-Mostafa suggests that identifying what specific structural aspects contribute to better generalization could be a significant theoretical advance. He emphasizes that the Vapnik-Chervonenkis (VC) theory, while providing upper bounds, does not violate the observed performance. Instead, the challenge lies in developing better generalization bounds that apply to models with a large number of parameters. Proving better generalization bounds for such models would be a major breakthrough.

Conclusion

In conclusion, despite the empirical success of DNNs, the theoretical understanding of why they generalize so effectively remains elusive. Abu-Mostafa's insights suggest that future research should focus on the structure of DNNs to uncover the underlying mechanisms that lead to better generalization. Such a breakthrough could greatly enhance our ability to design and optimize deep learning models for a wide range of applications.