{"id":22702,"date":"2021-03-23T08:14:00","date_gmt":"2021-03-23T08:14:00","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/deeper-neural-networks-lead-to-simpler-embeddings\/"},"modified":"2023-08-29T11:23:33","modified_gmt":"2023-08-29T11:23:33","slug":"deeper-neural-networks-lead-to-simpler-embeddings","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/deeper-neural-networks-lead-to-simpler-embeddings\/","title":{"rendered":"Deeper Neural Networks Lead To Simpler Embeddings"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

A surprising explanation for generalization in neural networks<\/p>\n

Recent research is increasingly investigating how neural networks, being as hyper-parametrized as they are, generalize. That is, according to traditional statistics, the more parameters, the more the model overfits. This notion is directly contradicted by a fundamental axiom of deep learning:<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Increased parametrization improves generalization.<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Although it may not be explicitly stated anywhere, it\u2019s the intuition behind why researchers continue to push models larger to make them more powerful.<\/p>

There have been many efforts to explain exactly why this is so. Most are quite interesting; the recently proposed\u00a0Lottery Ticket Hypothesis<\/a>\u00a0states that neural networks are just giant lotteries finding the best subnetwork, and\u00a0another paper<\/a>\u00a0suggests through theoretical proof that such phenomenon is built into the nature of deep learning.<\/p>

Perhaps one of the most intriguing, though, is one proposing that deeper neural networks lead to simpler embeddings. <\/em>Alternatively, this is known as the \u201csimplicity bias\u201d \u2014 neural network parameters have a bias towards simpler mappings.<\/p>\n

Minyoung Huh et al proposed in a recent paper,\u00a0\u201cThe Low-Rank Simplicity Bias in Deep Networks\u201d<\/a>, that depth increases the proportion of simpler solutions in the parameter space. This makes neural networks more likely \u2014 by chance \u2014 to find simple solutions rather than complex ones.<\/p>\n

On terminology: the authors measure the \u201csimplicity\u201d of a matrix based on its rank \u2014 roughly speaking, a measurement of how linearly independent parts of the matrix are on other parts. A higher rank can be considered more complex since its parts are highly independent and thus contain more \u201cindependent information\u201d. On the other hand, a lower rank can be considered simpler.<\/p>\n

Huh et al begin by analyzing the rank of linear <\/em>networks \u2014 that is, networks without any nonlinearities, like activation functions.<\/p>\n

The authors trained several linear networks of different depths on the MNIST dataset. For each network, they randomly drew 128 neural<\/a> network weights and the associated kernel, plotting their ranks. As the depth increases, the rank of the network\u2019s parameters decreases.<\/p>\n\n\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Deeper\t\t\t\t\t\t\t\t\t\t\t
Source:\u00a0Huh et al.<\/a><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

This can be derived from the fact that the rank of the product of two matrices can only decrease or remain the same from the ranks of each of its constituents. If we get a little bit more abstract, we can think of this intuitively as: each matrix contains its own independent information, but when they are combined, the information of one matrix can only get muddled and entangled with the information in another matrix.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

rank(AB) \u2264 min (rank(A), rank(B))<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

What is more interesting, though, is that the same applies to nonlinear<\/em> networks. When nonlinear activation functions like tanh or ReLU are applied, the pattern is repeated: higher depth, lower rank.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Effective\t\t\t\t\t\t\t\t\t\t\t
Source:\u00a0Huh et al.<\/a><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The authors also performed hierarchical cluster kernels for different depths of the network for the two nonlinearities (ReLU and tanh). As the depth increases, the presence of block structures shows decreasing rank. The \u201cindependent information\u201d each kernel carries decreases with depth.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Deeper\t\t\t\t\t\t\t\t\t\t\t
Source:\u00a0Huh et al.<\/a><\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Hence, although it may seem odd, over-parametrizing a network acts as implicit regularization<\/em><\/mark>. This is especially true with linearities; thus, a model\u2019s generalization can be improved by increasing linearities.<\/p>\n

In fact, the authors find that on both the CIFAR-10 and CIFAR-100 datasets, linearly expanding the network increases accuracy by 2.2% and 6.5% from a baseline simple CNN. On ImageNet, linear over-parametrization of AlexNet increases accuracy by 1.8%; ResNet10 accuracy increases by 0.9%, and ResNet18 accuracy increases by 0.4%.<\/p>\n

This linear over-parametrization \u2014 which is not adding any serious learning capacity<\/em> to the network, only more linear transformations \u2014 performs even better than explicit regularizers, like penalties. Moreover, this implicit regularization does not change the objective being minimized<\/em>.<\/p>\n

Perhaps what is most satisfying about this contribution is that it is in agreement with Occam\u2019s razor \u2014 a statement many papers have questioned or outright refuted as being relevant to deep learning.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The simplest solution is usually the right one.
– Occam\u2019s razor<\/p><\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Indeed, much of prior discourse has taken more parameters to mean more complex, leading many to assert that Occam\u2019s razor, while being relevant in the regime of classical statistics, does not apply to over parametrized spaces.<\/p>\n

This paper\u2019s fascinating contribution argues instead that simpler solutions are in fact better, and that more successful, highly parameterized neural networks arrive at those simpler solutions because<\/em>, not despite, their parametrization.<\/p>\n

Still, as the authors term it, this is a \u201cconjecture\u201d \u2014 it still needs refinement and further investigation. But it\u2019s a well-founded, and certainly interesting, one at that \u2014 perhaps with the potential to shift the conversation on how we think about generalization in deep learning.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

Recent research is increasingly investigating how neural networks, being as hyper-parametrized as they are, generalize. That is, according to traditional statistics, the more parameters, the more the model overfits.<\/p>\n","protected":false},"author":884,"featured_media":19018,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[97,206,1446,1447,92],"ppma_author":[3782],"class_list":["post-22702","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-artificial-intelligence","tag-deep-learning","tag-deeper-neural-networks","tag-embeddings","tag-machine-learning"],"authors":[{"term_id":3782,"user_id":884,"is_guest":0,"slug":"andre-ye","display_name":"Andre Ye","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/08\/Andre-Ye-150x150.jpg","user_url":"https:\/\/www.critiq.tech\/","last_name":"Ye","first_name":"Andre","job_title":"","description":"Andre Ye is Cofounder at Critiq, and Editor and Writer at Medium"}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22702","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/884"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=22702"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22702\/revisions"}],"predecessor-version":[{"id":31828,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22702\/revisions\/31828"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/19018"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=22702"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=22702"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=22702"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=22702"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}