Kernel-Based Smoothness Analysis of Residual Networks

Tom Tirer, Joan Bruna, Raja Giryes

Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, PMLR 145:921-954, 2022.

Abstract

A major factor in the success of deep neural networks is the use of sophisticated architectures rather than the classical multilayer perceptron (MLP). Residual networks (ResNets) stand out among these powerful modern architectures. Previous works focused on the optimization advantages of deep ResNets over deep MLPs. In this paper, we show another distinction between the two models, namely, a tendency of ResNets to promote smoother interpolations than MLPs. We analyze this phenomenon via the neural tangent kernel (NTK) approach. First, we compute the NTK for a considered ResNet model and prove its stability during gradient descent training. Then, we show by various evaluation methodologies that for ReLU activations the NTK of ResNet, and its kernel regression results, are smoother than the ones of MLP. The better smoothness observed in our analysis may explain the better generalization ability of ResNets and the practice of moderately attenuating the residual blocks.

Cite this Paper

BibTeX


@InProceedings{pmlr-v145-tirer22a,
  title = 	 {Kernel-Based Smoothness Analysis of Residual Networks},
  author =       {Tirer, Tom and Bruna, Joan and Giryes, Raja},
  booktitle = 	 {Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference},
  pages = 	 {921--954},
  year = 	 {2022},
  editor = 	 {Bruna, Joan and Hesthaven, Jan and Zdeborova, Lenka},
  volume = 	 {145},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {16--19 Aug},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v145/tirer22a/tirer22a.pdf},
  url = 	 {https://proceedings.mlr.press/v145/tirer22a.html},
  abstract = 	 {A major factor in the success of deep neural networks is the use of sophisticated architectures rather than the classical multilayer perceptron (MLP). Residual networks (ResNets) stand out among these powerful modern architectures. Previous works focused on the optimization advantages of deep ResNets over deep MLPs. In this paper, we show another distinction between the two models, namely, a tendency of ResNets to promote smoother interpolations than MLPs. We analyze this phenomenon via the neural tangent kernel (NTK) approach. First, we compute the NTK for a considered ResNet model and prove its stability during gradient descent training. Then, we show by various evaluation methodologies that for ReLU activations the NTK of ResNet, and its kernel regression results, are smoother than the ones of MLP. The better smoothness observed in our analysis may explain the better generalization ability of ResNets and the practice of moderately attenuating the residual blocks. }
}

Endnote

%0 Conference Paper
%T Kernel-Based Smoothness Analysis of Residual Networks
%A Tom Tirer
%A Joan Bruna
%A Raja Giryes
%B Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference
%C Proceedings of Machine Learning Research
%D 2022
%E Joan Bruna
%E Jan Hesthaven
%E Lenka Zdeborova	
%F pmlr-v145-tirer22a
%I PMLR
%P 921--954
%U https://proceedings.mlr.press/v145/tirer22a.html
%V 145
%X A major factor in the success of deep neural networks is the use of sophisticated architectures rather than the classical multilayer perceptron (MLP). Residual networks (ResNets) stand out among these powerful modern architectures. Previous works focused on the optimization advantages of deep ResNets over deep MLPs. In this paper, we show another distinction between the two models, namely, a tendency of ResNets to promote smoother interpolations than MLPs. We analyze this phenomenon via the neural tangent kernel (NTK) approach. First, we compute the NTK for a considered ResNet model and prove its stability during gradient descent training. Then, we show by various evaluation methodologies that for ReLU activations the NTK of ResNet, and its kernel regression results, are smoother than the ones of MLP. The better smoothness observed in our analysis may explain the better generalization ability of ResNets and the practice of moderately attenuating the residual blocks.

APA


Tirer, T., Bruna, J. & Giryes, R.. (2022). Kernel-Based Smoothness Analysis of Residual Networks. Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, in Proceedings of Machine Learning Research 145:921-954 Available from https://proceedings.mlr.press/v145/tirer22a.html.

Related Material

Download PDF