![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/ResBlock.png/220px-ResBlock.png)
A residual neural network (also referred to as a residual network or ResNet)[1] is a deep learning architecture in which the weight layers learn residual functions with reference to the layer inputs. It was developed in 2015 for image recognition and won that year's ImageNet Large Scale Visual Recognition Challenge (ILSVRC).[2][3]
As a point of terminology, "residual connection" or "skip connection" refers to the specific architectural motif of , where is an arbitrary neural network module. Residual connections had been used before ResNet, such as in the LSTM network[4] and the highway network[5]. However, the publication of ResNet made it widely popular, appearing in neural networks that are otherwise unrelated to ResNet.
The residual connection stabilizes the training and convergence of deep neural networks with hundreds of layers, and is a common motif in deep neural networks, such as Transformer models (e.g., BERT and GPT models such as ChatGPT), the AlphaGo Zero system, the AlphaStar system, and the AlphaFold system.
Mathematics
[edit]Residual connection
[edit]In a multi-layer neural network model, consider a subnetwork with a certain number of stacked layers (e.g., 2 or 3). Denote the underlying function performed by this subnetwork as , where is the input to the subnetwork. Residual learning re-parameterizes this subnetwork and lets the parameter layers represent a "residual function" . The output of this subnetwork is then represented as:
The operation of "" is implemented via a "skip connection" that performs an identity mapping to connect the input of the subnetwork with its output. This connection is referred to as a "residual connection" in later work. The function is often represented by matrix multiplication interlaced with activation functions and normalization operations (e.g., batch normalization or layer normalization). As a whole, one of these subnetworks is referred to as a "residual block".[1] A deep residual network is constructed by simply stacking these blocks together.
Importantly, the underlying principle of residual blocks is also the principle of the original LSTM cell,[4] a recurrent neural network that predicts an output at time as , which becomes during backpropagation through time.[6]
Projection connection
[edit]If the function is of type where , then is undefined. To handle this special case, a projection connection is used:where is typically a linear projection, defined by where is a matrix. The matrix is trained by backpropagation as any other parameter of the model.
Signal propagation
[edit]The introduction of identity mappings facilitates signal propagation in both forward and backward paths, as described below.[7]
Forward propagation
[edit]If the output of the -th residual block is the input to the -th residual block (assuming no activation function between blocks), then the -th input is:
Applying this formulation recursively, e.g.,
yields the general relationship:
where is the index of a residual block and is the index of some earlier block. This formulation suggests that there is always a signal that is directly sent from a shallower block to a deeper block .
Backward propagation
[edit]The residual learning formulation provides the added benefit of addressing the vanishing gradient problem to some extent. However, it is crucial to acknowledge that the vanishing gradient issue is not the root cause of the degradation problem, which is tackled through the use of normalization layers. To observe the effect of residual blocks on backpropagation, consider the partial derivative of a loss function with respect to some residual block input . Using the equation above from forward propagation for a later residual block :[7]
This formulation suggests that the gradient computation of a shallower layer, , always has a later term that is directly added. Even if the gradients of the terms are small, the total gradient resists vanishing thanks to the added term .
Variants of residual blocks
[edit]![](https://upload.wikimedia.org/wikipedia/commons/thumb/8/86/ResBlockVariants.png/220px-ResBlockVariants.png)
Basic block
[edit]A Basic Block is the simplest building block studied in the original ResNet.[1] This block consists of two sequential 3x3 convolutional layers and a residual connection. The input and output dimensions of both layers are equal.
Bottleneck block
[edit]A Bottleneck Block[1] consists of three sequential convolutional layers and a residual connection. The first layer in this block is a 1x1 convolution for dimension reduction, e.g., to 1/4 of the input dimension; the second layer performs a 3x3 convolution; the last layer is another 1x1 convolution for dimension restoration. The models of ResNet-50, ResNet-101, and ResNet-152 in [1] are all based on Bottleneck Blocks.
Pre-activation block
[edit]The Pre-activation Residual Block[7] applies the activation functions (e.g., non-linearity and normalization) before applying the residual function . Formally, the computation of a Pre-activation Residual Block can be written as:
where can be any non-linearity activation (e.g., ReLU) or normalization (e.g., LayerNorm) operation. This design reduces the number of non-identity mappings between Residual Blocks. This design was used to train models with 200 to over 1000 layers.[7]
Since GPT-2, the Transformer Blocks have been dominantly implemented as Pre-activation Blocks. This is often referred to as "pre-normalization" in the literature of Transformer models.[8]
Applications
[edit]![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/34/Transformer%2C_full_architecture.png/220px-Transformer%2C_full_architecture.png)
All Transformer architectures include residual connections. Indeed, very deep Transformer models cannot be successfully trained without Residual Connections.[9]
The original Residual Network paper made no claim on being inspired by biological systems. But later research has related ResNet to biologically-plausible algorithms.[10][11]
A study published in Science in 2023[12] disclosed the complete connectome of an insect brain (of a fruit fly larva). This study discovered "multilayer shortcuts" that resemble the skip connections in artificial neural networks, including ResNets.
History
[edit]Previous work
[edit]In 1961, Frank Rosenblatt described a three-layer multilayer perceptron (MLP) model with skip connections.[13] The model was referred to as a "cross-coupled system", and the skip connections were forms of cross-coupled connections.
During late 1980s, "skip-layer" connections were sometimes used in neural networks. Examples include [14][15]. An 1988 paper[16] trained a fully connected feedforward network where each layer residually connects to all subsequent layers, like the later DenseNet (2016).
![](https://upload.wikimedia.org/wikipedia/commons/thumb/9/93/LSTM_Cell.svg/300px-LSTM_Cell.svg.png)
Degradation problem
[edit]Sepp Hochreiter discovered the vanishing gradient problem in 1991[17] and argued that it explained why the then-prevalent forms of recurrent neural networks did not work for long sequences. He and Schrmidhuber later designed the long short-term memory (LSTM, 1997)[4][18] to solve this problem, which has a "cell state" that can function as a generalized residual connection. The highway network (2015)[5][19] applied the idea of an LSTM unfolded in time to feedforward neural networks, resulting in the highway network.
![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b5/Recurrent_neural_network_unfold.svg/220px-Recurrent_neural_network_unfold.svg.png)
During the early days of deep learning, there were attempts to train increasingly deep models. Notable examples included the AlexNet (2012) had 8 layers, and the VGG19 (2014) had 19 layers.[20] However, stacking too many layers led to a steep reduction in training accuracy,[21] known as the "degradation" problem.[1] In theory, adding additional layers to deepen a network should not result in a higher training loss, but this is exactly what happened with VGGNet.[1] If the extra layers can be set as identity mappings, though, then the deeper network would represent the same function as its shallower counterpart. This is the main idea behind residual learning, explained further below. It is hypothesized that the optimizer is not able to approach identity mappings for the parameterized layers.
In 2014, the state of the art was training “very deep neural network” with 20 to 30 layers.[22] The research team attempted to train deeper ones by empirically testing various tricks for training deeper networks until they discovered the deep residual network architecture.[23]
Subsequent work
[edit]DenseNet (2016)[24] connects the output of each layer to the input to each subsequent layer:
Neural networks with Stochastic Depth [25] were made possible given the Residual Network architectures. This training procedure randomly drops a subset of layers and lets the signal propagate through the identity skip connection. Also known as "DropPath", this is an effective regularization method for training large and deep models, such as the Vision Transformer (ViT).
References
[edit]- ^ a b c d e f g h He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (10 Dec 2015). Deep Residual Learning for Image Recognition. arXiv:1512.03385.
- ^ "ILSVRC2015 Results". image-net.org.
- ^ Deng, Jia; Dong, Wei; Socher, Richard; Li, Li-Jia; Li, Kai; Fei-Fei, Li (2009). "ImageNet: A large-scale hierarchical image database". CVPR.
- ^ a b c Sepp Hochreiter; Jürgen Schmidhuber (1997). "Long short-term memory". Neural Computation. 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735. PMID 9377276. S2CID 1915014.
- ^ a b Srivastava, Rupesh Kumar; Greff, Klaus; Schmidhuber, Jürgen (3 May 2015). "Highway Networks". arXiv:1505.00387 [cs.LG].
- ^ Szegedy, Christian; Ioffe, Sergey; Vanhoucke, Vincent; Alemi, Alex (2016). "Inception-v4, Inception-ResNet and the impact of residual connections on learning". arXiv:1602.07261 [cs.CV].
- ^ a b c d He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2015). "Identity Mappings in Deep Residual Networks". arXiv:1603.05027 [cs.CV].
- ^ Radford, Alec; Wu, Jeffrey; Child, Rewon; Luan, David; Amodei, Dario; Sutskever, Ilya (14 February 2019). "Language models are unsupervised multitask learners" (PDF). Archived (PDF) from the original on 6 February 2021. Retrieved 19 December 2020.
- ^ Dong, Yihe; Cordonnier, Jean-Baptiste; Loukas, Andreas (2021). "Attention is not all you need: pure attention loses rank doubly exponentially with depth". arXiv:2103.03404 [cs.LG].
- ^ Liao, Qianli; Poggio, Tomaso (2016). Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex. arXiv:1604.03640.
- ^ Xiao, Will; Chen, Honglin; Liao, Qianli; Poggio, Tomaso (2018). Biologically-Plausible Learning Algorithms Can Scale to Large Datasets. arXiv:1811.03567.
- ^ Winding, Michael; Pedigo, Benjamin; Barnes, Christopher; Patsolic, Heather; Park, Youngser; Kazimiers, Tom; Fushiki, Akira; Andrade, Ingrid; Khandelwal, Avinash; Valdes-Aleman, Javier; Li, Feng; Randel, Nadine; Barsotti, Elizabeth; Correia, Ana; Fetter, Fetter; Hartenstein, Volker; Priebe, Carey; Vogelstein, Joshua; Cardona, Albert; Zlatic, Marta (10 Mar 2023). "The connectome of an insect brain". Science. 379 (6636): eadd9330. bioRxiv 10.1101/2022.11.28.516756v1. doi:10.1126/science.add9330. PMC 7614541. PMID 36893230. S2CID 254070919.
- ^ Rosenblatt, Frank (1961). Principles of neurodynamics. perceptrons and the theory of brain mechanisms (PDF).
- ^ Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. (1986-10). "Learning representations by back-propagating errors". Nature. 323 (6088): 533–536. doi:10.1038/323533a0. ISSN 1476-4687.
{{cite journal}}
: Check date values in:|date=
(help) - ^ Venables, W. N.; Ripley, Brain D. (1994). Modern Applied Statistics with S-Plus. Springer. pp. 261–262. ISBN 9783540943501.
- ^ J, Lang K. (1988). "Learning to tell two spirals apart". Proceedings of the 1988 Connectionist Models Summer school: 52–59.
- ^ Hochreiter, Sepp (1991). Untersuchungen zu dynamischen neuronalen Netzen (PDF) (diploma thesis). Technical University Munich, Institute of Computer Science, advisor: J. Schmidhuber.
- ^ Felix A. Gers; Jürgen Schmidhuber; Fred Cummins (2000). "Learning to Forget: Continual Prediction with LSTM". Neural Computation. 12 (10): 2451–2471. CiteSeerX 10.1.1.55.5709. doi:10.1162/089976600300015015. PMID 11032042. S2CID 11598600.
- ^ Srivastava, Rupesh Kumar; Greff, Klaus; Schmidhuber, Jürgen (22 July 2015). "Training Very Deep Networks". arXiv:1507.06228 [cs.LG].
- ^ Simonyan, Karen; Zisserman, Andrew (2015-04-10), Very Deep Convolutional Networks for Large-Scale Image Recognition, doi:10.48550/arXiv.1409.1556, retrieved 2024-08-07
- ^ He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2016). "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification". arXiv:1502.01852 [cs.CV].
- ^ Simonyan, Karen; Zisserman, Andrew (2015-04-10), Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv:1409.1556
- ^ Linn, Allison (2015-12-10). "Microsoft researchers win ImageNet computer vision challenge". The AI Blog. Retrieved 2024-06-29.
- ^ Huang, Gao; Liu, Zhuang; van der Maaten, Laurens; Weinberger, Kilian (2016). Densely Connected Convolutional Networks. arXiv:1608.06993.
- ^ Huang, Gao; Sun, Yu; Liu, Zhuang; Weinberger, Kilian (2016). Deep Networks with Stochastic Depth. arXiv:1603.09382.