ResNets are a common neural network architecture used for deep learning computer vision applications like object detection and image segmentation.
ResNet can contain a large number of convolutional layers, commonly between 18-152, but supporting up to thousands of layers. There are newer variants called ResNext and DenseNet, which are more commonly used today. But understanding the original ResNet architecture is key to working with many common convolutional network patterns.
Pytorch is a Python deep learning framework, which provides several options for creating ResNet models:
In this article, you will learn:
Residual Network (ResNet) is a Convolutional Neural Network (CNN) architecture that overcame the “vanishing gradient” problem, making it possible to construct networks with up to thousands of convolutional layers, which outperform shallower networks.
Related content: read our guide to Pytorch CNN
A vanishing gradient occurs during backpropagation. When the neural network training algorithm tries to find weights that bring the loss function to a minimal value, if there are too many layers, the gradient becomes very small until it disappears, and optimization cannot continue.
ResNet solved the problem using “identity shortcut connections”. It operates in two stages:
Residual blocks are the essential building blocks of ResNet networks. To make very deep convolutional structures possible, ResNet adds intermediate inputs to the output of a group of convolution blocks. This is also called skip connections, identity mapping, and “residual connections.
The objective of skip connections is to allow smoother gradient flow, and ensure important features carry until the final layers. They do not add computational load to the network.
The following diagram illustrates a residual block, where:
PyTorch lets you run ResNet models, pre-trained on the ImageNet dataset. This is called “transfer learning”—you can make use of a model trained on an existing dataset, saving the time and computational effort of training it again on your own examples.
To import pre-trained ResNet into your model, use this code:
This will create ResNet18, an architecture with 18 convolutional layers.
If you need a different number of layers, substitute 18 with 34, 50, 101, or 152—these are the number of layers supported in the torchvision.models module.
You can use the pretrained flag to define if you want the ResNet network to leverage transfer learning or not:
PyTorch lets you customize the ResNet architecture to your needs. Liu Kuang provides a code example that shows how to implement residual blocks and use them to create different ResNet combinations.
Here is how to create a residual block for ResNets under 50 layers:
Kuang provides a different class called bottleneck to define the residual blocks for more than 50 layers—with expansion set to 4 and several other parameter changes.
He shows how to build the ResNet architecture with four persistent layers, defined like this:
And a make_layer function to create a variable number of additional residual layers.
Finally, you can define any ResNet architecture like this:
return ResNet(Bottleneck, [3, 8, 36, 3])
See the complete code example.
ResNet is a computing intensive neural network architecture. Run:AI automates resource management and workload orchestration for machine learning infrastructure. With Run:AI, you can automatically run as many compute intensive experiments as needed in PyTorch and other deep learning frameworks.
Here are some of the capabilities you gain when using Run:AI:
Run:AI simplifies machine learning infrastructure pipelines, helping data scientists accelerate their productivity and the quality of their models.
Learn more about the Run:AI GPU virtualization platform.