2.1.

LeNet

Lecun et al.⁶ proposed a CNN named LeNet-5. The network model of LeNet-5 is shown in Fig. 1. According to Fig. 1, CNN architecture is generally composed of convolution layers, subsampling (pooling) layers, and fully connected layers. The following three sections are explained in turn.

Fig. 1

LeNet-5 model diagram.⁶

2.1.1.

Convolution layer

The convolution layer consists of multiple feature maps, which are obtained by convolution of the convolution kernel with the input signal. Each convolution kernel is a weight matrix, which can be a $3\times 3$ or $5\times 5$ matrix for a two-dimensional (2-D) image of a single channel. Figure 2 illustrates an example of the 2-D convolution.

Fig. 2

An example of a 2-D convolution.

The convolution operation provides a way to process variable-size inputs using convolution kernels, and different input features are extracted through convolution operation in convolution layer. The first layer extracts lower-level features such as edges, end points, and corners. Then the higher layer extracts more complex and higher-level features by processing the lower-level features. Convolution layer mainly has the characteristics of sparse interactions and weight sharing.

Sparse interactions

Traditional neural networks use matrix multiplication to build connections between inputs and outputs. Each output unit interacts with each input unit. When an input image contains thousands of pixels, this connection will increase the storage requirements of the model and increase the amount of calculation. Different from the traditional connection, the convolution network has the characteristic of sparse interactions (also known as sparse connectivity), which is achieved by controlling the size of the convolution kernel far less than the size of the input. The graphical interpretation of the sparse connections is shown in Fig. 3. In this figure, the input unit $X_3$ and the output unit affected by $X_3$ are highlighted. If there are $m$ inputs and $n$ outputs, the fully connected form of the model requires $m\times n$ parameters and the complexity of the corresponding algorithm is $O(m\times n)$. In sparse connection, the connections number of per output is $k(k\ll m)$, so this connection only needs $k\times n$ parameters and the complexity of the corresponding algorithm is $O(k\times n)$. The sparse interaction of convolution layer not only reduces the storage requirements of the model but also requires less computation to obtain the output, thus improving the efficiency of the model.

Fig. 3

Sparse connections: (a) when $\mathit{S}$; is generated by convolution with a kernel width of 3, only three outputs are affected by $X_3$ and (b) when using the fully connected form, the connection is no longer sparse, and all outputs are affected by $X_3$.

Weight sharing

The convolution layer also has the characteristic of weight sharing, which is realized by the convolution kernel. Convolution kernels are used to control the number of parameters and to impose a spatially restricted weighting to handle variable-size inputs. Weight sharing means that units in a layer use the same weights and deviations. For example, the C1 layer of LeNet-5 is a convolution layer, which is obtained through the calculation of six convolution kernels, and each convolution kernel has a fixed weight when convolving with the previous layer. When the input is a single-channel signal, the C1 layer contains six convolution kernels with the size of $1\times 5\times 5$. If the bias is taken into account, the C1 layer contains a total of $(63\times 5\times 5+6)=156$ parameters. Compared with the fully connected network architecture, the weight sharing reduces the network training parameters to a greater extent, which can effectively prevent the network overfitting caused by a large number of parameters and improve the efficiency of network operation.

2.1.2.

Subsampling layer

Usually, a subsampling (pooling) layer is inserted periodically between the convolution layers, whose function is to gradually reduce the spatial size of the data, so as to reduce the number of parameters in the network and reduce the consumption of computing resources. The pooling layer can also learn some invariant features of the input. Commonly used pooling layer methods are global average pooling²⁷ and max pooling. The input data processed by the pooling layer is generally a feature map obtained after convolution operation. The most commonly used max pooling layer is shown in Fig. 4. It can be seen that the max pooling unit is only sensitive to the surrounding maximum, not to the exact location. Therefore, by pooling the obtained features, we can learn some invariant features of the input. In LeNet-5, the max pooling layer mainly uses a spatial window with a size of $2\times 2$ and a step size of 2 to convolute. The maximum value in this window is taken as the output result.

Fig. 4

Schematic diagram of the max pooling layer.

2.2.

AlexNet

Due to insufficient hardware computing and data, LeNet-5 did not attract enough attention after it was proposed. With the development of computer hardware and the increase in the amount of data available for neural network training, in 2012, AlexNet network¹³ won the ILSVRC-2012 image classification competition¹⁵ with a far lower error rate than the second place. Since then, deep neural networks have begun to attract widespread attention. The structure of AlexNet is shown in Fig. 5. Compared with LeNet-5, the improvements of AlexNet network architecture are as follows:

Fig. 5

AlexNet structure diagram.

AlexNet is a milestone in the development of deep CNN, which has caused a new wave of neural network research. The success of AlexNet mainly depends on the development of computer hardware and the enhancement of data sets.

2.3.

ZFNet

After AlexNet achieved excellent results in the ImageNet image classification competition, researchers began to study the CNN more deeply. However, there is no clear theoretical explanation for why a CNN model can perform well. Zeiler and Fergus³¹ proposed a visualization technique to understand CNNs and proposed ZFNet. The network has made minor improvements on AlexNet; the main contribution of Ref. 31 is to explain to a certain extent why CNNs are effective and how to improve network performance. The main contributions are detailed as follows:

The ZFNet is shown in Fig. 6. It changed the size of the convolution kernel in AlexNet’s first layer from $11\times 11$ to $7\times 7$ and changed the step size of the convolution kernel from 4 to 2. Comparing ZFNet model with AlexNet single model, the error rate of top-5 is reduced by 1.7%,³¹ which confirms the correctness of this improvement.

Fig. 6

ZFNet structure diagram.³¹

2.4.

VGG-16/19

The shallow neural network model has certain limitations in large-scale image recognition tasks. In order to further explore the performance of the deeper network model, Simonyan and Zisserman²⁰ proposed the VGG. The main contribution of VGG is a thorough evaluation of networks of increasing depth using an architecture with very small ($3\times 3$) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16 to 19 weight layers. Simonyan and Zisserman²⁰ mentioned six different network configurations and compared them on the ImageNet dataset. The configuration information of convolution network is shown in Table 1, and the performance of the corresponding network model is shown in Table 2.

Table 1

ConvNet configuration.20

Table 2

ConvNet performance at a single test scale.20

Unlike AlexNet and ZFNet, VGG uses a small convolution kernel of $3\times 3$ throughout the construction of the network and superimposes deep networks by superposing $3\times 3$ small convolution kernels. In the experiment, in order to keep the computational complexity of the constituent structures at each feature layer roughly consistent, the number of convolution kernels at the next layer is doubled when the size of the feature map is reduced by half through the max pooling layer. The various configurations in Table 1 almost have the same number of parameters, and Table 2 shows the results of various VGGNets in a single-scale test. The results show that the VGG-19 model achieved the best results, with an error rate of 8.0%. This also confirms that increasing network depth is beneficial to improve the accuracy of image classification. At the same time, it can be found that the result of A-LRN in Table 2 is worse than that of A. This also shows that the effect of LRN on classification results is not beneficial. With the introduction of batch normalization (BN),³² LRN is replaced already.

The innovation of VGG is mainly the application of $3\times 3$ small convolution kernels. The receptive field of two $3\times 3$ convolutions is equivalent to that of a $5\times 5$ convolution (as shown in Fig. 7), and the receptive field of three $3\times 3$ convolutions is equivalent to that of a $7\times 7$ convolution. The network used three $3\times 3$ convolutions instead of a $7\times 7$ convolution for two main reasons: First, it contains three ReLU layers instead of one, making the decision function more discriminatory; second, it can reduce the number of parameters. For example, if the input and output both have C channels, $3\times (3\times 3\times C\times C)=27\times C\times C$ parameters are required for three convolution layers of $3\times 3$, and $7\times 7\times C\times C=49\times C\times C$ parameters are required for one convolution layer of $7\times 7$.

Fig. 7

A stack of two $3\times 3$ convolutions replacing the $5\times 5$ convolutions.

Before VGG, An et al.³³ also used small convolution kernels for experiments, but the network was not as deep as VGG and was not tested on large-scale ImageNet datasets. Using small convolution kernels, VGG can make the CNN reach a depth of 19 layers. In the ILSVRC-2014 image classification competition, VGG took the second place with a 7.3% (Ref. 20) top-5 error rate, this also confirms the benefits of neural network depth for neural network classification results.

2.5.

GoogLeNet/Inception v1 to v3

GoogLeNet and VGG were the winner and runner-up of the ILSVRC-2014 image classification competition. VGG built a deeper network model through the construction of small convolution kernels, and GoogLeNet was inspired by network in network²⁷ to broaden the network structure and skillfully proposed the inception module.²¹ The network with the inception module allowed the model to better describe the input data content while further increasing the depth and width of the network model. The inception module has been constantly updated and improved since it was proposed. The different versions of the inception modules are described as follows.

2.5.1.

Inception v1

The biggest highlight of inception v1 is the introduction of $1\times 1$ convolution kernel inspired by network in network.²⁷ The structure of inception v1 is shown in Fig. 8.

Fig. 8

Inception v1 module.²¹

As can be seen from Sec. 2.1, one function of the convolution layer is to reduce and increase the dimension via using the number of channels (filters) in the convolution layer. In inception v1, the dimension is reduced mainly by $1\times 1$ convolution kernel, which can reduce the number of network parameters and feature maps. The input feature maps are convoluted by $1\times 1$ convolution kernel. This operation is equivalent to the original image scale transformation under the condition of unchanged size, which can greatly improves the accuracy of image classification. Inception v1 also uses convolution kernels of $1\times 1$, $3\times 3$, and $5\times 5$, which also increases the adaptability of the network to the scale transformation of the input image.

The GoogLeNet constructed by inception v1 is shown in Fig. 9. Compared with VGG, GoogleNet has 22 layers, and the network is deeper and wider. GoogLeNet took the first place in the ILSVRC-2014 image classification competition with a 6.7% (Ref. 21) top-5 error rate.

Fig. 9

GoogLeNet structure diagram.²¹

2.5.2.

Inception v2

The architecture of inception v2, as shown in Fig. 10, is mainly updated on the basis of inception v1 from the following aspects:

• (1) BN³² layer is added to normalize the output of each layer to a $N$ (0, 1) Gaussian distribution so that the network can be converges faster and can be initialized more freely. BN layer not only increases the robustness of the model but also reduces the use of dropout as a regularization technique.

• (2) In the model, a stack of two $3\times 3$ convolution kernels are used to replace $5\times 5$ convolution kernels in the inception v1 module, thus increasing the network depth. The overall depth of the 22-layer GoogLeNet built with inception v2 module has been increased by 9 layers.

Fig. 10

Inception v2 module.²²

Inception v2 architecture on the ImageNet test data set yielded a top-5 error rate of 4.9%,²² which was lower than the 4.94% top-5 error rate of PReLU proposed by He et al.³⁴ in the same time. PReLU’s top-5 error rate of 4.94% was the first to surpass human-level performance (5.1%)¹⁵ on the visual recognition challenge.

2.5.3.

Inception v3

The architecture of inception v3²² is shown in Fig. 11. It is mainly updated on the basis of inception v2 as follows:

• (1) Spatial factorization into asymmetric convolutions: Using a $3\times 1$ convolution followed by a $1\times 3$ convolution is equivalent to sliding a two-layer network with the same receptive field as in a $3\times 3$ convolution (see Fig. 12). If the number of input and output filters is equal, the two-layer solution is $\frac{[9-(3+3)]}{9}=33\%$ cheaper than a single convolution layer. In theory, if one can replace any $n\times n$ convolution by a $1\times n$ convolution followed by a $n\times 1$ convolution and the computational cost saving increases dramatically as $n$ grows.

• (2) The network width has increased, and the network input has changed from $224\times 224$ to $299\times 299$.

Fig. 11

Inception v3 module.²²

Fig. 12

Spatial factorization into asymmetric convolutions diagram.

Inception v3 module obtained 3.58% (Ref. 21) top-5 error rate of on the ImageNet test set.

2.6.

ResNets

2.6.1.

ResNet

It can be found from the above development of various CNN models that increasing the depth and width of neural network can improve the network performance. For example, VGG greatly improves network performance by adding network depth to AlexNet. For the original network such as VGG, simply increasing the depth will lead to vanishing/exploding gradients. He et al.²³ pointed out that the problem of vanishing gradients has been largely addressed by normalized initialization¹² and intermediate normalization layers. Although it is possible to train dozens of layers of networks by the above method, another problem arises, i.e., degradation problems. As shown in Fig. 13, when the number of network layers increased, the accuracy of training set was saturated or even decreased. This cannot be interpreted as overfitting, as overfit should be better in the training set. The degradation problem shows that deep networks cannot be optimized easily and well.

Fig. 13

Training error and test error on CIFAR-10 with 20-layers and 56-layers “plain” networks.²³

He et al.²³ proposed the ResNet in order to solve the above problems. The main contribution of ResNet is to solve the side effects (degradation) caused by increasing network depth so that network performance can be improved by simply increasing network depth. ResNet constructed by residual learning blocks can break through a 100-layers barrier and even reach 1000 layers.

The ResNet is mainly composed of the residual learning block, as shown in Fig. 14. In the residual learning block in Fig. 14, assuming the original function to be learned is $H(x)$, the residual learning block is then converted to $F(x)=H(x)-x$. These two expressions have the same effect, but the difficulty of optimization is different. To the extreme, if an identity mapping was optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.²³ The addition connection of the identity mapping does not add additional parameters and computation to the network but can greatly increase the training speed of the model and improve the training effect.

Fig. 14

Residual learning block.²³

Two residual blocks are used in the ResNet network structure for ImageNet. One is to concatenate two convolution kernels of size $3\times 3$ as one residual block shown in Fig. 15(left), and the other is to connect the $1\times 1$, $3\times 3$, and $1\times 1$ kernels together as a “bottleneck” building block shown in Fig. 15(right). In the “bottleneck” building block, the first $1\times 1$ convolution kernel mainly reduces the dimension of the feature map from 256-dimensional to 64-dimensional. Next, the convolution kernel of $3\times 3$ is used for calculation, and finally the data dimension is changed to 256 using the convolution kernel of $1\times 1$.

Fig. 15

A deeper residual function F for ImageNet.

ResNet, constructed by residual learning block, won the first place in the ILSVRC-2015 image classification competition with a top-5 error rate of 3.57%.²³ As the number of layers increases, ResNet solves the “degradation” problem well, as shown in Figs. 16 and 17.

Fig. 16

Results of ResNet classification on CIFAR-10.²³ Dashed lines denote training error and bold lines denote testing error.

Fig. 17

Results of ResNet classification on ImageNet.²³ Thin curves denote training error and bold curves denote validation error of the center crops.

2.6.2.

Improvement of ResNet

Soon after ResNet put forward, He et al.²⁴ further studied the identity mapping in the residual learning block and improved it. The method is compared with the original ResNet and Highway networks.^23,35 The results further confirm the importance of identity mapping. The main contributions of Ref. 24 are as follows:

• (1) By comparing the original shortcut connection with other shortcut connections as shown in Fig. 18, the importance of identity mapping is demonstrated further.

• (2) Change the location of the ReLU activation function in the residual learning block and preactivate it as shown in Fig. 19 (e) so that the output of the residual learning block is still an identity mapping.

Fig. 18

Various types of shortcut connections: (a) Original, (b) constant scaling, (c) exclusive gating, (d) short-cut only gating, (e) conv shortcut, and (f) dropout shortcut.²⁴

Fig. 19

Various usages of activation: (a) original, (b) BN after addition, (c) ReLU before addition, (d) ReLU-only preactivation, and (e) full preactivation.²⁴

Among the various types of shortcut connections shown in Fig. 18, the network consisting of the original shortcut connections achieved a 6.61% (Ref. 24) error rate on the CIFAR-10 data set, which is better than the other connections. This confirms the importance of the identity mapping. In the experiment,²⁴ among the various usages of activation, the best classification results were obtained using a full preactivated connection [Fig. 19(e)].

2.6.3.

Other residual networks

With the increasing depth of residual networks, the diminishing feature reuse will make the networks training very slow.³⁶ In order to reduce the impact of “feature disappearance,” Zagoruyko and Komodakis³⁷ proposed a wide-dropout block, as shown in Fig. 20(d). This block makes it possible to increase the depth of the original residual network by increasing the network width. The experiment also proves its feasibility.

Fig 20

Wide-dropout block: (a) basic, (b) bottleneck, (c) basic-wide, and (d) wide-dropout.³⁷

ResNeXt, proposed by Xie et al.,³⁸ puts forward the concept of cardinality beyond depth and width, and points out that increasing cardinality is more effective than increasing the depth and width. The residual learning block of ResNext is shown in Fig. 21. ResNeXt secured second place in ILSVRC-2016 image classification competition with 3.03% top-5 error rate.³⁸ In addition, Szegedy et al.³⁹ proposed inception v4 by combining inception module with residual learning block and constructed inception-ResNet-v2 network. The network achieves good results in the ILSVRC-2016 image classification competition with a top-5 error rate of 3.08%.³⁹

Fig. 21

Equivalent building blocks of ResNeXt.³⁸ (a) Aggregated residual transformations. (b) a block equivalent to (a), implemented as early concatenation. (c) a block equivalent to (a, b), implemented as grouped convolutions.

2.7.

DenseNet

Since ResNet was put forward, many networks have been developed using ResNet. Each network has its own characteristics and its performance has been improved. As the depth of CNNs increases, the input or gradient must passes through many layers, which will vanish and “wash out” when it reaches the end (or beginning) of the network.²⁵ This has aroused people’s rethinking of the network structure. Before the dense block was put forward, Huang et al.⁴⁰ trained deep network with stochastic depth to achieve good results. This shows that some network layers in the residual network carry unnecessary information for classification results, which can be discarded in training.

Based on Ref. 40, considering create short paths from early layers to later layers, Huang et al.²⁵ proposed DenseNet, which is mainly composed of dense blocks as shown in Fig. 22.

Fig. 22

Dense block.²⁵

There are $L$ connections with the traditional $L$ layer neural network, and the dense block of $L$ layer has $\frac{L\times (L+1)}{2}$ connections. The network setup growth rate $k$ indicates the added number of input channels when pass through a layer. For example, assuming that $K_0$ as the number of input feature maps, and the output of each nonlinear transformation $H$ is $k$ feature maps, then the input of the $i$’th layer is $K_0+(i-1)\times k$. One major difference between DenseNet and the previous mentioned networks is that DenseNet can accept fewer feature maps as the output of the network layer. DenseNet is constructed mainly by dense blocks, as shown in Fig. 23. In the same dense block, the feature size is required to be the same size. The transition layers are set between different dense blocks to achieve down sampling.

Fig. 23

DenseNet structure diagram.²⁵

The main advantage of DenseNet is that the features extracted by some earlier layers can still be directly used by deeper layers through dense connections. Through the setting of the growth rate $k$, DenseNet can adjust the number of feature maps, thus effectively reducing the number of parameters.

DenseNet outperformed ResNet on the CIFAR-10 dataset, and on the ImgeNet dataset, DenseNet was able to converge faster by increasing the number of layers.²⁵ DenseNet is also widely used as a commonly used neural network model today.

3.1.

Activation Function

Before the ReLU activation function, the traditional neural network mostly uses sigmoid as the activation function. In general, sigmoid functions can be divided into logistic sigmoid and tanh sigmoid. The sigmoid function in this paper generally refers to the former, as shown in Fig. 24.

Fig. 24

Sigmoid and tanh function diagram.

The output value of the sigmoid function is between 0 and 1, which is consistent with the definition of probability output. The nonlinear sigmoid function is widely used in the activation function because of its large signal gain in the central region and small signal gain on both sides, similar to the excitation and suppression states of neurons. However, when the number of neural network layers increases, the sigmoid gradient value will gradually become smaller, network learning becomes very slow, and even the gradient will vanish. Therefore, the network cannot be deepened indefinitely until the ReLU function is presented.

ReLU is the activation function used by many current network models. ReLU has the following advantages over sigmoid: unilateral inhibition, relatively wide excitation boundaries, sparse activation, and alleviate the vanishing-gradient problems.

Leaky ReLU⁴¹ improved the negative half axis of ReLU function to avoid zero gradient, but the experimental results were not greatly improved. He et al.³⁴ put forward the PReLU function on this basis, as shown in Fig. 25. The learnable parameter $a$ is added to PReLU. When $a=0$, PReLU becomes the ReLU function, and when $a=0.01$, PReLU becomes the leaky ReLU. Experiments show that this adaptive activation function can improve the classification results of the network.

Fig. 25

ReLU vs. PReLU.³⁴ (a) ReLU function and (b) PReLU function.

3.2.

Normalization

In the training process, when the input distribution in the hidden layer of the deep neural network is offset, the global distribution will gradually approach the upper and lower bounds of the value range of the nonlinear function, resulting in slow training convergence. Therefore, the network needs to be normalized.

The proposition of BN³² has a milestone significance in the field of deep learning. BN takes minibatch as the unit to unify the input distribution of the nonlinear function into a standard normal distribution with a mean of 0 variance of 1, which makes the input value of activation function to fall in the region where the nonlinear function is sensitive. BN improves the speed of training, accelerates the convergence process, and improves the classification results. Moreover, BN can be seen as a regularization technique that prevents overfitting, similar to dropout.³⁰ The addition of BN in the network also makes the network initialization less demanding.

The disadvantage of BN is that the network is dependent on minibatch dimension, and the change of batch dimension will affect the classification effect. When the batch size is small, the network effect of using BN layer is obviously worse as shown in Fig. 26. This does not satisfy some networks that require batch size 1 or 2 for other visual recognition tasks.^16,42,43

Fig. 26

Experimental results based on ImageNet dataset.

To alleviate this problem, layer normalization,⁴⁴ instance normalization,⁴⁵ group normalization,⁴⁶ and other normalization methods have been proposed. Figure 26 confirms that group normalization computation accuracy is more stable than BN when batch size changes. Figure 27 is a schematic diagram of various normalization methods.

Fig. 27

All kinds of standardized schematic diagram.

Normalization is an indispensable part of the modern convolution network architecture. It has made a vital contribution to the development of CNNs.

3.3.

Other Strategies

In the development of a CNN in image classification field, the improvement of some network initialization methods has also played a positive role. Network initialization is to ensure that the activation value of each layer does not appear saturation when the network is initialized, and the activation value of each layer is not 0. Sutskever et al.⁴⁷ proposed a Xavier initialization method to solve the network initialization problem. AlexNet¹³ used random initialization for network training, and VGGNet²⁰ initialized the deep network by initializing the shallow model first and then applying its parameters to the deeper model. Ioffe and Szegedy³² proposed BN and He et al.³⁴ proposed Microsoft Research Asia (MSRA) initialization method, they all considered the nonlinear ReLU function situation, and the deep neural network initialization problem was solved more effectively.

In addition to network initialization, the innovation of optimization method has also promoted the development of CNN. The optimization algorithm develops from stochastic gradient descent (SGD) to gradient descent with momentum⁴⁷, and then to Adam with adaptive learning rate,⁴⁸ which is widely used nowadays. In the latest work, Reddi et al.⁴⁹ explained how the exponential moving average used in Adam leads to nonconvergence through a simple convex optimization problem and proposed a beyond Adam algorithm.

5.1.

Application of Transfer Learning

In the application of deep neural network, when we are faced with a large amount of data, it takes a lot of calculation and time to train the model and optimize the parameters after building the deep neural network model. If a model that has been trained for a large amount of time can solve the same kind of problems, then the cost performance of the model will be greatly improved, which promotes the use of transferable model to solve the same kind of problems.

Zeiler and Fergus³¹ used a CNN for pretraining on ImageNet data sets, and then migrated the network to caltech-101 and caltech-256 for image classification data sets, respectively, for training and testing. The accuracy of image classification was improved by about 40%. Through transfer learning, we can apply a well-trained model to solve the similar problems by making small adjustments and achieve good results. At the same time, we can effectively solve the problem with less original data by adopting transferable model. Using transfer learning, the network model in image classification can be further applied to semantic segmentation, object detection, and other fields. In recent years, many researchers have devoted themselves to the field of transfer learning.

5.2.

Introduction of Visual Attention Mechanism

In recent years, attention mechanism has been adopted in the field of deep learning. Visual attention mechanism is a special brain signal processing mechanism of human vision. By rapidly scanning the whole image, human vision can obtain the target area that needs to be paid attention to, then devote more attention resources to this area to obtain more detailed information of the target, and inhibit other useless information. Hu et al.⁵² introduced attention mechanism to construct squeeze-and-excitation module, which reconstructs the relationship between feature channels by embedding multiscale processing. SENet won the ILSVRC 2017 image classification championship with a top-5 test set error rate of 2.251%.

In the future, the design of CNN framework can seek for introducing and strengthening attention mechanism in different layers to make computer vision closer to human visual ability.

5.3.

Study on the Stability of CNN

A CNN has a large number of parameters, so the experiment of CNN often fails to achieve the effect of network in corresponding papers. At present, the parameter setting in training CNN is mostly based on experience and practice. The optimization analysis of parameters and the study of system stability are the problems to be solved.

5.4.

Hardware Development and Data Set Building

The development of deep learning is inseparable from the innovation of hardware devices and the expansion of data sets. With the support of hardware devices and data sets, CNN will further help and solve the cognitive defects existing in the current network structure.