Hands-on Deep Learning (Part 4): Multilayer Perceptron

EliorFoy

Hands-on Deep Learning (Part 4): Multilayer Perceptron [Copy link]

<div class='showpostmsg'>Due to the problem of mathematical formula rendering, you can move to my personal blog to view [this article](https://eliorfoy.github.io/2024/10/25/%E5%A4%A7%E4%B8%89%E4%B8%8A/%E5%8A%A8%E6%89%8B%E5%AD%A6%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%EF%BC%88%E5%9B%9B%EF%BC%89%EF%BC%9A%E5%A4%9A%E5%B1%82%E6%84%9F%E7%9F%A5%E6%9C%BA/) # 1) Multilayer Perceptron## 1. Introduction to Multilayer Perceptron The figure shows a single hidden layer perceptron. When there are many layers, the last layer is regarded as a linear predictor. This architecture is called a multilayer perceptron (MLP).  Applying the activation function $\sigma$ will make the multilayer perceptron no longer a simple linear model and can represent any affine function. As shown below: $$ \begin{aligned} \mathbf{H}&=\sigma(\mathbf{X}\mathbf{W}^{(1)}+\mathbf{b}^{(1)}), \\\mathbf{O}&=\mathbf{H}\mathbf{W}^{(2)}+\mathbf{b}^{(2)}. \end{aligned} $$ ## 2. Common Activation Functions_Activation Functions_（activation function) determines whether a neuron should be activated by calculating a weighted sum and adding a bias, which are differentiable operations that transform input signals into outputs. ### 2.1 ReLU (Rectified linear unit) function $\mathrm{ReLU}(x) = max(x,0)$ Image:  ### 2.2 sigmoid function This function has been mentioned before: $\mathrm{sigmoid}(x)=\frac{1}{1+\exp(-x)}$ Image:  ### 2.3 tanh function Hyperbolic tangent function: $\tanh(x)=\frac{1-\exp(-2x)}{1+\exp(-2x)}$ Image:  # (II) Implementation of Multilayer Perceptron## 1. The complete code from scratch is as follows: ```python import torch from torch import nn from d2l import torch as d2l batch_size = 256 train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size) num_inputs, num_outputs, num_hiddens = 784, 10, 256 W1 = nn.Parameter(torch.randn( num_inputs, num_hiddens, requires_grad=True) * 0.01) b1 = nn.Parameter(torch.zeros(num_hiddens, requires_grad=True)) W2 = nn.Parameter(torch.randn( num_hiddens, num_outputs, requires_grad=True) * 0.01) b2 = nn.Parameter(torch.zeros(num_outputs, requires_grad=True)) params = [W1, b1, W2, b2] def relu(X): a = torch.zeros_like(X) return torch.max(X, a) loss = nn.CrossEntropyLoss(reduction='none') # Training num_epochs, lr = 10, 0.1 updater = torch.optim.SGD(params, lr=lr) # Stochastic Gradient Descent d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, updater) ``` It can be seen that it is basically similar to the previous linear neural network implementation from scratch, but with one more layer and the activation function using a custom relu function. ## 2. The concise implementation of the complete code is as follows: ```python import torch from torch import nn from d2l import torch as d2l net = nn.Sequential(nn.Flatten(), nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10)) def init_weights(m): if type(m) == nn.Linear: nn.init.normal_(m.weight, std=0.01) # The nn.init.normal_ function initializes each element in the weight vector of the linear layer to a normal random value net.apply(init_weights); # Training batch_size, lr, num_epochs = 256, 0.1, 10 loss = nn.CrossEntropyLoss(reduction='none') trainer = torch.optim.SGD(net.parameters(), lr=lr) train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size) d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, trainer) ``` # (III) Model selection The original text is full of nonsense. In short, the trained model will have training error and generalization error. The training error is the error obtained by training on the training set, and the generalization error is the expected value of the error in the infinite set. From this we can know that the model will have underfitting and overfitting. Generally speaking, underfitting means that the training error is large, the expression ability is insufficient, and accurate prediction cannot be made; while overfitting means that the model performs well on the training data, but performs poorly on new and unseen data. Overfitting has the following influencing factors:- The number of adjustable parameters. When the number of adjustable parameters (sometimes called degrees of freedom) is large, the model tends to overfit more easily. - The values that the parameters take. When the range of values that the weights take is large, the model may be more likely to overfit. - The number of training samples. Even if the model is simple, it is easy to overfit a dataset containing only one or two samples. Overfitting a dataset with millions of samples requires an extremely flexible model. This is where the **validation set** comes in: Split our data into three parts, In addition to the training and test datasets, add a validation dataset, also called a validation set. But the reality is that the boundary between validation data and test data is worryingly blurred... When training data is scarce, use **K-fold cross validation**: The original training data is divided into non-overlapping subsets. Then perform model training and validation, each time training on a subset, and validation on the remaining subset (the subset that was not used for training in that round). Finally, the training and validation errors are estimated by averaging the results of the experiments. # (IV) Regularization Model Technology Regularization is a technique used in machine learning to prevent model overfitting. ## 1. Weight decay ($L_2$ regularization) The norm has been mentioned before. The norm can measure the size of a vector. Because in many models, especially linear models, the size of the parameters can reflect the complexity of the model. Larger parameter values mean that the model is more sensitive to changes in the input data, which may cause the model to overfit on the training data and perform poorly on unseen data. So minimizing the $L_2$ norm prevents any single weight from becoming too large, which helps maintain the stability of the model. The $L_2$ regularization, or weight decay, is to add a norm as a penalty term to the loss function, adjusting the original training goal of minimizing the prediction loss on the training label to minimizing the sum of the prediction loss and the penalty term. Now, if our weight vector grows too large, our learning algorithm may focus more on minimizing the weight norm. In fact, there is also $L_1$ regularization, so why choose the $L_2$ norm? One reason is that it imposes a huge penalty on large components of the weight vector. This makes our learning algorithm biased towards models that evenly distribute weights on a large number of features. In contrast, the $L_1$ penalty causes the model to concentrate weights on a small number of features and clear other weights to zero. This is called feature selection. So the loss of the new additional penalty after balance becomes: $$ L(\mathbf{w},b)+\frac{\lambda}{2}\|\mathbf{w}\|^2 $$ $\lambda$ is called the regularization coefficient. Smaller $\lambda$ values correspond to less constrained $\mathbf{w}$, while larger $\lambda$ values impose greater constraints on $\mathbf{w}$. $\lambda$ is divided by 2 because the derivative can offset the squared term $\frac{1}{2}$. The squared norm is chosen instead of the standard norm for ease of calculation. So the mini-batch stochastic gradient descent becomes: $$ \begin{aligned} d2l import torch as d2l # 执行创建过程， # # English: \mathbf{w}\leftarrow(1-\eta\lambda)\mathbf{w}-\frac{\eta}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\mathbf{x}^{(i)}\left(\mathbf{w}^{\top}\mathbf{x}^{(i)}+by^{(i)}\right) \end{aligned} $$ So in the process of updating $\mathbf{w}$, we are also trying to reduce the size of $\mathbf{w}$ to zero. ### Example demonstration```python %matplotlib inline import torch from torch import nn from d2l import torch as d2l # Initialize model parameters def init_params(): w = torch.normal(0, 1, size=(num_inputs, 1), requires_grad=True) b = torch.zeros(1, requires_grad=True) return [w, b] # Define norm penalty def l2_penalty(w): return torch.sum(w.pow(2)) / 2 def train(lambd): w, b = init_params() net, loss = lambda X: d2l.linreg(X, w, b), d2l.squared_loss num_epochs, lr = 100, 0.003 animator = d2l.Animator(xlabel='epochs', ylabel='loss', yscale='log', xlim=[5, num_epochs], legend=['train', 'test']) for epoch in range(num_epochs): for X, y in train_iter: # Added L2 norm penalty, # The broadcast mechanism makes l2_penalty(w) a vector of length batch_size l = loss(net(X), y) + lambd * l2_penalty(w) l.sum().backward() d2l.sgd([w, b], lr, batch_size) if (epoch + 1) % 5 == 0: animator.add(epoch + 1, (d2l.evaluate_loss(net, train_iter, loss), d2l.evaluate_loss(net, test_iter, loss))) print('The L2 norm of w is:', torch.norm(w).item()) ``` When $\lambda$ is 0, that is, when no penalty is introduced, we get:  When $\lambda$ is set to 3, we get:  By comparison, we can find that after introducing $\lambda$, although the error of the training data set cannot be reduced later, the error in the test data set has been reduced. **Simple implementation** ```python # weight decay as parameter input def train_concise(wd): net = nn.Sequential(nn.Linear(num_inputs, 1)) for param in net.parameters(): param.data.normal_() loss = nn.MSELoss(reduction='none') num_epochs, lr = 100, 0.003 # bias parameter is not decayed trainer = torch.optim.SGD([ {"params":net[0].weight,'weight_decay': wd}, {"params":net[0].bias}] # only weights are set for weights, so the bias parameter b(bias) will not decay, lr=lr) animator = d2l.Animator(xlabel='epochs', ylabel='loss', yscale='log', xlim=[5, num_epochs], legend=['train', 'test']) for epoch in range(num_epochs): for X, y in train_iter: trainer.zero_grad() l = loss(net(X), y) l.mean().backward() trainer.step() if (epoch + 1) % 5 == 0: animator.add(epoch + 1, (d2l.evaluate_loss(net, train_iter, loss), d2l.evaluate_loss(net, test_iter, loss))) print('L2 norm of w:', net[0].weight.norm().item()) ``` ## 2. Dropout From the original text: > The generalization properties of deep networks are puzzling, and the mathematical basis of this generalization property remains an open research problem. We encourage readers who like to study theory to delve deeper into this topic. In this section, we will focus on the exploration of practical tools that tend to improve the generalization of deep networks. This shows that the problem of overfitting has not been perfectly solved... We expect a "good" prediction model to perform well on unseen data: Classical generalization theory suggests that in order to narrow the gap between training and test performance, a simple model should be the goal. The norm of the parameters in weight decay also represents a useful measure of simplicity. Another aspect of simplicity is smoothness, that is, the function should not be sensitive to small changes in its input. If it is smooth, then adding some random noise should have little effect. So [Srivastava _et al._, 2014](https://zh.d2l.ai/chapter_references/zreferences.html#id155 "Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929–1958.") and others came up with the idea of injecting noise into the network. The **temporary dropout method** is to randomly "discard" (i.e. temporarily remove) some neurons (and their connections) in the network during training to reduce the complex co-adaptation relationship between neurons and enhance the generalization ability of the model. Interestingly, the original paper on the dropout method mentioned the analogy of sexual reproduction, which shows that deep learning has an extremely important connection with biology, not to mention that this year's Nobel Prizes in Physics and Chemistry both fell into the hands of deep learning... ### Example demonstration```python import torch from torch import nn from d2l import torch as d2l def dropout_layer(X, dropout): assert 0 <= dropout <= 1 # In this case, all elements are dropped if dropout == 1: return torch.zeros_like(X) # In this case, all elements are retained if dropout == 0: return X mask = (torch.rand(X.shape) > dropout).float() # Use torch.rand(X.shape) to generate a random number tensor of the same shape as X, where each element is randomly drawn from a uniform distribution between 0 and 1, and generate a boolean mask by comparing this random number tensor with the dropout value return mask * X / (1.0 - dropout) # Keep the expected value unchanged num_inputs, num_outputs, num_hiddens1, num_hiddens2 = 784, 10, 256, 256 # Define a multilayer perceptron with two hidden layers, each hidden layer contains 256 units dropout1, dropout2 = 0.2, 0.5 class Net(nn.Module): def __init__(self, num_inputs, num_outputs, num_hiddens1, num_hiddens2, is_training = True): super(Net, self).__init__() # Parent class constructor self.num_inputs = num_inputs self.training = is_training self.lin1 = nn.Linear(num_inputs, num_hiddens1) self.lin2 = nn.Linear(num_hiddens1, num_hiddens2) self.lin3 = nn.Linear(num_hiddens2, num_outputs) self.relu = nn.ReLU() def forward(self, X): H1 = self.relu(self.lin1(X.reshape((-1, self.num_inputs)))) # Only use dropout when training the model if self.training == True:# Add a dropout layer after the first fully connected layer H1 = dropout_layer(H1, dropout1) H2 = self.relu(self.lin2(H1)) if self.training == True: # Add a dropout layer after the second fully connected layer H2 = dropout_layer(H2, dropout2) out = self.lin3(H2) return out net = Net(num_inputs, num_outputs, num_hiddens1, num_hiddens2) # Train num_epochs, lr, batch_size = 10, 0.5, 256 loss = nn.CrossEntropyLoss(reduction='none') train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size) trainer = torch.optim.SGD(net.parameters(), lr=lr) d2l.train_ch3(net, d2l.train_ch3(net, train_iter, nn.train_iter, loss, num_epochs, trainer) ``` **Simple implementation** ```python net = nn.Sequential(nn.Flatten(), nn.Linear(784, 256), nn.ReLU(), # Add a dropout layer after the first fully connected layer nn.Dropout(dropout1), nn.Linear(256, 256), nn.ReLU(), # Add a dropout layer after the second fully connected layer nn.Dropout(dropout2), nn.Linear(256, 10)) def init_weights(m): if type(m) == nn.Linear: nn.init.normal_(m.weight, std=0.01) net.apply(init_weights); # Training trainer = torch.optim.SGD(net.parameters(), lr=lr) d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, trainer) ``` # (V) Forward propagation, back propagation and computational graph## 1. Forward propagation Forward propagation (forward propagation or forward pass) refers to: calculating and storing the results of each layer in the neural network in sequence (from the input layer to the output layer). For example (for a multilayer perceptron with two hidden layers): $$ \begin{aligned} \mathbf{z}&=\mathbf{W}^{(1)}\mathbf{x}\\ \mathbf h&=\phi(\mathbf z) \\ \mathbf o&=\mathbf W^{(2)}\mathbf h \\ L&=l(\mathbf{o},y)\\ Regularization term: s&=\frac{\lambda}{2}\Big(\|\mathbf{W}^{(1)}\|_F^2+\|\mathbf{W}^{(2)}\|_F^2\Big)\\ Regularization loss (objective function): J&=L+s \end{aligned} $$ Forward propagation calculation graph:  ## 2. Backward propagation Backward propagation (or backpropagation) refers to the method of calculating the gradient of neural network parameters. For example: $$ \begin{aligned} \mathbf{Z}&=\mathbf{X}\mathbf{W}^{(1)}+\mathbf{b}^{(1)}, \\ \mathbf{H}&=\sigma(\mathbf{Z}),\\ \mathbf{O}&=\mathbf{H}\mathbf{W}^{(2)}+\mathbf{b}^{(2)}. \end{aligned} $$ The gradient of the regularized loss $J$ with respect to $\mathbf{W}^{(2)}$ is: $$ \frac{\partial J}{\partial\mathbf{W}^{(2)}}=\mathrm{prod}\left(\frac{\partial J}{\partial\mathbf{o}},\frac{\partial\mathbf{o}}{\partial\mathbf{W}^{(2)}}\right)+\mathrm{prod}\left(\frac{\partial J}{\partial s},\frac{\partial s}{\partial\mathbf{W}^{(2)}}\right)=\frac{\partial J}{\partial\mathbf{o}}\mathbf{h}^{\top}+\lambda\mathbf{W}^{(2)} $$ Note: We use the operator prod to multiply its arguments after performing necessary operations such as transposing and swapping the input positions. For vectors, this is simple, it is just matrix-matrix multiplication. For higher-dimensional tensors, we use the appropriate counterparts. The operator refers to all of these symbols. Regarding the gradient of $\mathbf{W}^{(1)}$: $$ \begin{aligned} \frac{\partial J}{\partial \mathbf{W}^{(1)}} &= \mathrm{prod}\left(\frac{\partial J}{\partial\mathbf{o}},\frac{\partial\mathbf{o}}{\partial\mathbf{h}} ,\frac {\partial\mathbf{h}}{\partial\mathbf{Z}} ,\frac{\partial\mathbf{Z}}{\partial\mathbf{\mathbf{W}^{(1)}}} \right) +\mathrm{prod}\left(\frac{\partial J}{\partial s},\frac{\partial s}{\partial \mathbf{W}^{(1)}}\right) \\ &= \mathrm{prod}\left(\frac{\partial J}{\partial\mathbf{o}},\frac{\partial\mathbf{o}}{\partial\mathbf{h}} ,\frac{\partial\mathbf{h}}{\partial\mathbf{Z}}\right)\mathbf{x}^{\top} +\lambda\mathbf{W}^{(1)}\\ & = {\frac{\partial J}{\partial\mathbf{h}}}\odot\phi^{\prime}\left(\mathbf{z}\right)\mathbf{x}^{\top} + \lambda\mathbf{W}^{(1)}\\ &= \frac{\partial J}{\partial\mathbf{o}}\mathbf{W}^{(2)^{\top}}\ odot\phi^{\prime}\left(\mathbf{z}\right)\mathbf{x}^{\top} +\lambda\mathbf{W}^{(1)}\\ \end{aligned} $$ ## 3. Training Neural Networks When training neural networks, forward propagation and back propagation depend on each other. , we traverse the computational graph in the direction of the dependencies and compute all variables on its path. These are then used for backpropagation, where the order of computation is the opposite of the computational graph. # (VI) Numerical stability and model initialization scheme Selection plays a pivotal role in neural network learning and is essential for maintaining numerical stability. Unstable gradients can lead to several problems: - Gradient exploding problem: Parameter updates are too large, destroying The stable convergence of the model - the gradient vanishing problem: the parameter updates are too small and hardly move at each update, causing the model to fail to learn. Another problem in neural network design is its parameterization. The inherent symmetry. As for what parameter symmetry is, I feel that the book does not explain it very clearly. I just need to understand that one way to solve this problem is parameter initialization. Attention during optimization and proper regularization can also further improve stability. [Xavier Initialization ](https://zhuanlan.zhihu.com/p/458373836) is available for reference. # (VII) Environment and distribution shift Considering the environment is actually considering feedback. The book gives an example: Suppose we train a loan application The default risk model was used to predict who would repay a loan or default. The model found that the applicant’s shoes were associated with default risk (applicants wearing oxfords would repay, applicants wearing sneakers would default). Once the model started to look at the shoes, Once a decision is made, customers will understand and change their behavior. Soon all applicants will be wearing Oxfords, with no corresponding improvement in creditworthiness. **By introducing model-based decisions into the environment, we may destroy the model**. Reinforcement learning actually introduces feedback, but it is difficult to implement. ## 1. Distribution Shift The book only briefly talks about the concept, This section should be considered as an extension. - **Covariate shift**: When the model is trained and applied, the distribution of input data (features) changes, but the distribution of output labels remains unchanged - **Label shift* *: The opposite of covariate shift. In the case of label shift, it is assumed that the distribution of features (input data) remains unchanged, but the marginal probability of the label $P(y)$ has changed, that is, the labels of different categories are The distribution in the training set and the test set is different, while the distribution of features $P(x|y)$ remains unchanged when the label is given. - **Concept shift**: The definition of labels changes, for example, as human civilization develops, We also call cats dogs (this is just an example, it sounds ridiculous, but in fact people's concepts have changed). ## 2. Distribution bias correction Since this section does not involve detailed examples and codes, the theory is also complicated, but The explanation is very concise... so I will skip this chapter for now... </div><script> var loginstr = '<div class="locked">To view all the contents of the post, please <a href="/bbs/member.php?mod=logging&action=login" style="color:#e60000" class="loginf">log in</a> or <a href="/bbs/member.php?mod=register_eeworld.php&action=wechat" style="color:#e60000" target="_blank">register</a></div>'; if(parseInt(discuz_uid)==0){ } </script><script type="text/javascript">(function(d,c){var a=d.createElement("script"),m=d.getElementsByTagName("script")[0],eewurl="//counter.eeworld.com.cn/pv/count/";a.src=eewurl+c;m.parentNode.insertBefore(a,m)})(document,523)</script>

zgnasd1950

I learned that as an electronics enthusiast, I am curious about all kinds of things and want to understand them.

Jacktang

Due to mathematical formula rendering issues, you can move to my personal blog to view this article

The blog cannot be opened