Hands-on Deep Learning (I): Prerequisites

EliorFoy

Hands-on Deep Learning (I): Prerequisites [Copy link]

<div class='showpostmsg'># First Lecture Miscellaneous Talk + Pre-school Preparation## 1. About Deep Learning I am a college student. Unfortunately, I knew nothing about deep learning before college. Since ChatGPT became popular all over the world, I gradually noticed that "intelligence" is closely related to deep learning. This book "Hands-on Deep Learning" actually has another theoretical version "DeepLearning" (the cover of the book is a sea of flowers, so it is called "Flower Book"), which is also a very classic tutorial, but because the whole article focuses on mathematical calculations, it can be said to be difficult to chew; and this book has a lot of cases as the title says, "hands-on learning" can make the abstract become vivid. The first chapter of this book does not have a lot of content, roughly introducing what deep learning is. I personally think it is important to know the concepts of "supervised learning", "unsupervised learning", "interaction with the environment" and "reinforcement learning". According to my understanding, supervised learning requires labeling of data, spontaneous learning is unsupervised learning, and interaction with the environment with feedback is reinforcement learning. ## 2. Environment Configuration This book is equipped with a complete [e-tutorial](https://zh-v2.d2l.ai/) (in fact, the e-tutorial has covered all the contents of the physical book, so for those who want to learn DL, these materials are open source). Since I have learned Python before and am relatively familiar with Python's virtual environment, I did not use the tutorial in the book to configure the environment. The Python environment in the book uses Miniconda, which is a lightweight Python distribution. It is convenient to create and manage Python environments, so it is widely used in data science. ### 2.1 Install JupyterI personally use Python 3.12. You can manually create a venv virtual environment and then execute `pip install jupyter` to install jupyter. As for [what is jupyter](https://blog.csdn.net/franklfeng/article/details/117562667), you can search it yourself to understand why this thing exists. Jupyter notebook has plugins that can be installed, but it seems that the `nbextensions` plugin is no longer supported in the 7.x version of notebook. You need to manually downgrade the notebook version to 6.x. Plugins are not necessary. After the installation is complete, enter `jupyter notebook` to start the webui interface. ### 2.2 Install CUDA, cuDNN, and Pytorch Deep learning is generally matrix calculation, and GPU is more suitable for these calculations than CPU. To call GPU, you need to install the driver. NVIDIA series graphics cards have CUDA toolkits for our convenience, and cuDNN is NVIDIA's CUDA deep neural network library, which can significantly improve the training speed and inference speed. It needs to match the corresponding cuda version. Installing the Pytorch framework also requires matching the corresponding cuda version. When I installed pytorch, I found that the cuda version I installed was 12.6 and the latest version matched by pytorch was 12.4, so I would like to tell you that you must check all matching versions before installing. CPU is not unusable... Wait until it is not enough to change cuda. ### 2.3 Cloud Environment For those who don’t have a graphics card, there are many [free GPU resources (reprinted)](https://www.bilibili.com/read/cv20275242/) available online. # Lecture 2 Preliminary Knowledge This lecture mainly introduces some mathematical basics and common Python operations. ## 1. Data operation```python import torch # This library is used for almost all subsequent operationsx = torch.arange(12) # 0-11 tensorlen(x) # How many rowsx.shape # How many rows and columnsx.numel() # Total numberx.reshape(3,4) # Become three rows and four columns, reshape(-1,4) input -1 will automatically calculate torch.zeros(2,3,4) torch.ones(2,3,4) torch.randn(3,4) # Create tensors with all 0 and all 1, shape (3,4) and mean 0 and variance 1 respectivelytorch.tensor(l:list) # Input array to create tensor``` ## 2. Data operation``` python # + - * / ** are all operated on each elementtorch.exp(x) # Perform e^x on each element of xtorch.cat(X,Y,dim=0) # Tensor connection, dim is 0 for row connection, dim is 1 for column connectionX == Y # Returns a tensor in which the corresponding elements are equal and the elements in the corresponding positions are Truex.sum() # For summation, you can specify axis=0 to form a row formed by adding all columns separately (along the column), and 1 to form a row formed by adding all rows separately. At this time, by specifying keepdims=True, the number of axes remains unchanged or it is still one columnA.cumsum(axis=0) # It will not reduce the dimension, which is equivalent to appending x.sum() to the last row``` **Broadcast mechanism**: When operating on tensors of different shapes, they will be automatically copied and expanded to match the operation. **Index and slice**: Tensor features are similar to Python array features, with index and slice**Memory saving**: The difference between `X = X + Y` and `X += Y`. In Python, if `X` is a simple data type (such as integer, floating point number, character, etc.), there is no difference between the two. If it is a complex data type, the former may recreate an object and assign it to `X`, while the latter will assign it to the original memory, so the latter is more memory-efficient (or written as `X[:] = X + Y`). **Object conversion**: Convert torch tensors to numpy tensors (ndarray). Both of them actually look like Python arrays, so the conversion is a matter of course. After the conversion, the underlying memory is shared, so the operations on the variables will affect each other. ```python A = X.numpy() B = torch.tensor(A) B.item() # A tensor with only one element (length 1) can be directly converted to a Python data type, or directly int(B), float(B) ``` ## 3. Data preprocessing Data preprocessing is the process of processing raw data into tensors. :, 2] inputs = inputs.fillna(inputs.mean(numeric_only=True)) # Fill NaN (missing data) with the mean inputs = pd.get_dummies(inputs, dummy_na=True) # pd.get_dummies() The function converts each category of a categorical variable into a binary column (0 or 1), where 1 indicates that the observation belongs to that category and 0 indicates that it does not.# The dummy_na parameter means that NaN values are treated as a separate category x = torch.tensor(inputs.to_numpy(dtype=float)) # Convert pandas data to pytorch tensors``` ## 4. Mathematical Basics### 4.1 Linear Algebra```python AT # Transpose of matrix A torch.dot(x,y) # Dot product of two vectors x,y torch.mv(A,x) # Matrix-vector product torch.mm(A,B) # Matrix-matrix multiplication torch.norm(u) # L2 norm, that is, the square root of the sum of squares of vector elements torch.abs(u).sum # Another L1 norm, that is, absolute value sum # Both L1 norm and L2 norm are special cases of the more general norm Lp (Frobenius norm), and also use the torch.norm method``` ### 4.2 Calculus```python %matplotlib inline # This is the Jupyter magic command to display the image directly embedded import numpy as np from matplotlib_inline import backend_inline from matplotlib import pyplot as plt def f(x): return 3 * x ** 2 - 4 * x # The following is some drawing code. You can reuse it for subsequent drawing. This part of the code has been included in the d2l package that comes with this book def use_svg_display(): #[url=home.php?mod=space&uid=472666]@save[/url] """Use SVG format to display the drawing in Jupyter""" backend_inline.set_matplotlib_formats('svg') def set_figsize(figsize=(3.5, 2.5)): #@save """Set the matplotlib chart size""" use_svg_display() plt.rcParams['figure.figsize'] = figsize #@save def set_axes(axes, xlabel, ylabel, xlim, ylim, xscale, yscale, legend): """Set the axis of matplotlib""" axes.set_xlabel(xlabel) axes.set_ylabel(ylabel) axes.set_xscale(xscale) axes.set_yscale(yscale) axes.set_xlim(xlim) axes.set_ylim( ylim) if legend: axes.legend(legend) axes.grid() #@save def set_axes(axes, xlabel, ylabel, xlim, ylim, xscale, yscale, legend): """Set the axis of matplotlib""" axes.set_xlabel(xlabel) axes.set_ylabel(ylabel) axes.set_xscale(xscale) axes.set_yscale(yscale) ax es.set_xlim(xlim) axes.set_ylim(ylim) if legend: axes.legend(legend) axes.grid() #@save def plot(X, Y=None, xlabel=None, ylabel=None, legend=None, xlim=None, ylim=None, xscale='linear', yscale='linear', fmts=('-', 'm--', 'g-.', 'r:'), figsize=(3.5, 2.5), axes=None): """Plot data point """ if legend is None: legend = [] set_figsize(figsize) axes = axes if axes else plt.gca() # If X has an axis, output True def has_one_axis(X): return (hasattr(X, "ndim") and [X] if Y is None: X, Y = [[]] * len(X), X elif has_one_axis(Y): Y = [Y] if len(X) != len(Y): X = X * len(Y) axes.cla() for x, y, fmt in zip(X, Y, fmts): if len(x): axes.plot(x, y, fmt) else: axes.plot(y, fmt) set_axes(axes, xlabel, ylabel, xlim, ylim, xscale, yscale, legend) x = np.arange(0, 3, 0.1) plot(x, [f(x), 2 * x - 3], 'x', 'f(x)', legend=['f(x)', 'Tangent line (x=1)']) ``` The concepts of partial derivatives and gradients in non-code parts are introduced in detail in advanced mathematics books and will not be repeated here. ### 4.3 Automatic differentiation The framework provides automatic differentiation to speed up differentiation, and automatic differentiation enables the system to subsequently backpropagate gradients. Here, backpropagation means tracking the entire computational graph and filling in the partial derivatives with respect to each parameter. This sentence in the book explains the meaning of automatic differentiation. I don’t understand it very well now, but I should understand it later. - `x.requires_grad_(True) # Equivalent to x=torch.arange(4.0,requires_grad=True)`This means that in the computational graph, `x` will be treated as a node that requires gradient calculation. When you perform operations on this tensor and ultimately calculate the loss function, PyTorch will automatically track all operations on `x` and calculate the gradients with respect to `x` during backpropagation. ```python x.grad # The default is None y = 2 * torch.dot(x, x) # dot producty.backward() x.grad # Automatically calculate the gradient of y with respect to each component of x by calling the backpropagation functionx.grad.zero_() # By default, PyTorch accumulates gradients and we need to clear the previous values```==I feel that I need to supplement the knowledge of [matrix differentiation (vector differentiation)](https://gaomj.cn/matrixgradient/), [matrix gradient](https://zlearning.netlify.app/math/matrix/matrix-gradient.html) and backpropagation here. (I'll make up for it later, it's a headache) ==The above differentiation is simply the following formula:$\nabla_{\mathbf{x}} f(\mathbf{x})=\left[\frac{\partial f(\mathbf{x})}{\partial x_1}, \frac{\partial f(\mathbf{x})}{\partial x_2}, \ldots, \frac{\partial f(\mathbf{x})}{\partial x_n}\right]^{\top}$ ```python # Calling backward for a non-scalar value requires passing in a gradient parameter, which specifies the gradient of the differential function with respect to self. # In this example, we only want to sum the partial derivatives, so passing a gradient of 1 is appropriate x.grad.zero_() y = x * x # Equivalent to y.backward(torch.ones(len(x))) y.sum().backward() x.grad ```== I just vaguely understand it here...:cry:== ### 4.4 Probability```python %matplotlib inline import torch from torch.distributions import multinomial # Simulate the process of rolling a dice fair_probs = torch.ones([6]) / 6 # This vector represents the probability of each side of a fair six-sided dice appearing. multinomial.Multinomial(1, fair_probs).sample() # Create a multinomial distribution object, where 1 means we only draw one sample, and fair_probs is the fair probability vector we just created. The .sample() method is used to draw a sample from this distribution. multinomial.Multinomial(10, fair_probs).sample() # tensor([0., 2., 1., 2., 4., 1.]) There are 10 possible results. Each of them has the above index value. Each index value corresponds to a specific result. ``` # Lecture 3 Linear Neural Networks It mainly includes two parts: linear regression and softmax regression. ## 3.1 Linear Regression### 3.1.1 Linear Model That is, linear fitting, which is consistent with the probability knowledge learned in high school, so I won’t go into details (it is actually described by matrices here). ### 3.1.2 Loss function $$ L(\mathbf{w}, b)=\frac{1}{n} \sum_{i=1}^nl^{(i)}(\mathbf{w}, b)=\frac{1}{n} \sum_{i=1}^n \frac{1}{2}\left(\mathbf{w}^{\top} \mathbf{x}^{(i)}+by^{(i)}\right)^2 . $$The loss function is a function that describes the error between the estimated value and the observed value. When training the model, we hope to find a set of parameters $\left(\mathbf{w}^*, b^*\right)$ that minimizes the total loss on all training samples. $$ \mathbf{w}^*, b^*=\underset{\mathbf{w}, b}{\operatorname{argmin}} L(\mathbf{w}, b) . $$ ### 3.1.3 Analytical solution Minimize $\|\mathbf{y}-\mathbf{X} \mathbf{w}\|^2$ (that is, set the derivative of the loss with respect to $\mathbf{w}$ to 0) and get the analytical solution:==It is not very clear how the calculation here is: anguished: to be added later== $$ \mathbf{w}^*=\left(\mathbf{X}^{\top} \mathbf{X}\right)^{-1} \mathbf{X}^{\top} \mathbf{y} $$Linear regression has an analytical solution, but not all problems have analytical solutions, so it cannot be widely used in DL. ### 3.1.4 Stochastic Gradient Descent When an analytical solution cannot be obtained, the error can be reduced by continuously updating the parameters in the direction of decreasing loss function through the **gradient descent** method. For square loss and affine transformation, it is the following form: $$ \begin{aligned} \mathbf{w} & \leftarrow \mathbf{w}-\frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \partial_{\mathbf{w}} l^{(i)}(\mathbf{w}, b)=\mathbf{w}-\frac{\eta}{ |\mathcal{B}|} \sum_{i \in \mathcal{B}} \mathbf{x}^{(i)}\left(\mathbf{w}^{\top} \mathbf{x}^{(i)}+by^{(i)}\right), \\ b & \leftarrow b-\frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \partial_b l^{(i)}(\mathbf{w}, b)=b-\frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}}\left(\mathbf{w}^{\top} \mathbf{x}^{(i)}+by^{(i)}\right) .\end{aligned} $$$|\mathcal{B}|$ represents the number of samples in each mini-batch, $\eta$ represents the learning rate, and the values of batch size and learning rate are usually manually pre-specified (called hyperparameters) rather than obtained through model training. Minimizing the loss on the training set does not help much. The difficulty is to find a set of parameters that can achieve lower loss on data we have never seen (also known as generalization). ### 3.1.5 Vectorization acceleration Using a loop to traverse a vector is not as efficient as directly adding the vectors, probably because these libraries have already optimized the operation. ### 3.1.6 Normal distribution and square loss$$ p(x)=\frac1{\sqrt{2\pi\sigma^2}}\mathrm{exp}\left(-\frac1{2\sigma^2}(x-\mu)^2\right). $$ ```python # Normal function def normal(x, mu, sigma): p = 1 / math.sqrt(2 * math.pi * sigma**2) return p * np.exp(-0.5 / sigma**2 * (x - mu)**2) # Draw three different normal distributionsparams = [(0, 1), (0, 2), (3, 1)] d2l.plot(x,[normal(x,mu,sigma) for mu,sigma in params],xlabel='x',ylabel='p(x)',figsize=(4.5,2.5),legend=[f'mean {mu}, std {sigma}' for mu, sigma in params]) # figsize controls the display size of the graph, legend specifies the legend```Minimize negative log-likelihood: $$ -\log P(\mathbf{y}\mid\mathbf{X})=\sum_{i=1}^n\frac12\mathrm{log}(2\pi\sigma^2)+\frac1{2\sigma^2}\left(y^{(i)}-\mathbf{w}^\top\mathbf{x}^{(i)}-b\right)^2. $$ ## 3.2 Implementation of linear regression from scratch### 3.2.1 Generate a data set```python def synthetic_data(w, b, num_examples): #@save """Generate y=Xw+b+noise""" X = torch.normal(0, 1, (num_examples, len(w))) y = torch.matmul(X, w) + b # The behavior of the torch.matmul function depends on the shape of the input tensors: # If both inputs are 1D tensors (vectors), it will perform a dot product. # If at least one input is a 2D tensor (matrix), it will perform a matrix multiplication. # If the input has more than 2 dimensions, it will perform a batch matrix multiplication. detach().numpy(), labels.detach().numpy(), 1) # detach() method is used to create a new tensor that shares data with the original tensor, but does not contain gradient information. # Note that the .numpy() method can only be used for CPU tensors. If the tensor is on the GPU (i.e. its device is CUDA), you need to move it to the CPU before converting it to a NumPy array. ``` > I don't know what I am doing anymore. I feel that I still need to go through the concepts in "Deep Learning" (Hua Shu) before I can understand them. So I will change my study plan first. It is out of the plan. I overestimated my foundation. Also, I use Typora to edit Markdown documents. The forum's Markdown editor does not seem to support LaTex mathematical formula rendering. Readers can only copy it to their own Markdown editor to view it. I will see if this can be solved in the next note.The behavior of the matmul function depends on the shape of the input tensors: # If both inputs are 1D tensors (vectors), it will perform a dot product. # If at least one of the inputs is a 2D tensor (matrix), it will perform a matrix multiplication. # If the inputs have more than 2 dimensions, it will perform a batch matrix multiplication. detach().numpy(), labels.detach().numpy(), 1) # detach() method is used to create a new tensor that shares data with the original tensor, but does not contain gradient information. # Note that the .numpy() method can only be used for CPU tensors. If the tensor is on the GPU (i.e. its device is CUDA), you need to move it to the CPU before converting it to a NumPy array. ``` > I don't know what I am doing anymore. I feel that I still need to go through the concepts in "Deep Learning" (Hua Shu) before I can understand them. So I changed my study plan first. It was out of my plan. I overestimated my foundation. Also, I used Typora to edit Markdown documents. The forum's Markdown editor does not seem to support LaTex mathematical formula rendering. Readers can only copy it to their own Markdown editor to view it. I will see if this can be solved in the next note.The behavior of the matmul function depends on the shape of the input tensors: # If both inputs are 1D tensors (vectors), it will perform a dot product. # If at least one of the inputs is a 2D tensor (matrix), it will perform a matrix multiplication. # If the inputs have more than 2 dimensions, it will perform a batch matrix multiplication. detach().numpy(), labels.detach().numpy(), 1) # detach() method is used to create a new tensor that shares data with the original tensor, but does not contain gradient information. # Note that the .numpy() method can only be used for CPU tensors. If the tensor is on the GPU (i.e. its device is CUDA), you need to move it to the CPU before converting it to a NumPy array. ``` > I don't know what I am doing anymore. I feel that I still need to go through the concepts in "Deep Learning" (Hua Shu) before I can understand them. So I changed my study plan first. It was out of my plan. I overestimated my foundation. Also, I used Typora to edit Markdown documents. The forum's Markdown editor does not seem to support LaTex mathematical formula rendering. Readers can only copy it to their own Markdown editor to view it. I will see if this can be solved in the next note. </div><script> var loginstr = '<div class="locked">To view all the contents of the post, please <a href="/bbs/member.php?mod=logging&action=login" style="color:#e60000" class="loginf">log in</a> or <a href="/bbs/member.php?mod=register_eeworld.php&action=wechat" style="color:#e60000" target="_blank">register</a></div>'; if(parseInt(discuz_uid)==0){ } </script><script type="text/javascript">(function(d,c){var a=d.createElement("script"),m=d.getElementsByTagName("script")[0],eewurl="//counter.eeworld.com.cn/pv/count/";a.src=eewurl+c;m.parentNode.insertBefore(a,m)})(document,523)</script>

nmg

The issue of not supporting LaTex mathematical formula rendering has been reported

Can I use the basics of mathematics while reading it? Seeing so much mathematics knowledge at once makes me feel overwhelmed.

御坂10032号

nmg posted on 2024-9-6 14:49 The problem of not supporting LaTex mathematical formula rendering has been reported. Can the basic mathematics be used while reading? I saw so much at once...

I feel like I can't continue without understanding the formula

liholo

You are learning Python, probability, calculus, and linear algebra. I feel like learning electronic technology is not easy!

EliorFoy

nmg posted on 2024-9-6 14:49 The problem of not supporting LaTex mathematical formula rendering has been reported. Can the basic mathematics be used while reading? I saw so much at once...

Well, the math knowledge required seems quite diverse. I have been improving my math knowledge these days, otherwise I would not know where the code came from. I have not learned some subjects of numerical analysis in college, and some concepts are not very clear to me.