Multi-Layer Perceptron (MLP)

Concept

• Batch: The number of input vectors, which is 2 in the example.
• Hidden layer: Intermediate layers where the network extracts features using weights, biases, and non-linear activation functions.
• Input/Output layer: As the name goes.

For each layer

$$A_{i+1}=ReLU(W_{i}X+b_i)$$

Before final result

We should use Sigmoid or Softmax to squash the result into 0 and 1.

Sigmoid

Sigmoid is used when there's only ONE column in each batch.

It is defined as: $$\sigma (x)=\frac{1}{1+e^{-x}}$$

Softmax

Softmax is the most commonly used activation function. It is the average number in a $e^x$ perspective.

$$Softmax(d_i)=\frac{e^{d_i}}{{\sum }_{k=0}^i{e}^{d_k}}$$

Backpropagation

Backpropagation is short for Backward Propagation of Errors.

Formula & Expression

First, let's look at the basic expression of neuron network which has only one node in each layer.

$$a^{(L)}=\sigma (w^{(L)}{a}^{(L-1)}+b^{(L)})$$

The following will be based on a more complex situation, which has multiple nodes in each layer, reflecting how they are structured in practice.

Let's take a look at some concepts.

• $L$: Layer, the variable in this formula.
• $a_n^{(L)}$: The value of node $n$ in layer $L$.
• $w_{jk}^{(L)}$: The weight between $a_k^{(L-1)}$ and $a_j^{(L)}$.
• $b$: Bias, a specific number.
• $\sigma$: A non-linear function, usually Stigma or ReLU.

We use $z$ to represent the whole bunch of things inside $\sigma$.

$$z_j^{(L)}=\sum _{k=0}^{n_L-1}{w}_{jk}^{(L)}{a}_k^{(L-1)}+b_j^{(L)}$$

So that $a$ can be written as:

$$a_j^{(L)}=\sigma (z_j^{(L)})$$

And the loss $C_0$ would be the sum of errors in the last layer.

$$C_0=\sum _{j=0}^{n_L-1}(a_j^{(L)}-y_j{)}^2$$

where $y_j$ is the given target.

Chain Rule

$$\frac{\partial C_0}{\partial w_{jk}^{(L)}}=\frac{\partial z_j^{(L)}}{\partial w_{jk}^{(L)}}\frac{\partial a_j^{(L)}}{\partial z_j^{(L)}}\frac{\partial C_0}{\partial a_j^{(L)}}$$

According to the previous formulas, each part would be:

$$\frac{\partial C_0}{\partial a_j^{(L)}}=2(a_j^{(L)}-y_j)$$ $$\frac{\partial a_j^{(L)}}{\partial z_j^{(L)}}={\sigma }^{\prime }(z^{(L)})$$ $$\frac{\partial z_j^{(L)}}{\partial w_{jk}^{(L)}}=a^{(L-1)}$$

So,

$$\frac{\partial C_0}{\partial w_{jk}^{(L)}}=a^{(L-1)}\cdot {\sigma }^{\prime }(z^{(L)})\cdot 2(a^{(L)}-y)$$

Oops, there must be something wrong... $w_{jk}^{(L)}$ has many paths to affect $C_0$, so $\frac{\partial C_0}{\partial w_{jk}^{(L)}}$ couldn't just depend on this simple formula.

Excel Document

The document is too complicated without any explanation. I would dive that deep someday when I read the source code of deep learning algorithm.

RNN

This document is also highly abstract, so I would ignore it and try to learn something by my own.

Why RNN?

In the traditional neural network, the layers are fully connected, and nodes in the same layer don't connect to each other. So the data flows from the input layer, passing through the hidden layers and finally to the output layer. It treats the input as a whole object, considering them statically.

What about RNN? In RNN, the node is connected to itself, continuously updating its value. Of course, we can also draw the chart in a row, each node representing the value of that node at a specific time. So it is not good at processing sequences, of which the orders are also important.

What is RNN?

The main node in RNN is continuously updating the value of itself, so it can be considered as a neural network with some loop weights.

The formula below is written by myself according to my understanding. It's just a note without preciseness.

$$a_t=\sigma (w_1\cdot I_t+w_2\cdot a_{t-1}+b_1)$$ $$O_t=w_3\cdot a_t+b_2$$

• $I_t$: The input at time $t$. We can also call this time the depth of loop.
• $O_t$: The output that time $t$.
• $w$, $b$: Weight and bias. They remain the same in every loop.
• $\sigma$: The activation function, again usually ReLU here.

Advantage: It's good at handling sequences, especially the data with time axis.

Disadvantage: The vanishing/exploding gradient problem, which I'll talk about in the next part.

Long Short Term Memory

Complaint: Why are these documents more and more complex??? Without any explanation??? I can hardly understand even if I have learned so many things from online and AI.

Why LSTM?

As a specialized form of RNN, LSTM solves the vanishing gradient problem.

What is gradient? In backpropagation, we need to calculate the adjustment size based on the magnitude of the error, and this is called the "gradient".

What is the Vanishing Gradient Problem? When calculating the adjustment in a layer far from the output layer, we should use the Chain Rule. However, as we all know, the gradient is usually small, and they will be tinier and tinier when multiplied together. When the gradient signal reaches the first few layers, it has "vanished", especially because of the precision limit of our computer. What's worse, in RNN, every loop is like a "layer", and the vanishing gradient problem is severe with a fast increasing network "depth". Another reason is that the activation function, such as Sigmoid, often have a small derivative, which makes the gradient even smaller.

What will happen with the Vanishing Gradient Problem? For general neural networks, the weights in early layers would hardly update, sometimes failing to recognize the fundamental features of the data. For RNN especially, it will "forget" what it saw at the beginning of a long sequence.

Of course, there's an opposite problem, the Exploding Gradients, which have exactly the same reason.

To solve all the problems, here's LSTM.

What is LSTM?

Refer this video on Bilibili.

In this part, I will use $f_n$ to represent a linear function with fixed weight and bias, and use $n$ to identify different functions. In detail, the function is defined like this.

$$f_n(x,h)=Wx+Uh+b$$

And I will also introduce a new activation function, the $\tanh$ function. It always gives a result between -1 and 1.

For the convenience of communication, let's define some short forms.

• $LTM$: long-term memory.
• $STM$: short-term memory.
• $IPT$: input value.
• $OPT$: output value.
• $\sigma (x)$: the Sigmoid function.
• $\tanh (x)$: the $\tanh$ function.

$$\tanh (x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$$

Now, let's look at the LSTM itself.

For each loop in LSTM, there are the long-term memory, the short-term memory, the input and the output. And in each loop, there are mainly three parts. I'm now going to dive into one single loop and explain these parts.

The first part is the Forget Gate. As the name suggests, it updates the long-term memory by forgetting part of it.

$$LTM=LTM\cdot \sigma (f_1(STM,IPT))$$

Here, $LTM$ is multiplied by a factor which indicates in what percentage the long-term memory should be remembered.

The second part is the Input Gate. It also updates the long-term memory, but by adding a new value to it.

$$LTM=LTM+(\tanh (f_2(STM,IPT))\cdot \sigma (f_3(STM,IPT)))$$

In this formula, the cyan part gives the Potential Long-Term Memory, and the yellow part shows the percentage that potential long-term memory should be reserved.

The final part is called the Output Gate. It generates a new value for the Short-Term Memory.

$$STM=\tanh (LTM)\cdot \sigma (f_4(STM,IPT))$$

Almost the same with the formal formula, the cyan part is the Potential Short-Term Memory and the yellow is its percentage to be remembered.

After a whole loop, the updated long-term and short-term memory, combined with a new input value, will be transferred to the next loop.

A more rigorous statement

To make a more rigorous statement, we have to define some variables.

• $c_t$: the Cell State at time $t$, which is actually the value of the long-term memory.
• $h_t$: the Hidden State at time $t$, representing the value of the short-term memory.
• $x_t$: Input value at time $t$.

And we also have to introduce a symbol, $\odot$, which is called the element-wise product.

$$\left[\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_n\end{array}\right]\odot \left[\begin{array}{c}{b}_1\\ b_2\\ ⋮\\ b_n\end{array}\right]=\left[\begin{array}{c}{a}_1{b}_1\\ a_2{b}_2\\ ⋮\\ a_n{b}_n\end{array}\right]$$

We can finally appreciate the formula.

Here's the Forget Gate.

$$f_t=\sigma (W_f\cdot x_t+U_f\cdot h_{t-1}+b_f)$$ $$c_t=f_t\odot c_{t-1}$$

And here's the Input Gate ($i_t$) and Candidate Cell State (${\tilde{c}}_t$)

$$i_t=\sigma (W_i\cdot x_t+U_i\cdot h_{t-1}+b_i)$$ $${\tilde{c}}_t=\tanh (W_c\cdot x_t+U_c\cdot h_{t-1}+b_c)$$ $$c_t=(f_t\odot c_{t-1})+(i_t\odot {\tilde{c}}_t)$$

And the Output Gate ($o_t$)

$$o_t=\sigma (W_o\cdot x_t+U_o\cdot h_{t-1}+b_o)$$ $$h_t=o_t\odot \tanh (c_t)$$

The $h_t$ here is the final output.

xLSTM

Skip for now.

The PDF file of the paper is here.

ResNet

Almost about adding the value of former node into latter calculation. But I also won't dive deep. I'm not going to admit that the fact is I couldn't understand. 🤣

Transformer

What is Transformer

Transformer is mainly based on Query, Key, and Value, also called QKV.

In Transformer model, every token has its representing vector. And the Q, K, and V value of a token is calculated by multiplying its vector and the matrix $W_Q$, $W_K$, and $W_V$.

What are Q, K, and V? Query selects the tokens around where more attention should be given. Key is the value directly questioned by Query. Value represents the real meaning of the token.

Then, here comes the main formula.

$$Attention(Q,K,V)=Softmax(\frac{Q\cdot K}{\sqrt{d_K}})\cdot V$$