Artificial neural networks and deep learning basics notes — Unit 5

Computing systems loosely inspired by the biological neural networks that constitute animal brains.

⇒1.1 The Artificial Neuron (Perceptron)

The fundamental unit of an ANN. It receives inputs (x1, x2), multiplies them by weights (w1, w2), sums them up, adds a bias (b), and passes the result through an Activation Function to produce an output.

`z = (w1x1 + w2x2 ... + wn*xn) + b`

`Output = ActivationFunction(z)`

Without activation functions, a neural network, no matter how many layers it has, is mathematically equivalent to a single linear regression model. Activation functions introduce Non-Linearity, allowing the network to learn complex, curved boundaries.

⇒2.1 Common Functions

• Sigmoid: Maps input to [0, 1]. Historically popular, but suffers from the Vanishing Gradient problem.
• Tanh: Maps input to [-1, 1]. Better than Sigmoid as it is zero-centered.
• ReLU (Rectified Linear Unit): `f(x) = max(0, x)`. The industry standard for hidden layers. It doesn't suffer from vanishing gradients and is computationally extremely fast.
• Softmax: Used exclusively in the final output layer for multi-class classification. Converts raw outputs into a normalized probability distribution (all outputs sum to 1.0).

Neurons are organized into distinct layers.

• Input Layer: Receives the raw features of the dataset. Number of neurons = number of features.
• Hidden Layers: Intermediate layers where the computation happens. A network is considered 'Deep Learning' when it has more than one hidden layer.
• Output Layer: The final prediction. For binary classification, this is 1 neuron (Sigmoid). For a 10-class problem, this is 10 neurons (Softmax).

In a 'Fully Connected' (Dense) network, every neuron in layer `L` is connected to every neuron in layer `L+1`.

⇒4.1 Forward Propagation

Input data is fed into the network. Calculations flow forward through the hidden layers until an output prediction is generated. The network compares this prediction to the true label using a Loss Function (e.g., Cross-Entropy).

⇒4.2 Backpropagation

The heart of neural network training. The algorithm calculates the calculus derivative (gradient) of the Loss Function with respect to every single weight in the network, starting from the output layer and working backwards to the input layer.

It uses the Chain Rule of calculus to figure out exactly how much each weight contributed to the final error. It then uses an optimizer (like Gradient Descent or Adam) to update the weights to reduce the error for the next pass.

Training a network on a massive dataset of 1 million images requires breaking the data down.

• Epoch: One complete pass of the entire training dataset through the neural network.
• Batch Size: Instead of updating the weights after every single image (too chaotic) or after all 1 million images (too memory-intensive), we update weights after a 'batch' (e.g., 32 or 64 images).
• Iterations: The number of batches needed to complete one Epoch. (1 million images / 64 batch size = 15,625 iterations per epoch).

Standard Fully Connected networks (Multi-Layer Perceptrons) are terrible at processing images or sequential text because they ignore spatial structure and order.

⇒6.1 Convolutional Neural Networks (CNNs)

Designed specifically for computer vision (Images). Instead of connecting every pixel to every neuron, they slide 'filters' over the image to detect spatial patterns like edges, textures, and eventually complex objects (like faces or cars).

⇒6.2 Recurrent Neural Networks (RNNs)

Designed for sequential data (Text, Time-series, Audio). Unlike standard networks, RNNs have 'memory'—the output of a previous step is fed back in as input for the next step, allowing them to understand context in a sentence.