Regression and Bayesian learning notes — Unit 2

Regression is a supervised learning task used to predict continuous numerical values (e.g., predicting temperature, stock prices, or a person's weight).

⇒1.1 The Linear Model

Simple Linear Regression assumes a straight-line relationship between a single input feature (X) and the target output (Y).

`y_pred = mX + c` (where `m` is the weight/slope, and `c` is the bias/intercept).

Multiple Linear Regression extends this to multiple features: `y_pred = w1X1 + w2X2 + ... + wnXn + b`.

To determine how 'wrong' our current line is, we calculate the difference between our predictions and the actual data points. This is called the Cost Function.

⇒2.1 Mean Squared Error (MSE)

The standard cost function for regression. It takes the square of the difference between the actual value and the predicted value, and averages it over all data points.

Squaring the error does two things: it makes all errors positive (so over-predicting and under-predicting don't cancel each other out), and it heavily penalizes large errors.

The optimization algorithm used to train the model. Its goal is to find the weights (`w`) and bias (`b`) that minimize the Cost Function.

⇒3.1 How it works

Imagine standing blindfolded on a mountainous terrain and trying to reach the absolute bottom of the valley. You feel the slope of the ground under your feet (the Gradient, calculated using calculus derivatives) and take a step downhill.

The size of the step you take is controlled by the Learning Rate (α). If α is too small, learning takes forever. If α is too large, you might step completely over the valley and never converge.

Despite the name, Logistic Regression is used for Classification (predicting discrete categories like True/False or Dog/Cat), not regression.

⇒4.1 The Sigmoid Function

If we use Linear Regression for classification, our predictions can shoot off to +1000 or -50. We need probabilities between 0 and 1. We achieve this by passing the linear output through a Sigmoid activation function.

`S(z) = 1 / (1 + e^-z)`

If the output is >= 0.5, classify as Class 1. If < 0.5, classify as Class 0.

Instead of learning a single, fixed set of weights (like gradient descent), Bayesian Learning calculates the probability distributions over all possible hypotheses.

⇒5.1 Prior vs Posterior

• Prior Probability `P(h)`: Our initial belief that hypothesis h is correct, before seeing any data.
• Likelihood `P(D|h)`: The probability of observing the dataset D if hypothesis h were true.
• Posterior Probability `P(h|D)`: Our updated belief in hypothesis h, after seeing the data. This is what we want to calculate.

⇒6.1 Maximum A Posteriori (MAP)

Bayes theorem: `P(h|D) = [ P(D|h) * P(h) ] / P(D)`. The MAP hypothesis is the single hypothesis that maximizes this posterior probability. It relies heavily on our Prior assumptions `P(h)`.

⇒6.2 Maximum Likelihood (ML)

If we assume that every hypothesis is equally likely beforehand (a uniform Prior), we can ignore `P(h)`. The goal simplifies to finding the hypothesis that maximizes the likelihood of the data `P(D|h)`. This is Maximum Likelihood Estimation.

A highly practical and surprisingly effective classification algorithm based on Bayes Theorem.

⇒7.1 The 'Naive' Assumption

It is called 'Naive' because it assumes that all input features are conditionally independent of each other, given the target class. (e.g., It assumes that in a spam email, the presence of the word 'Viagra' is completely independent of the presence of the word 'Discount').

Despite this assumption almost never being true in the real world, Naive Bayes performs exceptionally well in text classification and spam filtering, and is extremely fast to train.