Unsupervised learning, clustering and dimensionality reduction notes — Unit 4

In unsupervised learning, the dataset has no labels (no target 'Y' variable). The algorithm's job is not to predict, but to discover the hidden structure, patterns, or groupings within the raw data (X).

⇒1.1 Key Applications

• Clustering: Grouping similar data points together (e.g., customer segmentation).
• Dimensionality Reduction: Compressing data while retaining its structure (e.g., visualizing high-dimensional data).
• Anomaly Detection: Finding outliers (e.g., credit card fraud detection).

The most popular clustering algorithm. It attempts to divide the data into `K` distinct, non-overlapping subgroups (clusters).

⇒2.1 The Algorithm

• 1. Choose the number of clusters `K`.
• 2. Randomly initialize `K` points in the data space. These are the initial 'Centroids' (cluster centers).
• 3. Assignment Step: Assign every data point to the Centroid it is closest to (usually Euclidean distance).
• 4. Update Step: Calculate the true center (mean) of all the points assigned to a cluster, and move the Centroid to that new mean location.
• 5. Repeat steps 3 and 4 until the Centroids stop moving (convergence).

K-Means requires you to specify `K` before running the algorithm. How do you know what the right number of clusters is?

⇒3.1 Within-Cluster Sum of Squares (WCSS)

For a given `K`, we calculate the distance of every point to its centroid, square it, and sum them up. A lower WCSS is better (it means the clusters are tight).

⇒3.2 The Elbow Curve

We run K-Means for `K = 1, 2, 3... 10` and plot the WCSS. The curve will drop rapidly at first, then flatten out. The 'Elbow' (the point of inflection where the drop slows down) represents the optimal number of clusters.

Unlike K-Means, this algorithm builds a hierarchy of clusters and does not require pre-specifying `K`.

⇒4.1 Agglomerative (Bottom-Up) Approach

• 1. Start by treating every single data point as its own cluster (N clusters).
• 2. Find the two clusters that are closest to each other and merge them into one.
• 3. Repeat step 2 until all points are merged into a single giant cluster.

⇒4.2 The Dendrogram

The history of these mergers is plotted as a tree diagram called a Dendrogram. To get the final clusters, you simply draw a horizontal line cutting across the Dendrogram at your desired height.

Modern datasets often have thousands of features (dimensions). High dimensions cause the 'Curse of Dimensionality' (algorithms become exponentially slower, and data becomes sparse, causing overfitting).

⇒5.1 Feature Selection vs Extraction

• Feature Selection: Keeping the most important existing features and dropping the rest (e.g., dropping 'Eye Color' when predicting 'Income').
• Feature Extraction: Mathematically combining existing features to create a smaller number of entirely new features that capture the same information.

The most popular technique for Feature Extraction. It uses linear algebra (Eigenvectors and Eigenvalues) to project high-dimensional data into a lower-dimensional space while preserving as much variance (information) as possible.

⇒6.1 How it works conceptually

Imagine a cloud of points in 3D shaped like a flat pancake. PCA finds the 'Principal Components' (axes). The First Principal Component (PC1) is drawn in the direction where the data varies the most (the length of the pancake). PC2 is perpendicular to PC1 (the width). PC3 is the thickness.

Because the pancake is very thin, PC3 captures almost no variance. We can drop PC3 entirely, successfully reducing the data from 3D to 2D without losing significant information.