NumPy, Pandas and data visualisation basics notes — Unit 5

While standard Python is great for general programming, its built-in Data Structures (like Lists) are extremely slow for massive mathematical operations (like processing a 10-Megapixel image or a million rows of sales data).

To solve this, the Python ecosystem relies on highly optimized, third-party C-extensions. The "Big Three" libraries are:

• NumPy (Numerical Python): For high-speed mathematical operations on multi-dimensional arrays.
• Pandas: For cleaning, analyzing, and manipulating tabular data (like Excel sheets).
• Matplotlib: For rendering data into graphs and charts.

The core of NumPy is the `ndarray` (N-Dimensional Array).

⇒2.1 Lists vs. NumPy Arrays

Why use NumPy instead of Python Lists?

• Memory: Python lists store pointers to objects scattered across memory. NumPy arrays store raw numerical data in a single, contiguous block of C-memory.
• Speed: Because of contiguous memory and C-level execution, NumPy operations are 50x to 100x faster than looping through Python lists.
• Homogeneous: Unlike Lists, all elements in a NumPy array MUST be of the exact same data type (e.g., all 64-bit floats).

⇒2.2 Creating Arrays

```python
import numpy as np

# From a list
arr = np.array([1, 2, 3, 4])
# Generating arrays
zeros = np.zeros(5) # [0., 0., 0., 0., 0.]
seq = np.arange(0, 10, 2) # [0, 2, 4, 6, 8]
```

Arrays can have multiple dimensions (axes).
1D = Vector.
2D = Matrix.
3D = Tensor (like an RGB Image).

⇒3.1 Creating 2D Arrays

Pass a list of lists to `np.array()`.

```python
matrix = np.array([
[1, 2, 3],
[4, 5, 6]
])
```

⇒3.2 Key Attributes

• `arr.ndim`: Number of dimensions (axes). For the matrix above, ndim is 2.
• `arr.shape`: A tuple indicating the size of each dimension. For the matrix above, shape is `(2, 3)` (2 rows, 3 columns).
• `arr.size`: Total number of elements. Size is $2 \times 3 = 6$.
• `arr.dtype`: The data type of the elements (e.g., `int32`, `float64`).

The greatest power of NumPy is Vectorization—the ability to perform operations on entire arrays at once without writing `for` loops.

If `lst = [1, 2, 3]`, trying `lst * 2` yields `[1, 2, 3, 1, 2, 3]` (repetition).

If `arr = np.array([1, 2, 3])`, executing `arr * 2` performs element-wise multiplication yielding `[2, 4, 6]` instantly.

⇒4.1 Array Arithmetic

```python
a = np.array([10, 20, 30])
b = np.array([1, 2, 3])

print(a + b) # [11, 22, 33]
print(a - b) # [9, 18, 27]
print(a * b) # [10, 40, 90] (Element-wise multiplication!)
```

Note: For actual Matrix Multiplication (Dot Product), use `np.dot(a, b)` or `a @ b`.

While NumPy is great for raw numbers, it struggles with messy, real-world data (like a mix of dates, text, and missing numbers). Pandas is built on top of NumPy to provide Excel-like data manipulation.

Pandas relies on two primary data structures:

• Series: A 1-Dimensional column of data.
• DataFrame: A 2-Dimensional table of data (made of multiple Series).

A Series is a one-dimensional array capable of holding any data type. Crucially, every Series has an Index (labels for each data point).

⇒6.1 Creating a Series

```python
import pandas as pd

data = [85, 90, 75]
# Creating with default integer index (0, 1, 2)
s1 = pd.Series(data)

# Creating with custom index
marks = pd.Series(data, index=["Math", "Physics", "Chem"])
print(marks["Physics"]) # Output: 90
```

A Series acts like a hybrid between a Python List and a Dictionary.

A DataFrame is a 2-dimensional data structure with columns of potentially different types (like a SQL table or Excel sheet).

⇒7.1 Creating a DataFrame

The easiest way is passing a dictionary of lists.

```python
data = {
"Name": ["Alice", "Bob", "Charlie"],
"Age": [20, 21, 19],
"Grade": ["A", "B", "A"]
}
df = pd.DataFrame(data)
```

The dictionary Keys become the Column headers. The Lists become the columns themselves (Series). An integer index `0, 1, 2` is automatically generated for the rows.

⇒7.2 Reading from Files

Pandas can ingest massive datasets with a single line of code:

`df = pd.read_csv("sales_data.csv")`
`df = pd.read_excel("data.xlsx")`

⇒8.1 Quick Inspection

When loading a massive dataset, you can't print the whole thing.

• `df.head(5)`: Displays the first 5 rows.
• `df.tail(5)`: Displays the last 5 rows.
• `df.info()`: Shows memory usage, column names, and missing value counts.
• `df.describe()`: Instantly generates statistical summaries (Mean, Std Dev, Min, Max) for all numerical columns.

⇒8.2 Selecting Columns

Extracting a single column returns a Series.
`ages = df["Age"]`

Extracting multiple columns returns a smaller DataFrame (Note the double brackets!).
`subset = df[["Name", "Grade"]]`

How do you select specific rows? You cannot just use `df[0]`. You must use special indexers.

⇒9.1 `df.loc[]` (Label-based)

Selects rows based on their explicit Index label.
If your index consists of student IDs (e.g., "S100"):
`df.loc["S100"]` gets that student's row.

⇒9.2 `df.iloc[]` (Integer-based)

Selects rows based purely on their physical integer position (0-indexed row number), completely ignoring the index labels.

`df.iloc[0]` always gets the very first physical row in the table.
`df.iloc[0:5]` slices the first 5 rows.

Pandas allows you to filter rows based on conditions without writing any loops.

```python
# Creates a boolean Series [True, False, True...]
condition = df["Age"] > 20

# Pass the condition into the dataframe
older_students = df[condition]
```

Or as a single line:
`adults = df[df["Age"] >= 18]`

⇒10.1 Multiple Conditions

Use `&` (AND) and `|` (OR). Each condition MUST be wrapped in parentheses due to operator precedence rules.

`honor_roll = df[(df["Age"] < 20) & (df["Grade"] == "A")]`

Real-world data is messy. CSVs often have empty cells, which Pandas loads as `NaN` (Not a Number). Machine Learning algorithms will crash if fed `NaN`s.

⇒11.1 Detecting Nulls

`df.isnull().sum()`: Shows a count of how many missing values exist in each column.

⇒11.2 The Two Solutions

• Drop them: `df.dropna()` deletes any row that contains at least one `NaN`. Quick, but destroys valuable data if used carelessly.
• Fill them (Imputation): `df.fillna(value)` replaces all `NaN`s with a specific value. Often, data scientists fill missing numerical values with the Mean or Median of that column.

```python
mean_age = df["Age"].mean()
df["Age"].fillna(mean_age, inplace=True)
```

Humans are bad at reading thousands of numbers, but great at seeing visual trends. Matplotlib (specifically its `pyplot` module) is the foundation of Python visualization.

⇒12.1 The Basic Plot

```python
import matplotlib.pyplot as plt

x = [1, 2, 3, 4]
y = [10, 20, 25, 30]

plt.plot(x, y) # Draws a line chart
plt.show() # Opens the window to display it
```

⇒12.2 Graph Customization

A graph is useless without context.

```python
plt.title("Company Revenue")
plt.xlabel("Quarter")
plt.ylabel("Revenue in $M")
# Change line color and style
plt.plot(x, y, color="red", linestyle="--", marker="o")
```

Different data requires different charts.

• Line Plot (`plt.plot()`): Best for showing trends over time (Time-series data).
• Bar Chart (`plt.bar(x, y)`): Best for comparing categorical data (e.g., Sales across different cities).
• Scatter Plot (`plt.scatter(x, y)`): Plots individual points. Best for finding correlations between two numerical variables (e.g., Height vs. Weight).
• Histogram (`plt.hist(data, bins)`): Visualizes the distribution of a single numerical variable. It groups data into 'bins' to show frequency (e.g., Age distribution of a population).
• Pie Chart (`plt.pie(data)`): Shows percentage share of a whole. (Generally disliked by statisticians, but popular in business).

Unit 5 introduces the standard industry toolkit for Data Analysis.

⇒14.1 University Exam Checklist

• Why is NumPy an `ndarray` significantly faster than a Python List?
• Explain the concept of Vectorization in NumPy.
• What is the difference between a Pandas Series and a Pandas DataFrame?
• Write a Pandas snippet to load `data.csv`, and filter out all rows where the 'Salary' column is less than 50000.
• What is the difference between `df.loc[]` and `df.iloc[]`?
• Write a script using Matplotlib to plot a Sine wave ($y = \sin(x)$).
• When would you use a Scatter Plot versus a Histogram?