Sampling, estimation and hypothesis testing notes — Unit 4

Up until now, probability theory assumed we knew the exact mathematical parameters (like the exact mean $\mu$ or rate $\lambda$) of the universe, and we used them to predict future data.

Inferential Statistics does the exact opposite. In the real world, the true parameters of the massive, infinite "Population" are forever unknown. We must take a small, random "Sample" of data, and use it to infer, estimate, and guess the true parameters of the population.

⇒1.1 Terminology

• Population: The entire, complete set of all items of interest. (e.g., All 1.4 billion citizens of India).
• Parameter: A fixed numerical characteristic of the Population. It is a constant, but usually unknown.
  Examples: Population Mean ($\mu$), Population Variance ($\sigma^2$), Population Proportion ($P$).
• Sample: A randomly selected subset drawn from the population. (e.g., 5,000 citizens surveyed).
• Statistic: A numerical characteristic calculated directly from the Sample data. It is a random variable because a different sample will yield a different statistic.
  Examples: Sample Mean ($\bar{x}$), Sample Variance ($s^2$), Sample Proportion ($\hat{p}$).

The CLT is the most profound and magical theorem in all of statistics. It is the sole reason why the Normal Bell Curve is so important.

⇒2.1 The Phenomenon

Suppose the original population follows a bizarre, heavily skewed, completely non-normal distribution. You draw a sample of size $n$, and calculate the Sample Mean ($\bar{x}$). You do this thousands of times, collecting thousands of $\bar{x}$ values.

Theorem: Regardless of the shape of the original underlying population distribution, as the sample size $n$ gets larger, the distribution of the Sample Means will mathematically converge to a perfect Normal Distribution.

⇒2.2 The Parameters of the New Distribution

If the population has mean $\mu$ and variance $\sigma^2$, the distribution of Sample Means will be Normal with:

• Mean of Sample Means: Exactly equal to the population mean $\mu$.
• Variance of Sample Means: Shrinks to $\sigma^2 / n$.
• Standard Error (SE): The standard deviation of the sample means is $\sigma / \sqrt{n}$.

Rule of Thumb: The CLT activates and the normal approximation is considered highly accurate when the sample size $n \ge 30$.

Because the true population parameter $\mu$ is unknown, we must use our sample statistic $\bar{x}$ as an "Estimator".

⇒3.1 Point Estimation

Providing a single, specific number as the best guess for the parameter.
Example: We sample 50 CPUs, find the average lifespan is 4.2 years, and bluntly declare the population mean is exactly 4.2.

Point estimation is inherently flawed because it gives absolutely zero indication of how confident we are or what the margin of error is.

⇒3.2 Properties of a Good Estimator

How do we know $\bar{x}$ is a good estimator for $\mu$?

• Unbiasedness: An estimator is unbiased if its expected value exactly equals the true parameter. Mathematically: $E[\bar{x}] = \mu$. It doesn't systematically over-guess or under-guess.
• Consistency: As the sample size $n$ approaches infinity, the estimator converges exactly to the true parameter. Bigger samples always yield closer guesses.
• Efficiency: Among all unbiased estimators, the one with the smallest variance is the most efficient. It is the "tightest" guess.

To fix the flaw of Point Estimation, we use Interval Estimation. Instead of giving one single number, we provide a range of values and attach a probability (confidence level) that the true parameter actually lies within that range.

⇒4.1 The Margin of Error

A Confidence Interval is formatted as:
Point Estimate $\pm$ Margin of Error

The Margin of Error depends on three factors:
1. The Standard Error of the statistic (how noisy the data is).
2. The Sample Size $n$ (bigger $n$ shrinks the interval, making it more precise).
3. The Confidence Level (e.g., 95% or 99%). Requiring higher confidence forces the interval to become wider.

⇒4.2 Confidence Interval for Mean (Large Sample, $n \ge 30$)

Because $n \ge 30$, the CLT guarantees normality. We use the Standard Normal $Z$ distribution.

Interval: $\bar{x} \pm Z_{\alpha/2} \left( \frac{\sigma}{\sqrt{n}} \right)$

Where $Z_{\alpha/2}$ is the critical Z-score from the table. For a standard 95% confidence interval, $Z_{0.025} = 1.96$.

In machine learning and A/B testing, you must mathematically prove that a new algorithm or feature actually improves performance, rather than just producing better results by sheer random luck.

Hypothesis testing is a strict judicial procedure for evaluating claims about population parameters using sample evidence.

⇒5.1 The Two Hypotheses

We set up two mutually exclusive, mathematically opposing claims:

• Null Hypothesis ($H_0$): The assumption of the status quo, no change, no difference, or no effect. It is the "innocent until proven guilty" baseline. It ALWAYS contains an equality sign ($=, \le, \ge$).
  Example: $H_0: \mu = 50$ (The new algorithm takes exactly the same 50ms as the old one).
• Alternate Hypothesis ($H_1$ or $H_a$): The claim you are trying to mathematically prove. It represents change, difference, or effect. It never contains equality ($\neq, <, >$).
  Example: $H_1: \mu < 50$ (The new algorithm is strictly faster).

Because we are judging the entire massive population based on a tiny sample, our conclusion can sometimes be wrong due to extreme statistical bad luck.

⇒6.1 Type I and Type II Errors

• Type I Error (False Positive): We Reject the Null Hypothesis when it was actually True.
  Analogy: Sending an innocent person to jail.
  Probability: The probability of committing a Type I error is denoted by $\alpha$ (Alpha), also known as the Level of Significance. Usually set by the engineer at 0.05 (5%).
• Type II Error (False Negative): We Fail to Reject the Null Hypothesis when it was actually False.
  Analogy: Letting a guilty criminal walk free.
  Probability: Denoted by $\beta$ (Beta).

There is an inherent trade-off. If you demand massive evidence to avoid Type I errors (low $\alpha$), you automatically increase your risk of committing Type II errors.

Power of the Test: Calculated as $1 - \beta$. This is the probability that the test correctly detects a real effect and rejects a false Null Hypothesis.

How much evidence is enough to convict the Null Hypothesis? We define a specific line in the sand based on our chosen $\alpha$ (e.g., 5%).

⇒7.1 One-Tailed vs Two-Tailed Tests

The Alternate Hypothesis $H_1$ dictates the shape of the rejection region on the bell curve.

• Two-Tailed Test ($H_1: \mu \neq 50$): You are looking for any difference, higher or lower. The 5% error risk is split in half (2.5% in the extreme left tail, 2.5% in the extreme right tail).
• Right-Tailed Test ($H_1: \mu > 50$): You only care if the value is higher. The entire 5% error risk is concentrated entirely in the right tail.
• Left-Tailed Test ($H_1: \mu < 50$): The entire 5% error region is in the left tail.

⇒7.2 The Procedure

1. Find the Critical Value (from the Z-table or t-table) that marks the boundary of the rejection region based on $\alpha$.
2. Calculate the Test Statistic from your sample data.
3. If your Test Statistic physically falls into the Rejection Region (beyond the critical value), you Reject $H_0$. If it doesn't, you Fail to Reject $H_0$.

Modern software (like Python SciPy or R) does not use Rejection Regions. It outputs a universal metric called the P-value.

⇒8.1 The Definition

The P-value is the probability of obtaining sample data as extreme as (or more extreme than) your current sample, assuming that the Null Hypothesis is completely, 100% True.

If the P-value is extremely small (e.g., 0.01 or 1%), it means your sample data would be a bizarre, astronomically rare coincidence if the Null Hypothesis were true. Therefore, the Null Hypothesis is probably garbage.

⇒8.2 The Golden Rule of Decision Making

You compare the calculated P-value directly against your pre-defined $\alpha$ (Level of Significance, usually 0.05).

• If P-value $\le \alpha$: The evidence is overwhelmingly strong. Reject $H_0$. The results are statistically significant.
• If P-value $> \alpha$: The evidence is weak. Fail to Reject $H_0$. The results are not statistically significant (they could just be random noise).

The Z-test is used to test claims about the population mean $\mu$.

⇒9.1 When to use the Z-Test

You can ONLY use the Z-test if the Central Limit Theorem is active, guaranteeing a Normal Distribution. This requires:

• The sample size is large ($n \ge 30$).
• OR the sample is small, but you miraculously know the exact true variance $\sigma^2$ of the entire massive population (extremely rare in reality).

⇒9.2 The Test Statistic Formula

You must convert your sample mean $\bar{x}$ into a Standard Normal Z-score. We use the Standard Error $\sigma / \sqrt{n}$ in the denominator.

$Z_{calc} = \frac{\bar{x} - \mu_0}{\sigma / \sqrt{n}}$

Where:
- $\bar{x}$: The average calculated from your small sample.
- $\mu_0$: The baseline population mean claimed in the Null Hypothesis.
- $\sigma$: Population standard deviation (if unknown, substitute with sample standard deviation $s$ since $n \ge 30$).
- $n$: Sample size.

What if testing 30 prototypes is too expensive? What if you can only afford to test $n = 10$ items?

When $n < 30$, the Central Limit Theorem collapses. You cannot use the Z-table because the distribution is no longer perfectly Normal. It becomes "fatter" in the tails to account for the extreme uncertainty of having so little data.

We must use the Student's t-distribution, discovered by William Sealy Gosset (who published under the pen name "Student").

⇒10.1 Degrees of Freedom (df)

There is no single t-distribution. There is an infinite family of t-curves, one for every sample size. You identify the correct curve using Degrees of Freedom.

$df = n - 1$

⇒10.2 The Test Statistic

The formula is algebraically identical to the Z-test, but you strictly use the sample standard deviation $s$, and you MUST look up the critical value in the t-table using your $df$, NOT the Z-table.

$t_{calc} = \frac{\bar{x} - \mu_0}{s / \sqrt{n}}$

Unit 4 is the core of practical statistics. Every single experiment in modern science and A/B testing relies entirely on these concepts.

⇒11.1 University Exam Checklist

• Define Population, Parameter, Sample, and Statistic.
• Explain the Central Limit Theorem (CLT) and why the sample size $n=30$ is critical.
• List the three mathematical properties of a good Point Estimator (Unbiased, Consistent, Efficient).
• Calculate a 95% Confidence Interval for the population mean given a large sample. Explain what the interval physically represents.
• Define the Null Hypothesis ($H_0$) and Alternate Hypothesis ($H_1$).
• Explain the exact difference between a Type I error (False Positive) and a Type II error (False Negative). What is $\alpha$?
• Explain the concept of a p-value. If $p = 0.02$ and $\alpha = 0.05$, do you reject the null hypothesis?
• Execute a full 5-step Hypothesis Test using the Z-test formula for a large sample. Formulate hypotheses, calculate $Z_{calc}$, compare to $Z_{critical}$, and state the conclusion.
• Execute a Hypothesis Test using the Student's t-test for a small sample ($n < 30$). Calculate Degrees of Freedom.