Data analytics lifecycle and data preparation notes — Unit 1

Data analytics is the science of analyzing raw data to make conclusions about that information. Many of the techniques and processes of data analytics have been automated into mechanical processes and algorithms that work over raw data for human consumption.

⇒1.1 Types of Analytics

• Descriptive Analytics: What happened? (e.g., summarizing past sales data).
• Diagnostic Analytics: Why did it happen? (e.g., investigating a sudden drop in website traffic).
• Predictive Analytics: What is likely to happen? (e.g., forecasting next quarter's revenue based on trends).
• Prescriptive Analytics: What should we do about it? (e.g., recommending a pricing strategy to maximize profit).

The data analytics lifecycle is designed specifically for Big Data problems and data science projects. It defines a structured approach to solving complex analytical problems.

⇒2.1 Phase 1: Discovery

The team learns the business domain, including relevant history such as whether the organization or similar organizations have attempted similar projects in the past from which they can learn.

• Identify business objectives and frame the analytics problem.
• Identify key stakeholders and their expectations.
• Formulate initial hypotheses (`H0` and `H1`) to test.
• Assess available resources (data, tools, personnel).

This phase requires the presence of an analytic sandbox, in which the team can work with data and perform analytics for the duration of the project. This is often the most time-consuming phase (taking up to 80% of the project time).

⇒3.1 Steps in Data Preparation

• Extract, Load, Transform (ELT): Extract data from source systems and load it into the sandbox. Transform it within the sandbox to prepare for analysis.
• Familiarization: Understand the data structure, variable types, and initial patterns.
• Data Conditioning: Cleanse the data (handle missing values, remove duplicates, correct inconsistencies).
• Survey and Visualize: Use data visualization tools to explore distributions and identify anomalies or outliers.

In this phase, the team determines the methods, techniques, and workflow it intends to follow for the subsequent model building phase.

⇒4.1 Key Activities

• Data Exploration: Explore the data to learn about relationships between variables. Feature selection is critical here.
• Variable Selection: Select the most significant variables that influence the outcome. Use techniques like correlation analysis.
• Model Selection: Choose the appropriate analytical techniques (e.g., regression, classification, clustering) based on the problem statement and the nature of the data.

⇒5.1 Phase 4: Model Building

The team develops datasets for testing, training, and production purposes. They execute the models based on the work done in the model planning phase. They evaluate whether the models meet the business objectives and are robust enough.

⇒5.2 Phase 5: Communicate Results

The team, in collaboration with major stakeholders, determines if the results of the project are a success or a failure based on the criteria developed in Phase 1. The team should identify key findings, quantify the business value, and develop a narrative to summarize and convey findings to stakeholders.

The team delivers final reports, briefings, code, and technical documents. In addition, the team may run a pilot project to implement the models in a production environment.

⇒6.1 Deployment Strategy

Deploying a model involves integrating it into the organization's existing software systems. It is crucial to set up monitoring to track the model's performance over time, as models can degrade (concept drift) when underlying data patterns change.

Raw data is inherently messy. Data cleaning (or cleansing) is the process of detecting and correcting (or removing) corrupt or inaccurate records from a record set, table, or database.

⇒7.1 Handling Missing Data

• Deletion: Remove rows with missing values (Listwise or Pairwise deletion). Useful when the dataset is large and the missing values are random.
• Imputation: Fill in missing values using statistical methods like Mean, Median, or Mode imputation. More advanced techniques include KNN (K-Nearest Neighbors) imputation or regression imputation.
• Flagging: Add a binary variable indicating whether a value was missing, which can sometimes carry predictive power itself.

An outlier is an observation that lies an abnormal distance from other values in a random sample from a population. Outliers can skew statistical analyses and model training.

⇒8.1 Detection and Treatment

• Detection: Visual methods (Box plots, Scatter plots) or Statistical methods (Z-score, IQR - Interquartile Range).
• Treatment:
• Removal:* If the outlier is due to a data entry error.
• Transformation:* Apply log or square root transformations to reduce the impact of extreme values.
• Capping/Winsorizing:* Replace extreme values with a specified percentile value (e.g., 99th percentile).

Data transformation converts data from one format or structure into another to make it suitable for analysis.

⇒9.1 Key Techniques

• Normalization (Min-Max Scaling): Scales data to a fixed range, usually 0 to 1. Sensitive to outliers.
• Standardization (Z-score Scaling): Centers the data around a mean of 0 with a standard deviation of 1. Better for data with outliers.
• Encoding Categorical Data: Machine learning models require numerical input. Use techniques like One-Hot Encoding (creating dummy variables) or Label Encoding (assigning integers to categories).

• Analytics Types: Descriptive (past), Diagnostic (why), Predictive (future), Prescriptive (action).
• Lifecycle: A 6-phase approach for complex analytical projects: Discovery, Data Prep, Model Planning, Model Building, Communicating Results, and Operationalizing.
• Data Preparation: The most critical and time-consuming phase. Involves ELT, data conditioning, and addressing anomalies.
• Cleaning & Transformation: Essential steps including handling missing values (imputation), treating outliers, and scaling data (normalization/standardization) for robust model performance.