Stream analytics and data visualisation notes — Unit 5

⇒1.1 Batch Processing

Hadoop MapReduce is a batch processing framework. It requires data to be fully loaded into storage (HDFS) before processing begins. It answers questions about the past (e.g., analyzing yesterday's web traffic logs overnight).

⇒1.2 Stream Processing

Stream processing engines analyze data in motion, instantly as it is generated. It answers questions about 'right now' (e.g., detecting a fraudulent credit card swipe in milliseconds before approving the transaction).

Processing infinite, high-velocity streams of data presents unique challenges.

• Unbounded Data: The data never stops. You cannot sort the 'whole' dataset or calculate a final average because the dataset is infinite.
• Latency: Processing must be done in milliseconds.
• Out-of-Order Data: Network delays mean Event A might arrive after Event B, even though A occurred first. The system must reconstruct the correct timeline.

Because the data stream is infinite, analytical operations (like 'average' or 'sum') must be performed over finite slices of time called Windows.

• Tumbling Windows: Non-overlapping time blocks (e.g., count clicks from 1:00-1:05, then 1:05-1:10).
• Hopping/Sliding Windows: Overlapping time blocks (e.g., calculate the moving average of the last 5 minutes, updated every 1 minute).
• Session Windows: Grouped by user activity, bounded by a period of inactivity (e.g., analyzing a user's browsing session until they go idle for 10 minutes).

⇒4.1 Apache Kafka

Often the backbone of streaming architectures. Kafka is a distributed publish-subscribe messaging system. It acts as a massive, high-throughput buffer, decoupling the systems generating data (Producers) from the systems analyzing it (Consumers).

⇒4.2 Processing Engines

Tools like Apache Flink, Apache Storm, and Spark Streaming consume data from Kafka, apply windowing and analytical logic in real-time, and output alerts or feed real-time dashboards.

Data visualization is the graphical representation of information and data. By using visual elements like charts, graphs, and maps, visualization tools provide an accessible way to see and understand trends, outliers, and patterns in data.

⇒5.1 The Purpose

The human brain processes visual information significantly faster than text or numbers in a spreadsheet. Visualization bridges the gap between complex analytical models and business stakeholders who need to make decisions.

Choosing the right chart depends on the data type and the message.

• Comparison: Bar charts (discrete categories), Line charts (trends over time).
• Composition: Pie charts (parts of a whole), Stacked bar charts.
• Distribution: Histograms (frequency of values), Box plots (showing quartiles and outliers).
• Relationship: Scatter plots (correlation between two variables), Heatmaps (matrix of values).

A bad visualization is worse than no visualization because it misleads the viewer.

• Data-to-Ink Ratio: Maximize the ink used to display data, minimize the ink used for non-data decorations (gridlines, 3D effects, background colors).
• Context: Always provide axis labels, legends, and a clear title.
• Color: Use color intentionally to highlight anomalies or separate categories, not just for decoration. Beware of red/green combinations for colorblind accessibility.
• Honesty: Do not truncate the Y-axis to exaggerate minor differences.

A dashboard is an interactive visual interface that aggregates multiple key performance indicators (KPIs) into a single screen.

⇒8.1 Modern BI Tools

Tools like Tableau, Microsoft Power BI, and Looker connect directly to data warehouses or streaming engines. They allow non-technical business users to filter, drill down, and interact with the data without writing SQL queries.

• Stream Processing: Analyzes infinite 'data in motion' with sub-second latency, contrasting with batch processing of historical data.
• Windowing: Essential technique (Tumbling, Sliding, Session) to perform calculations on infinite streams.
• Kafka: The industry-standard distributed messaging buffer for streaming data.
• Visualization: Critical for communicating insights. Requires choosing the right chart type and adhering to strict design principles (Data-to-Ink ratio) to avoid misleading stakeholders.