Big data characteristics and the Hadoop ecosystem notes — Unit 2

Big Data refers to datasets whose size or type is beyond the ability of traditional relational databases to capture, manage, and process with low latency.

⇒1.1 The Drivers of Big Data

The explosion of data is driven by ubiquitous computing, social media, IoT sensors, mobile devices, and digital transactions. We have moved from gigabytes to terabytes, petabytes, and exabytes of data generated daily.

Big Data is classically defined by the 'Three V's', though the definition has expanded over time.

• Volume: The sheer scale of data. This is the most obvious characteristic. It requires distributed storage architectures rather than a single massive hard drive.
• Velocity: The speed at which new data is generated and moves around. For example, processing millions of credit card transactions per second to detect fraud in real-time.
• Variety: The different types of data. Unlike legacy systems that stored perfectly structured tables, Big Data includes unstructured text (emails, tweets), semi-structured data (JSON, XML), and binary data (images, video, audio).

Two additional V's are frequently cited in Big Data literature.

• Veracity: The messiness or trustworthiness of the data. With massive volume and variety, data quality drops. Dealing with missing values, typos, and unreliable sensor readings is a core challenge.
• Value: Data itself is useless unless actionable insights can be extracted from it. This is the ultimate goal of Big Data analytics—turning a cost center (storage) into a profit center (insights).

⇒4.1 Traditional Business Intelligence

Traditional BI relies on highly structured data from internal systems (ERP, CRM) stored in relational Data Warehouses. It answers 'known questions' through standard reporting and dashboards (Descriptive Analytics).

⇒4.2 Big Data Analytics

Big Data explores unstructured, external data (social feeds, logs) stored in Data Lakes. It uses advanced machine learning to discover 'unknown unknowns' and predict future trends (Predictive/Prescriptive Analytics).

Apache Hadoop is an open-source framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

⇒5.1 The Core Philosophy

Instead of relying on expensive, highly reliable supercomputers, Hadoop is designed to run on clusters of cheap, commodity hardware. It assumes hardware failures are common and handles them automatically at the software layer.

Hadoop's core innovation is bringing the computation to the data rather than moving massive amounts of data across the network to the computation node.

The core of Hadoop consists of two main layers: Storage and Processing.

• HDFS (Hadoop Distributed File System): The storage layer. It breaks massive files into blocks and distributes them across the cluster.
• MapReduce: The processing layer. A programming model that allows parallel processing of the data stored in HDFS.
• YARN (Yet Another Resource Negotiator): Introduced in Hadoop v2, YARN is the cluster resource manager. It schedules tasks and allocates resources (CPU/Memory) across the cluster.

HDFS uses a Master/Slave architecture.

⇒7.1 NameNode (Master)

There is one NameNode per cluster. It manages the file system namespace and controls access to files. It keeps the directory tree and the mapping of file blocks to DataNodes entirely in its RAM for speed. If the NameNode fails without a backup, the cluster is dead.

⇒7.2 DataNodes (Slaves)

There are usually hundreds of DataNodes. They manage the actual physical storage of data. They serve read and write requests from clients and perform block creation, deletion, and replication upon instruction from the NameNode.

⇒8.1 Block Size

Unlike a standard OS file system where a block is 4KB, HDFS uses a massive default block size (128MB or 256MB). This minimizes the cost of disk seeks and reduces the metadata burden on the NameNode.

⇒8.2 Replication Factor

To ensure fault tolerance, HDFS replicates every block (default replication factor is 3) across different DataNodes, often ensuring replicas are placed on different physical server racks to survive a rack switch failure.

Writing raw MapReduce code in Java is difficult and time-consuming. An entire ecosystem of tools grew around Hadoop to make Big Data accessible.

• Hive: Provides a SQL-like interface (HiveQL) to query data stored in HDFS. It translates SQL queries into MapReduce jobs automatically. Essential for data analysts.
• Pig: A high-level scripting language (Pig Latin) used for data transformation and ETL pipelines. Also compiles down to MapReduce.
• Sqoop: A tool designed to transfer bulk data efficiently between Hadoop and structured relational databases (like MySQL or Oracle).

• Flume / Kafka: Distributed services for collecting, aggregating, and moving massive amounts of streaming event data (like server logs) into HDFS.
• HBase: A non-relational (NoSQL), distributed database that runs on top of HDFS, providing real-time read/write access to massive datasets.
• Oozie: A workflow scheduler system to manage Hadoop jobs (e.g., Run Sqoop, then run Pig, then run Hive query).
• Zookeeper: A centralized service for maintaining configuration information, naming, and providing distributed synchronization across the cluster.

• Big Data V's: Volume (scale), Velocity (speed), Variety (types), Veracity (quality), Value (insights).
• Hadoop Philosophy: Distributed processing on commodity hardware, moving computation to data.
• HDFS Core: Master (NameNode) manages metadata; Slaves (DataNodes) store massive 128MB blocks with 3x replication for fault tolerance.
• Ecosystem: Hive (SQL), Pig (ETL scripting), Sqoop (RDBMS import), Flume (Log ingestion), HBase (NoSQL atop HDFS).