Distributed file systems and case studies notes — Unit 5

A distributed file system allows programs to store and access remote files exactly as they do local ones, allowing multiple users on multiple machines to share files and storage resources.

⇒1.1 Key Goals

Network transparency (clients shouldn't know the file is on a remote server), high availability (files survive server crashes), and performance (caching to reduce network traffic).

Most systems use a Client-Server architecture.

⇒2.1 The Components

• File Service: Operates on individual files (read, write, append).
• Directory Service: Maps text names (like '/home/user/file.txt') to the internal file identifiers used by the file service.

Servers can be Stateful (remembering which client has which file open) or Stateless (treating every request as independent, like HTTP).

Developed by Sun Microsystems in 1984, NFS is the classic standard for sharing files over a local network.

⇒3.1 Architecture

NFS v3 is inherently stateless. The server does not maintain tables of open files. Every RPC request from a client (like 'Read block 5') contains all information needed (file handle, offset, credentials). This makes the server highly resilient to crashes.

NFS v4 introduced statefulness to support file locking and better security.

⇒4.1 Caching

To improve performance, NFS clients aggressively cache file data, file attributes, and directory contents in local RAM. When an app reads a file, it reads from RAM, avoiding the network.

⇒4.2 Cache Consistency

What if two clients cache the same file, and Client A modifies it? NFS uses a 'polling' approach. The client periodically checks the server's file modification timestamp. This provides poor consistency guarantees (changes might take 3 seconds to propagate).

Developed at Carnegie Mellon, AFS was designed to scale across a massive campus network with thousands of workstations, a scenario where NFS's polling architecture would crush the network.

⇒5.1 Whole-File Caching

When a user opens a file, AFS downloads the entire file to the local hard drive. All reads and writes occur purely locally. When the file is closed, the modified file is uploaded back to the server.

To avoid clients constantly polling the server to check if their cached file is stale, AFS uses Callbacks.

⇒6.1 The Mechanism

When the server gives a client a file, it issues a 'Callback Promise' (a guarantee that the server will notify the client if anyone else changes the file).

The client can now read the local file infinitely without ever checking the network. If someone else uploads a new version, the server sends an RPC to the client ('Callback Break'), invalidating the cache.

NFS and AFS were built for human users editing small text documents. GFS was built for massive machines processing multi-terabyte web-crawling logs.

⇒7.1 The Assumptions

• Hardware failures are the norm, not the exception.
• Files are massive (Gigabytes to Terabytes).
• Most modifications are large appends at the end of the file, not random writes in the middle.
• High sustained bandwidth is vastly more important than low latency.

⇒8.1 The Master and Chunkservers

A GFS cluster consists of a single Master node and hundreds of Chunkservers.

• Master: Stores all metadata (directory structure, file-to-chunk mappings). It holds all metadata in RAM for speed.
• Chunkservers: Files are broken into huge 64MB chunks. These are stored as raw Linux files on the commodity hard drives of the chunkservers.

To read a file, a client asks the Master which chunkserver holds the data. The Master replies with the IP address. The client then downloads the massive chunk directly from the Chunkserver, bypassing the Master entirely to prevent bottlenecks.

HDFS is the open-source implementation of GFS, forming the storage backbone of the Big Data ecosystem.

⇒9.1 Data Replication

Because hardware failures are guaranteed, HDFS automatically replicates every 64MB chunk to three different Chunkservers (DataNodes).

If a hard drive burns out, the Master (NameNode) notices that a chunk only has 2 replicas left. It automatically commands another server to copy the data, restoring the replication factor to 3.

How do you locate data in a massive, decentralized system without a Master node (like in P2P)? You use a DHT.

⇒10.1 The Mechanism

Data (e.g., a file) is hashed to generate a massive integer (a key). Nodes in the network are also hashed to generate their own IDs in the same integer space.

The data is stored on the node whose ID is numerically closest to the data's key.

Chord organizes node IDs into a logical ring.

⇒11.1 Finger Tables

If a node needs to find key 100, it could pass the request around the ring one-by-one. This is slow (O(N)).

Instead, every node maintains a routing table (Finger Table) containing IP addresses of nodes halfway around the ring, a quarter around, etc.

By jumping across the ring, Chord finds any piece of data in just `O(log N)` network hops, even in a network of millions of computers.

Having petabytes of data in HDFS is useless if you can't process it. MapReduce is a programming model for processing huge datasets across a distributed cluster.

⇒12.1 Move Code to Data

Downloading 10TB of data from HDFS to a central server to analyze it would cripple the network. MapReduce does the opposite: it sends the small analytical code script TO the HDFS servers holding the chunks, processing the data locally.

• Map Phase: The system breaks the computation into thousands of parallel Map tasks. Each task runs on a server, reads its local chunk of data, and outputs intermediate Key-Value pairs.
• Shuffle Phase: The system groups all identical Keys together over the network.
• Reduce Phase: The system runs Reduce tasks on the grouped data to aggregate it into the final result.

If a node fails during a Map task, the Master simply restarts that task on another node holding a replica of the data.

• NFS: Stateless, block-level network sharing. Fast but high network traffic.
• AFS: Whole-file caching with Server Callbacks to reduce polling.
• GFS/HDFS: Master-slave architecture built for massive, append-only files with aggressive 3x replication on commodity hardware.
• DHTs (Chord): Decentralized lookup routing using Finger Tables in O(log N) steps.
• MapReduce: Sending computation to the data chunks for massively parallel analytics.