The Building Blocks of Hadoop: An Overview of Key Components#

A Brief History#

Apache Hadoop was born from the need to handle web-scale data processing challenges. Inspired by Google’s MapReduce paper and its Google File System (GFS), Doug Cutting and Mike Cafarella created Hadoop. Initially, Hadoop was part of the Lucene project (an open-source search engine library), but it eventually became its own top-level Apache project. Today, it is a widely used platform for distributed data storage and processing.

Why Hadoop?#

Traditional relational database management systems (RDBMS) struggle to handle the velocity, volume, and variety (often called the “three Vs”) of big data. Hadoop, on the other hand:

1. Scales Out: By simply adding more commodity hardware, you can grow storage and computational power.
2. Fault Tolerance: Data is replicated across multiple nodes, so if one node fails, another holds a copy of the data.
3. Cost-Effective: Hadoop is designed to run on commodity hardware, cutting down the cost compared to specialized servers.
4. Parallel Processing: Data is processed in parallel across the cluster, significantly increasing throughput.

These attributes enable organizations to handle massive datasets distributed across hundreds or even thousands of machines.

Hadoop Distributed File System (HDFS)#

HDFS is the foundational data storage system in the Hadoop ecosystem:

• Distributed Storage: Files are split into blocks (default block size is often 128 MB, though older versions used 64 MB), which are replicated across multiple nodes.
• Append-Only: Once written, HDFS files are rarely modified. This design choice simplifies data consistency management.
• Designed for High Throughput: Optimized for reading large files sequentially, rather than random access.

MapReduce#

MapReduce is the programming model that sparked the Hadoop revolution:

• Mapping Phase: Input data is split into chunks. Each chunk is processed by a mapper which outputs key-value pairs.
• Shuffle and Sort: The intermediary key-value pairs are redistributed across the system based on the key.
• Reducing Phase: The pairs for each unique key are aggregated and combined to produce final output results.

Yet Another Resource Negotiator (YARN)#

YARN separates cluster resource management from the programming model:

• ResourceManager: Manages resources and schedules applications.
• ApplicationMaster: Manages the lifecycle of individual tasks (MapReduce jobs, Spark jobs, etc.).
• NodeManager: Runs on each node to manage resources on that node.

HDFS Daemons: NameNode, Secondary NameNode, DataNode#

HDFS works through a master-slave architecture:

1. NameNode:

   • The master server that manages the file system namespace and regulates access to data.
   • Stores metadata about file locations, block locations, and permission details.

2. Secondary NameNode:

   • It’s a common misconception that this is a hot standby. Instead, it performs housekeeping tasks like merging the filesystem edit logs into a single checkpoint.
   • If the NameNode were to fail, the Secondary NameNode’s checkpoint files could help in partial recovery, but full data recovery may need additional steps.

3. DataNode:

   • The slave nodes where actual data blocks are stored.
   • Regularly sends heartbeat and block reports to the NameNode to confirm its health and the blocks it holds.

YARN Daemons: ResourceManager, NodeManager#

YARN decoupled the resource management from MapReduce, allowing multiple data processing engines to coexist:

1. ResourceManager:

   • Global resource scheduler for the entire cluster.
   • Keeps track of how many CPU cores and how much memory is in use, and allocates containers (logical bundles of resources) to applications.

2. NodeManager:

   • Runs on each worker node.
   • Oversees the lifecycle of containers, monitors resource usage, and reports to the ResourceManager.

JobHistoryServer#

When a MapReduce job finishes, log data and job counters are stored in history files. The JobHistoryServer:

• Centralizes the job history and counters in a location accessible to the end-users or cluster administrators.
• Facilitates easier debugging and retrospective performance analysis.

Apache Hive#

• Data Warehousing: Hive provides an SQL-like interface (HiveQL) for querying and managing large datasets.
• Schema on Read: Unlike traditional RDBMS, Hive uses schema-on-read, meaning data is checked against the schema only when a query is run.
• Integration: Hive queries run as MapReduce jobs (in older versions) or via Tez or Spark in modern deployments, eliminating the need to write Java MapReduce code.

Apache Pig#

• Scripting Language: Pig Latin is a data flow language designed to analyze huge datasets with minimal boilerplate.
• Ease of Use: Pig’s built-in operators for joins, filters, grouping, and more allow you to handle complex data transformations quickly.
• Under the Hood: Pig translates scripts into sequences of MapReduce jobs.

Apache HBase#

• NoSQL Database: Modeled after Google’s Bigtable, HBase is a distributed, column-oriented database.
• Random Access: While HDFS is optimized for sequential reads, HBase allows random read/write of large datasets.
• Real-Time: If you need real-time queries or analytics on top of massive data stored in Hadoop, HBase is often the go-to solution.

Apache Spark#

• Fast Engine: Spark offers in-memory computation capabilities that can be significantly faster than MapReduce for iterative workloads.
• Unified Stack: Spark has modules for streaming (Spark Streaming), interactive analytics (Spark SQL), machine learning (MLlib), and graph processing (GraphX).
• Integration: Spark runs on top of Hadoop YARN, leveraging HDFS for data storage, or can run in standalone mode, on Kubernetes, or in the cloud.

The MapReduce Pseudocode#

A typical WordCount job consists of a Mapper class and a Reducer class:

1
public class WordCount {
2

3
    // Mapper class
4
    public static class TokenizerMapper
5
         extends Mapper<Object, Text, Text, IntWritable> {
6

7
        private final static IntWritable one = new IntWritable(1);
8
        private Text word = new Text();
9

10
        @Override
11
        public void map(Object key, Text value, Context context)
12
                throws IOException, InterruptedException {
13
            String[] tokens = value.toString().split("\\s+");
14
            for (String token : tokens) {
15
                word.set(token);
16
                context.write(word, one);
17
            }
18
        }
19
    }
20

21
    // Reducer class
22
    public static class IntSumReducer
23
         extends Reducer<Text, IntWritable, Text, IntWritable> {
24

25
        @Override
26
        public void reduce(Text key, Iterable<IntWritable> values, Context context)
27
                throws IOException, InterruptedException {
28
            int sum = 0;
29
            for (IntWritable val : values) {
30
                sum += val.get();
31
            }
32
            context.write(key, new IntWritable(sum));
33
        }
34
    }
35

36
    // Driver method
37
    public static void main(String[] args) throws Exception {
38
        Configuration conf = new Configuration();
39
        Job job = Job.getInstance(conf, "word count");
40
        job.setJarByClass(WordCount.class);
41
        job.setMapperClass(TokenizerMapper.class);
42
        job.setCombinerClass(IntSumReducer.class);
43
        job.setReducerClass(IntSumReducer.class);
44
        job.setOutputKeyClass(Text.class);
45
        job.setOutputValueClass(IntWritable.class);
46
        FileInputFormat.addInputPath(job, new Path(args[0]));
47
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
48
        System.exit(job.waitForCompletion(true) ? 0 : 1);
49
    }
50
}

How It Works#

1. Mapper: Reads each line, splits it into words, and emits each word along with a count of 1.
2. Combiner (Optional): Responsible for local aggregation before sending data across the network to the reducer, minimizing data shuffle.
3. Reducer: Receives all values for a given key (word) and sums them.

Running this on Hadoop involves:

1. Packaging the Java classes into a JAR file.
2. Uploading the JAR and the input data to HDFS.
3. Submitting the job to YARN using a command like:

   1
   hadoop jar wordcount.jar WordCount /input/path /output/path
   

4. Viewing the results in the output directory after the job completes.

Memory and Resource Management#

• YARN Container Sizes: Configure the container memory and CPU cores to match the resource capacity of each node.
• Java Heap Settings: Adequately size the Java heap (e.g., -Xmx) for your Mapper and Reducer tasks to prevent out-of-memory errors.
• Resource Quotas: Impose quotas so that one user or department does not starve others of cluster resources.

Data Locality and Shuffling#

• Data Locality: Wherever possible, push the computation to the data, not the other way around. This correlation reduces expensive network I/O.
• Combiner Functions: Applying a combiner in MapReduce jobs can reduce the volume of data shuffled from the mappers to the reducers.
• Compression: Use compression (e.g., snappy, LZO, gzip) during the shuffle phase to decrease data movement overhead.

Tuning MapReduce Parameters#

Below is a short table showing some crucial MapReduce parameters and their roles:

Finding optimal values involves monitoring job performance, checking logs, and iterative fine-tuning.

Monitoring and Logging#

• Metrics: Tools like Ganglia, Grafana, and Cloudera Manager provide metrics on CPU usage, memory utilization, disk I/O, and network usage across the cluster.
• Log Aggregation: YARN aggregates logs from individual containers. Storing and analyzing logs in a centralized system (e.g., Elastic Stack, Splunk) is essential for timely troubleshooting.
• Alerting: Set up alert thresholds for critical cluster components (NameNode, ResourceManager) to detect potential failures early.

High Availability#

• NameNode HA: Configuring an active/passive NameNode setup with Zookeeper-based automatic failover prevents cluster downtime if the primary NameNode fails.
• ResourceManager HA: Similarly, you can implement YARN ResourceManager in HA mode, ensuring that application scheduling and resource allocation continue even if a ResourceManager goes down.
• Multiple DataNode Replicas: Stick to the recommended replication factor (commonly 3) to avoid data loss when hardware or network failures occur.

Deploying on the Cloud#

Several cloud platforms provide managed Hadoop solutions:

• Amazon Elastic MapReduce (EMR): Fully managed offering tightly integrated with AWS services.
• Microsoft Azure HDInsight: Manages Hadoop clusters with enterprise security features on Azure infrastructure.
• Google Cloud Dataproc: Quickly spin up Hadoop or Spark clusters for on-demand workloads.

Cloud deployments help organizations scale up or down quickly, pay only for what they use, and tap into a rich ecosystem of analytical services.