Mastering Hadoop YARN: A Chef’s Recipe for Resource Management#

Resource Manager#

The Resource Manager (RM) is the “head chef” who decides which applications get resources and how much they receive. It has two major components:

• Scheduler: Allocates resources to different applications based on constraints like capacity or fairness.
• Applications Manager: Manages the application lifecycle, including accepting job submissions and negotiating resources.

Node Manager#

Each worker node in the cluster runs a Node Manager (NM) process. The Node Manager is responsible for:

• Launching and monitoring containers (the “cooking stations”).
• Reporting container and resource usage back to the Resource Manager.
• Managing logs and metrics that track container performance.

Application Master#

The Application Master (AM) is unique to each application submitted to YARN. After the scheduler allots resources, the AM supervises how tasks are executed within containers. If you’re running a MapReduce job, the MapReduce AM (often referred to as MRAppMaster) handles tasks like map and reduce scheduling, data shuffle, and failure recoveries.

Containers#

A container is a self-contained execution environment for a unit of work. The Node Manager launches containers based on resource requests made by the Application Master. YARN containers don’t necessarily have to follow the classical “map” or “reduce” approach; they’re flexible enough for any computational tasks, be it a Spark executor or other specialized workloads.

Prerequisites#

1. Hadoop Installation: YARN is a part of Hadoop, so install a compatible Hadoop distribution (e.g., Apache Hadoop 3.x).
2. Java: Java 8 or above is usually recommended.
3. Network Configuration: Each node should be able to communicate with the Resource Manager.
4. SSH Access: For managing nodes in the cluster.

Important Configuration Files#

1. core-site.xml: Common settings like file system defaults.
2. hdfs-site.xml: Configuration for the Hadoop Distributed File System.
3. yarn-site.xml: Primary configuration file for YARN.
4. mapred-site.xml: If you’re running MapReduce on YARN, you’ll need to configure this.

Within yarn-site.xml, typical properties you might configure include:

1
<property>
2
  <name>yarn.resourcemanager.hostname</name>
3
  <value>resourcemanager.example.com</value>
4
</property>
5

6
<property>
7
  <name>yarn.nodemanager.resource.memory-mb</name>
8
  <value>4096</value>
9
</property>
10

11
<property>
12
  <name>yarn.nodemanager.resource.cpu-vcores</name>
13
  <value>2</value>
14
</property>
15

16
<property>
17
  <name>yarn.scheduler.minimum-allocation-mb</name>
18
  <value>512</value>
19
</property>

These properties define how many memory megabytes (MB) and CPU cores are available for scheduling on each node. The minimum-allocation-mb sets the smallest container size.

Quick Setup Demo#

Below is an example of a minimal local YARN setup for experimentation (ideal for a single-node cluster):

1. Configure yarn-site.xml in your Hadoop configuration directory:

   1
   <property>
   2
     <name>yarn.nodemanager.aux-services</name>
   3
     <value>mapreduce_shuffle</value>
   4
   </property>
   5
   <property>
   6
     <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
   7
     <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   8
   </property>
   9
   <property>
   10
     <name>yarn.resourcemanager.scheduler.class</name>
   11
     <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
   12
   </property>
   

2. Start YARN:

   1
   $ start-yarn.sh
   

3. Verify Processes:

   • ResourceManager: Runs on the node designated as the RM.
   • NodeManager: Runs on every worker node.

4. Check the Web UI: By default, the Resource Manager is usually accessible on port 8088 (e.g., http://localhost:8088).

Now you have a functional environment to submit YARN-based applications.

FIFO Scheduler#

• Basic Idea: First In, First Out (FIFO). Earliest submitted job gets priority.
• Use Case: Small-scale clusters or straightforward, sequential processing.
• Limitations: No advanced resource sharing or priority queues.

Capacity Scheduler#

• Basic Idea: Each organization (or queue) is assigned a slice of cluster capacity.
• Use Case: Multi-tenant cluster with guaranteed resource minimums for each queue.
• Benefits: Allows unused resources from one queue to be borrowed by others, ensuring high utilization.
• Configuration: Queues have properties like capacity, maximum capacity, and user limits.

Fair Scheduler#

• Basic Idea: Dynamically balances resources so that all jobs get roughly an equal share.
• Use Case: Enables multiple jobs to progress at similar rates.
• Configuration: Pools (schedulers) can be defined with weights, controlling how resources are distributed.

Comparative Table of Schedulers#

CPU and Memory Allocation#

1. Node Manager Resource Definitions:

   • yarn.nodemanager.resource.memory-mb: Total memory available for containers on a NodeManager.
   • yarn.nodemanager.resource.cpu-vcores: CPU cores available for containers.

2. Minimum and Maximum Container Sizes:

   • yarn.scheduler.minimum-allocation-mb: The smallest container that YARN can allocate to an application.
   • yarn.scheduler.maximum-allocation-mb: The largest container chunk YARN can allocate.

3. Balancing CPU vs. Memory Constraints:

   • If your job is memory-intensive, allocate more memory.
   • For CPU-bound tasks, ensure sufficient vcores are available to maximize parallelism.

Queue Configuration Strategies#

1. Capacity Scheduler Queues:

   • Example queue definitions in capacity-scheduler.xml:

     1
     <property>
     2
       <name>yarn.scheduler.capacity.root.default.capacity</name>
     3
       <value>50</value>
     4
     </property>
     5
     <property>
     6
       <name>yarn.scheduler.capacity.root.research.capacity</name>
     7
       <value>50</value>
     8
     </property>
     

   • The above example splits cluster capacity between a default queue and a research queue, each getting 50% of resources.

2. Fair Scheduler Pools:

   • Pools can be configured in the fair-scheduler.xml using <pool> elements.
   • Assign priority or weight to each pool to reflect business needs.

Monitoring Tools#

• ResourceManager Web UI (port 8088): Check resource usage, queue status, application wrappers.
• NodeManager Web UI (port 8042): Inspect container logs, local resource usage.
• Third-party Tools: Tools like Cloudera Manager or Ambari provide consolidated dashboards for deeper analysis.

YARN CLI Reference#

The YARN command-line interface (CLI) offers direct control over applications and queues. Here are some commonly used commands:

1
# List all submitted applications
2
yarn application -list
3

4
# Kill a specific application by its application ID
5
yarn application -kill application_1630000000000_0001
6

7
# Display cluster metrics
8
yarn cluster --listresource
9

10
# Check the logs for an application
11
yarn logs -applicationId application_1630000000000_0001

Launching a Simple MapReduce Job#

When running MapReduce jobs on YARN, the job submission goes through the Resource Manager, which then negotiates container resources:

1. Step 1: Submit the job:

   1
   hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount /input /output
   

2. Step 2: Resource Manager spawns a MapReduce Application Master.
3. Step 3: The Application Master requests containers from the Node Manager for map tasks and reduce tasks.
4. Step 4: Once completed, you can view the results in /output and check the metrics via the web UI or CLI.

Advanced Workflow with Spark on YARN#

Running Spark on YARN can streamline your data pipelines:

1
spark-submit \
2
  --class org.apache.spark.examples.SparkPi \
3
  --master yarn \
4
  --deploy-mode cluster \
5
  /path/to/spark-examples.jar \
6
  10

• Cluster Deploy Mode: The driver runs inside the cluster.
• Client Deploy Mode: The driver runs on the client machine.
• Resource Allocation: Spark executors are launched in YARN containers, and their resource consumption is governed by synergy between Spark’s configuration (like spark.executor.memory) and YARN’s scheduling algorithms.

Enabling Secure Clusters#

Security in YARN involves setting up Kerberos authentication, configuring service-level authorization, and securing data in transit. Key points:

1. Kerberos: Principal-based authentication for Hadoop components.
2. SSL/TLS: Secure the communication between clients and YARN daemons.
3. Access Control Lists (ACLs): Limit who can submit jobs or administer queues.

A snippet from yarn-site.xml for secure contexts:

1
<property>
2
  <name>yarn.acl.enable</name>
3
  <value>true</value>
4
</property>
5
<property>
6
  <name>yarn.admin.acl</name>
7
  <value>alice,bob</value>
8
</property>
9
<property>
10
  <name>yarn.resourcemanager.webapp.cross-origin.enabled</name>
11
  <value>false</value>
12
</property>

YARN High Availability Setup#

High availability (HA) eliminates single points of failure in the Resource Manager. Typically, you have:

1. Active Resource Manager: Responsible for scheduling.
2. Standby Resource Manager: Ready to take over if the active RM fails.
3. Zookeeper Integration: Coordinates failover.

Configuration typically involves specifying multiple RM hostnames:

1
<property>
2
  <name>yarn.resourcemanager.ha.enabled</name>
3
  <value>true</value>
4
</property>
5
<property>
6
  <name>yarn.resourcemanager.cluster-id</name>
7
  <value>yarn-cluster</value>
8
</property>
9
<property>
10
  <name>yarn.resourcemanager.ha.rm-ids</name>
11
  <value>rm1,rm2</value>
12
</property>
13
<property>
14
  <name>yarn.resourcemanager.hostname.rm1</name>
15
  <value>rm1.example.com</value>
16
</property>
17
<property>
18
  <name>yarn.resourcemanager.hostname.rm2</name>
19
  <value>rm2.example.com</value>
20
</property>

Containerization and Docker Integration#

• Running Docker Containers on YARN: YARN can launch Docker containers on NodeManagers, isolating dependencies for applications.
• Benefits: Consistent environments, simplified deployments, and better isolation.
• Configuration: You need to enable the container-executor support and configure relevant Docker parameters in yarn-site.xml.

Federation for Large Clusters#

YARN Federation allows multiple YARN clusters to combine into a single, infinitely scalable logical cluster. This setup is ideal for geographically distributed data centers or extremely high-scale operations. Federation ensures that multiple Resource Managers can respond to scheduling requests while presenting a unified interface to applications.

Future Trends#

1. GPU and FPGA Integration: YARN is evolving to schedule heterogeneous resources, especially for AI/ML workloads that need accelerators.
2. Edge Computing: Explore how YARN can manage resources not just in data centers but also in edge scenarios.
3. Serverless Architectures: Investigating beyond container-based scheduling to ephemeral, function-level resource management.