From Cluster Chaos to Harmony: Taming Resources with YARN#

Early Hadoop Limitations#

In Hadoop’s early days, MapReduce was king. The JobTracker, a single machine, handled resource allocation, job scheduling, and fault tolerance for thousands of tasks. While this worked initially, one glaring weakness was the single point of failure. When the JobTracker was overloaded or failed, the entire cluster ground to a halt.

Furthermore, MapReduce’s processing was inherently batch-oriented. As new data processing paradigms appeared—interactive queries, machine learning, graph analytics, streaming—it became clear that Hadoop’s resource management needed a more flexible and general-purpose approach. YARN was born to fill that gap.

YARN’s Mission#

YARN aims to separate resource management from the application logic. It provides a ResourceManager that knows about all the cluster’s resources and an ApplicationMaster that knows about job-specific tasks. This separation of concerns ensures that each cluster node runs tasks side by side—whether MapReduce, Spark, or anything else—without stepping on each other’s toes.

For organizations wrestling with numerous data processing demands, YARN reduces complexity by decoupling compute and resource allocation. It effectively transforms the Hadoop cluster into a multi-tenant, dynamic operating system for big data jobs.

ResourceManager: The Cluster Maestro#

The ResourceManager is the single entry point for all job submission requests. Think of it as the conductor of an orchestra. It knows how many resources are available across the entire cluster, sets global policies, and delegates tasks to the right nodes.

Key sub-components of the ResourceManager include:

1. Scheduler: Allocates resources based on configured policies (such as FIFO, Fair, or Capacity scheduling). It doesn’t track application status or restart tasks; its sole job is to allocate containers to applications.
2. ApplicationsManager (ASM): Manages the submission and monitoring of applications. When a job is submitted, the ASM hands off resources to the Scheduler so that an ApplicationMaster for that specific application can be launched.

NodeManager: The Workhorse#

While the ResourceManager orchestrates globally, the NodeManager is responsible for managing resources on a single node within the cluster. Each node runs exactly one NodeManager, which listens for instructions to launch or kill containers. Tasks running in these containers periodically report back resource usage (CPU, memory, disk, etc.) to the NodeManager.

Responsibilities of the NodeManager include:

• Launching and monitoring containers.
• Reporting container status and resource metrics back to the ResourceManager.
• Managing logs and data for tasks running on the node.

ApplicationMaster: The Localized Brain#

The ApplicationMaster is a process launched for each application. It negotiates resources from the ResourceManager, decides which tasks to run and where, and reports job progress.

Unlike the old JobTracker that oversaw every detail for every job, the ApplicationMaster only cares about its own application. This division of labor allows YARN to scale better, as there is no single master node that gets overloaded.

Containers: The Execution Units#

Containers, allocated by the ResourceManager, encapsulate the resources (CPU, memory, and potentially other metrics) required to run tasks. In a typical MapReduce job, mappers and reducers each run in their own container. Other applications (Spark executors, for example) also run within containers, but their exact processes differ based on the framework.

Prerequisites#

To set up YARN, you should already have:

• A working Hadoop installation (HDFS).
• Properly configured Java environment on each cluster node.
• Sufficient networking and DNS resolution for node discovery.

Suppose you have a Hadoop cluster installed in the default directory /usr/local/hadoop, and you have SSH access to all data nodes. Your next steps will revolve around adjusting configuration files for YARN-specific parameters.

Configuration Files#

1
<configuration>
2
  <property>
3
    <name>yarn.resourcemanager.hostname</name>
4
    <value>master-node</value>
5
    <description>The hostname of the ResourceManager</description>
6
  </property>
7

8
  <property>
9
    <name>yarn.nodemanager.resource.memory-mb</name>
10
    <value>4096</value>
11
    <description>Maximum memory a NodeManager can allocate to containers (in MB)</description>
12
  </property>
13

14
  <property>
15
    <name>yarn.scheduler.minimum-allocation-mb</name>
16
    <value>256</value>
17
    <description>Minimum container allocation, in MB</description>
18
  </property>
19

20
  <property>
21
    <name>yarn.scheduler.maximum-allocation-mb</name>
22
    <value>2048</value>
23
    <description>Maximum container allocation, in MB</description>
24
  </property>
25
</configuration>

1
<configuration>
2
  <property>
3
    <name>mapreduce.framework.name</name>
4
    <value>yarn</value>
5
  </property>
6

7
  <property>
8
    <name>mapreduce.jobhistory.address</name>
9
    <value>master-node:10020</value>
10
  </property>
11

12
  <property>
13
    <name>mapreduce.jobhistory.webapp.address</name>
14
    <value>master-node:19888</value>
15
  </property>
16
</configuration>

Steps to Bring Up YARN#

1. Update Configuration: Ensure your yarn-site.xml and mapred-site.xml are placed on every node or accessible via a configuration management tool.
2. Format & Start HDFS (if not already):

   1
   hdfs namenode -format
   2
   start-dfs.sh
   

3. Start ResourceManager & NodeManagers:

   1
   start-yarn.sh
   

4. Validate Services: Navigate to the ResourceManager UI (commonly at http://master-node:8088). You should see the cluster resource summary.

If all services are running, your cluster should now be ready to take advantage of the YARN resource scheduling engine.

FIFO Scheduler#

First-In-First-Out (FIFO) is the simplest scheduler. Applications are queued, and the earliest submitted job gets processed first until it completes. While trivial to set up, FIFO lacks any notion of fairness or guaranteed capacities. This setup works well for small clusters with few applications but is generally not ideal for multi-tenant environments.

Capacity Scheduler#

Developed primarily at Yahoo, the Capacity Scheduler is designed for large, shared clusters. It enforces strict capacity guarantees, ensuring that each organization, department, or user group can get a certain percentage of the cluster’s resources. The concept of queues is central: each queue is allocated a percentage of the total cluster resources.

Properties like yarn.scheduler.capacity.root.default.capacity=50 can define queue capacities. If one queue is underutilized, its resources can (optionally) be borrowed by other queues. This ensures high cluster utilization while allowing different groups to share resources effectively.

Fair Scheduler#

The Fair Scheduler, popularized by Facebook, focuses on distributing resources so that all running applications get, on average, an equal share over time. An application that starts later can still catch up and receive its fair share once resources become available.

Weighted scheduling is also possible, where certain applications get more priority. This is useful in cases where you want approximate fairness but need to prioritize some jobs over others.

Spark on YARN Example#

1
spark-submit \
2
  --class org.apache.spark.examples.SparkPi \
3
  --master yarn \
4
  --deploy-mode cluster \
5
  /path/to/spark-examples.jar

Here, YARN launches a Spark driver in the ApplicationMaster, and each executor runs in its own container on a NodeManager.

Container Allocation#

YARN tracks resource requests in terms of “containers.” A container can be defined by multiple dimensions—commonly CPU cores and memory. For example, a container might request 2 CPU cores and 1024 MB of memory. The Scheduler then finds a suitable node that can fulfill this request.

Resource Guarantees vs. Resource Limits#

YARN can employ policies at two levels:

• Minimum and maximum container sizes: These constraints prevent jobs from asking for unreasonably large or small resources.
• Cluster-level resource monitoring: Even if a container is allocated 1 GB, jobs that exceed this limit can be killed (depending on configuration and available over-allocation policies).

Proper settings for yarn.scheduler.minimum-allocation-mb and yarn.scheduler.maximum-allocation-mb help maintain a balance between fine-grained scheduling (small container sizes) and overhead (each container imposes some overhead in terms of management).

Container Memory and CPU#

• yarn.nodemanager.resource.memory-mb: Total memory available on the NodeManager for container usage.
• yarn.nodemanager.resource.cpu-vcores: Total CPU cores (virtual cores) available to containers.

Ideally, you set these values to slightly less than physical resources to leave room for OS overhead. If your node has 32 GB RAM and 8 cores, you might set yarn.nodemanager.resource.memory-mb to 29696 (around 29 GB) and yarn.nodemanager.resource.cpu-vcores to 7, leaving some breathing space for system processes.

MapReduce-Specific Tuning#

• mapreduce.map.memory.mb and mapreduce.reduce.memory.mb: Controls how much memory each mapper and reducer is allocated.
• mapreduce.map.java.opts and mapreduce.reduce.java.opts: Adjusts the JVM heap size within each container. Typically set at 80% of the container memory to allow overhead for the container itself.

As an example:

1
<property>
2
  <name>mapreduce.map.memory.mb</name>
3
  <value>1024</value>
4
</property>
5

6
<property>
7
  <name>mapreduce.map.java.opts</name>
8
  <value>-Xmx820m</value>
9
</property>

Web Interfaces#

• ResourceManager UI: Default at http://<resource-manager-host>:8088. Lets you see running applications, resource usage, and application states.
• NodeManager UI: Each NodeManager typically hosts a web UI on port 8042. It shows container logs and resource usage.

Application Logs and Log Aggregation#

By default, container logs might be stored locally on each node. However, YARN can aggregate logs into HDFS for centralized access. Adjusting properties like yarn.log-aggregation-enable can help you quickly view logs without SSH-ing into multiple nodes.

Common Issues#

1. Container Killed by YARN: Could be triggered by memory overruns. Ensure your application’s memory request and usage align.
2. ResourceManager Overloaded: The RM can become a bottleneck if too many concurrent applications demand resources. Increasing RM heap size or distributing load via queue configurations can help.
3. NodeManager Crashes: Often tied to OS-level resource exhaustion or misconfigurations (e.g., specifying more memory to containers than physically available).

YARN High Availability (HA)#

For production environments, high availability is crucial:

• Active/Standby ResourceManagers: One RM is active, while another runs in standby mode. Using Apache ZooKeeper, you can automatically failover from active to standby upon failures.
• Automatic Client Failover: YARN clients can be configured to auto-reconnect to the new active ResourceManager.

Opportunistic Containers#

Newer YARN features (e.g., in Hadoop 3.x) introduce Opportunistic Containers, which can run at lower priority without immediately stealing resources from guaranteed containers. This is especially useful in environments that need QoS (Quality of Service) guarantees for critical jobs but also want to utilize free resources for best-effort tasks.

Federation#

YARN Federation allows multiple YARN sub-clusters to appear as one large, virtual YARN cluster. This is beneficial for organizations whose single cluster could span thousands of nodes, and they need to break it down into manageable sub-clusters without losing a global resource perspective.

Long-Running Services on YARN#

Beyond traditional short-lived batch jobs, YARN can run long-running services. For example, you might spin up a microservice or a streaming engine within a container for days, thanks to the improved node-level resource controls. Tools such as Apache Slider (now replaced by other solutions) and various Docker-on-YARN strategies have explored this domain.

Multi-Cluster Deployments#

In large enterprises, it’s common to have multiple YARN clusters for different teams or different data centers. You can tie them together with Federation or treat them as separate pools for different applications. The best approach depends on your organizational and networking constraints.

Integrating with Cloud#

YARN doesn’t mandate on-premise hardware. You can deploy Hadoop clusters in the cloud, automatically scaling NodeManagers based on demand. Tools like autoscaling groups can be configured to spin up new nodes (and hence new NodeManagers) when resource usage surpasses a defined threshold.

Containerization and Kubernetes#

YARN was designed before the rise of container orchestrators like Kubernetes. However, you can still run Hadoop on containerized infrastructure, bridging the gap between YARN resource management and Kubernetes cluster management. Some organizations run Hadoop workers within Kubernetes pods, or they integrate YARN directly into Docker containers to maintain consistent environments.