Behind the Scenes of Hadoop: How YARN Handles Resources#

What Is YARN?#

YARN stands for Yet Another Resource Negotiator. It was introduced in Hadoop 2.x as a key component enabling multiple data processing engines (MapReduce, Spark, Tez, etc.) to run simultaneously on the same cluster.

YARN essentially decouples resource management from job scheduling. This means that instead of tying resource allocation directly to the MapReduce job flow, there is a general-purpose resource manager that can flexibly allocate resources to different types of applications.

Why YARN?#

Before YARN, Hadoop used a monolithic approach called the MapReduce JobTracker/TaskTracker model:

• The JobTracker did both cluster resource management and job scheduling.
• This made it complex to run anything other than MapReduce jobs in a Hadoop cluster.
• Coupling scheduling and cluster resource management limited Hadoop’s extensibility and scalability.

YARN resolves this by:

1. Separating cluster resource management (handled by the ResourceManager) from specific application logic (handled by per-application ApplicationMasters).
2. Enabling multiple parallel processing frameworks to coexist and share resources, allowing for more flexible data processing pipelines.

ResourceManager#

The ResourceManager is the ultimate authority on resource allocation for the entire cluster. It receives resource requests from various applications and decides how many containers each application can run. The RM runs two essential services:

• Scheduler: Allocates resources in response to resource requests based on scheduling policies (FIFO, Capacity, Fair, etc.).
• ApplicationManager: Manages the lifecycle of applications as a whole. It negotiates the first container for an application where its ApplicationMaster will run.

If you consider YARN analogous to an operating system, the ResourceManager is like a “kernel-level” authority distributing CPU and memory among processes.

NodeManager#

The NodeManager runs on each worker node of the cluster. Its main duties are:

• Managing the local resources (CPU, memory, disk) on that node.
• Launching and monitoring containers assigned by the ResourceManager.
• Enforcing resource usage constraints (killing containers that exceed their memory limits, for example).

The NodeManager reports resource usage back to the ResourceManager, so the cluster as a whole stays balanced.

ApplicationMaster#

Each application (e.g., a MapReduce job, Spark application, Hive query, etc.) has its own ApplicationMaster. The AM is responsible for:

1. Negotiating with the ResourceManager for resources (containers).
2. Working with NodeManagers to execute tasks (launch containers).
3. Managing the application’s logic (e.g., the job’s DAG, progress tracking, fault tolerance, etc., depending on the framework).

Essentially, the AM is your application’s personal coordinator, communicating with the cluster’s ResourceManager for partial slices of the cluster resources.

Containers#

In YARN, a container is the basic unit of physical resources for an application. It is assigned a specific amount of memory, CPU cores (vCores), disk, etc., to run a particular task. Containers sandbox individual tasks, ensuring one runaway process doesn’t consume the resources reserved for other tasks or applications.

1. Resource Containers#

Containers hold the resources. An application can request and launch multiple containers. The AM typically requests additional containers after it starts to bootstrap parallel tasks.

Examples of container resource definitions:

• Memory (in MB)
• CPU cores (vCores)
• Disk resources (less commonly used, but can be configured)

2. Scheduling#

The ResourceManager runs a Scheduler service to control how container capacity is allocated. It receives resource requests from each AM and then decides which request to satisfy (and in what order) based on the configured scheduling policy.

3. Resource Requests#

An AM makes resource requests by specifying:

1. The capability required (memory, CPU).
2. The location preferences (e.g., to place tasks close to the data, it may request containers on specific nodes or racks).
3. The priority of the request.

4. Container Launch#

When the Scheduler allocates a container, the AM is notified and communicates with the relevant NodeManager to launch the container (which runs the specific task or process).

FIFO Scheduler#

• First-In-First-Out (FIFO) is the simplest algorithm.
• It queues up applications in the order they arrive and allocates resources (containers) accordingly.
• Although easy to understand, it lacks sophisticated resource-sharing capabilities (e.g., guaranteeing certain percentages of resources to certain departments).

Capacity Scheduler#

• Designed for multi-tenant environments where each organization or department has access to a secure “queue” with a guaranteed fraction of resources in the cluster.
• Each queue can also have sub-queues.
• Administrators can set capacity limits to ensure one queue doesn’t starve others while also making use of any unused capacity.

Fair Scheduler#

• Aims to fairly distribute resources among multiple applications (or users).
• Idle users or applications don’t hog resources if they stand idle; the available resources are shared fairly among active users.
• Allows enforced min/max resource limits, scheduling policies, and hierarchical queues.

Application Priority#

In a multi-user environment, you might need certain apps to have higher priority for resources. YARN supports an Application Priority feature, allowing users or administrators to assign priorities to applications within the same queue. Applications with higher priority requests are serviced first (subject to resource availability).

Work-Preserving Restart#

This feature allows the ResourceManager to restart without necessarily killing all the running applications. It preserves state, so your applications continue running during an RM restart. This is critical for high availability in production environments.

Docker Integration#

Recent Hadoop distributions allow running YARN containers in Docker images. This provides additional isolation and makes it easier to manage dependencies for different applications that might require conflicting libraries or frameworks. Administrators can specify Docker images for tasks, and the NodeManager will launch containers using Docker.

Resource Types Beyond CPU/Memory#

While CPU cores and memory have always been the primary resource types, advanced deployments can extend YARN to track and schedule GPU and FPGA resources (particularly important in machine learning and deep learning use cases).

1. Basic YARN Commands#

You can interact with YARN using a set of command-line tools. Here are a few handy commands:

1
# List all running applications
2
yarn application -list
3

4
# Kill an application
5
yarn application -kill <application_id>
6

7
# Check container logs
8
yarn logs -applicationId <application_id>

Monitor node statuses:

1
# List available cluster nodes
2
yarn node -list
3

4
# Show detailed node information
5
yarn node -status <node_id>

2. Example: Configuring Yarn-Site.xml#

Part of customizing YARN’s behavior involves editing the yarn-site.xml file. For instance:

1
<configuration>
2

3
  <!-- Maximum memory (in MB) for a NodeManager -->
4
  <property>
5
    <name>yarn.nodemanager.resource.memory-mb</name>
6
    <value>8192</value>
7
    <description>Total memory available to be allocated by containers on a node.</description>
8
  </property>
9

10
  <!-- Maximum vCores for a NodeManager -->
11
  <property>
12
    <name>yarn.nodemanager.resource.cpu-vcores</name>
13
    <value>4</value>
14
    <description>Total CPU cores available to be allocated by containers on a node.</description>
15
  </property>
16

17
  <!-- Minimum allocation for each container request -->
18
  <property>
19
    <name>yarn.scheduler.minimum-allocation-mb</name>
20
    <value>1024</value>
21
  </property>
22

23
  <!-- Maximum allocation for each container request -->
24
  <property>
25
    <name>yarn.scheduler.maximum-allocation-mb</name>
26
    <value>8192</value>
27
  </property>
28

29
</configuration>

3. Using the Capacity Scheduler#

In the Capacity Scheduler, resources are divided by configuring multiple named queues, each with a guaranteed share of the cluster. Here is an example snippet in capacity-scheduler.xml:

1
<property>
2
  <name>yarn.scheduler.capacity.root.queues</name>
3
  <value>default,analytics</value>
4
</property>
5

6
<property>
7
  <name>yarn.scheduler.capacity.root.default.capacity</name>
8
  <value>50</value>
9
</property>
10

11
<property>
12
  <name>yarn.scheduler.capacity.root.analytics.capacity</name>
13
  <value>50</value>
14
</property>

In this configuration, there are two queues (default and analytics), each allocated 50% of cluster resources by default.

4. Table of Common YARN Configuration Properties#

Below is a brief table of commonly tuned YARN properties and their typical usage:

1. YARN Federation#

YARN Federation allows you to combine multiple, possibly geographically distributed YARN clusters into a single federated cluster. This can be vital for organizations with multiple data centers wanting to manage resources more cohesively.

2. Queue Management Strategies#

Some enterprises create multiple levels of hierarchical queues, with each department or project having sub-queues. Combining the advanced features of the Capacity Scheduler (such as user limits, ACLs) ensures large organizations can operate effectively without stepping on each other’s toes.

3. ApplicationMaster Plugins#

For specialized scheduling requirements, companies can write custom AM logic or even custom schedulers within the ResourceManager. A well-crafted plugin can handle advanced tasks, such as bin-packing for memory-intensive machine learning tasks or exposing GPU resources to container requests.

4. Container Reuse and Speculative Execution#

MapReduce jobs in YARN can reuse containers between tasks to reduce container launch overhead. Speculative execution spawns extra tasks (for slow-running tasks) to speed up job completion. Tuning these can boost performance in production clusters.

5. Real-Time Streaming + Batch Mixed Workloads#

Modern big data pipelines often combine batch and streaming jobs on the same cluster. Tuning YARN’s scheduling options (e.g., setting priority for streaming tasks) ensures low-latency pipelines don’t get blocked by huge batch jobs.

6. Deep Integration with Kubernetes#

Although YARN and Kubernetes are distinct systems, modern enterprise platforms sometimes integrate the two. Spark on K8s is popular, but some multi-layer solutions route YARN requests to Kubernetes as a back-end orchestrator.