Unlocking Peak Performance: YARN Scheduler Secrets Revealed#

ResourceManager#

The ResourceManager (RM) is the central authority that decides how resources should be allocated among different applications in the cluster. It houses two sub-components crucial for scheduling:

1. Scheduler – Implements scheduling policies and enforces resource allocation based on defined constraints and configurations.
2. ApplicationsManager – Manages the job submission process and keeps track of the application lifecycle.

Because the ResourceManager makes global scheduling decisions, correct configuration is vital for peak performance.

NodeManager#

A NodeManager (NM) runs on each node within the cluster and is responsible for:

• Managing the containers, which are the fundamental computation units in YARN.
• Tracking resource usage (CPU, memory, disk).
• Reporting health and resource availability to the ResourceManager.
• Managing logs and local storage.

Each NM ensures that local resources are efficiently utilized and enforced, so that tasks do not exceed their resource quotas.

ApplicationMaster#

Every user application that runs on YARN has its own ApplicationMaster (AM). The AM acts as the job-specific process that negotiates resources via the ResourceManager’s scheduler. It coordinates the execution of tasks, handles failures, and provides status updates. Different frameworks like Spark or MapReduce each implement their own AM logic, so they can optimize resource utilization according to specific workload needs.

FIFO Scheduler#

The FIFO (First In, First Out) Scheduler is the simplest available. It processes jobs in the order of their arrival. While straightforward, it lacks advanced features like resource sharing among multiple tenants. It’s rarely used in multi-tenant production environments but can be ideal for smaller, dedicated clusters where job order is strictly linear.

Capacity Scheduler#

The Capacity Scheduler is designed for multi-tenant environments. It divides cluster resources into hierarchical queues, each having a configured guaranteed capacity. This ensures predictable performance for different business groups or applications, while allowing unused capacity to be shared among other queues when available.

Fair Scheduler#

The Fair Scheduler aims to allocate resources so that each application gets its fair share of the cluster over time. Instead of relying on reserved capacities, it adjusts resource distribution dynamically based on current demand and utilization. It’s suitable for environments where varying workloads or interactive jobs coexist with batch processes.

Queues and Sub-Queues#

At its core, the Capacity Scheduler organizes resources into a tree-like hierarchy of queues. Each queue (or sub-queue) has a guaranteed share of resources expressed as a percentage of total cluster capacity. For example:

• root (100%)
  • marketing (30%)
    • data_import (15%)
    • analysis (15%)
  • engineering (70%)
    • etl (35%)
    • ml_research (35%)

In this structure, the “marketing” queue is allocated 30% of the cluster by default, split equally between “data_import” and “analysis” sub-queues. The “engineering” queue receives 70%, half of which goes to “etl” and half to “ml_research.”

Resource Allocation#

Each queue enforces a minimum capacity to guarantee baseline performance. When the capacities of other queues are idle, the scheduler may allow the queue to use more resources, up to a specified maximum capacity. Some key properties include:

• yarn.scheduler.capacity.root.marketing.capacity: Assigns minimum (guaranteed) capacity.
• yarn.scheduler.capacity.root.marketing.maximum-capacity: Limits maximum resources the queue can consume.
• yarn.scheduler.capacity.root.marketing.user-limit-factor: Controls the maximum share an individual user can claim within the queue.

Access Control Lists (ACLs)#

Administrators can set ACLs on queues to control which users and groups can submit jobs or administer queue configurations. This is particularly useful in multi-tenant environments where each queue might serve a different department.

Example Capacity Scheduler Configuration#

Below is a simple snippet from a capacity-scheduler.xml configuration file:

1
<property>
2
  <name>yarn.scheduler.capacity.root.queues</name>
3
  <value>marketing,engineering</value>
4
</property>
5

6
<property>
7
  <name>yarn.scheduler.capacity.root.marketing.capacity</name>
8
  <value>30</value>
9
</property>
10
<property>
11
  <name>yarn.scheduler.capacity.root.marketing.maximum-capacity</name>
12
  <value>60</value>
13
</property>
14

15
<property>
16
  <name>yarn.scheduler.capacity.root.engineering.capacity</name>
17
  <value>70</value>
18
</property>
19
<property>
20
  <name>yarn.scheduler.capacity.root.engineering.maximum-capacity</name>
21
  <value>100</value>
22
</property>
23

24
<property>
25
  <name>yarn.scheduler.capacity.root.marketing.user-limit-factor</name>
26
  <value>2</value>
27
</property>

In this configuration:

• “marketing” is guaranteed 30% of cluster resources but can go up to 60%.
• “engineering” is guaranteed 70% and can use up to 100%.
• Each user in “marketing” can consume up to twice the configured capacity when no other users are competing.

Pools and Fair Sharing#

In a Fair Scheduler context, resources can be divided into “pools,” each of which defines fair share policies. When multiple applications are running, the Fair Scheduler tries to equalize resource usage across all running applications or pools. The actual algorithm can be more sophisticated, taking into account weights and minimum allocations specified for each pool.

Fair Scheduler Preemption#

Preemption is a mechanism by which the Fair Scheduler reclaims resources from over-allocated applications to reassign them to under-served pools or applications. If one application grows beyond its fair share for too long, preemption can kill or move containers to align resource usage with overall fairness constraints. This ensures that long-running applications that hog resources don’t starve smaller but urgent jobs.

Example Fair Scheduler Configuration#

Below is a fair-scheduler.xml snippet illustrating a basic pool configuration:

1
<fairscheduler>
2
  <queue name="root">
3
    <queue name="production">
4
      <minResources>1024 mb,1 vcores</minResources>
5
      <weight>2.0</weight>
6
    </queue>
7
    <queue name="dev">
8
      <minResources>512 mb,1 vcores</minResources>
9
      <weight>1.0</weight>
10
    </queue>
11
  </queue>
12
  <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
13
</fairscheduler>

In this example:

• Two pools exist: “production” and “dev.”
• By assigning a weight of 2 to “production,” the scheduler prioritizes this pool twice as much as “dev.”
• “production” has a higher minimum resource reservation, ensuring a stronger guarantee under contested loads.

Scheduler Configuration Best Practices#

1. Define Clear SLAs – Before setting up your scheduler, establish service-level agreements (SLAs) for each queue or pool. This provides a measurable target for configuration decisions.
2. Use Priority and Weights – Both the Capacity and Fair Schedulers support priority definitions to differentiate critical workloads from less time-sensitive tasks.
3. Enable Preemption Carefully – Preemption can be a double-edged sword. It helps ensure fairness, but may disrupt long-running tasks. Balance these trade-offs by testing preemption in a controlled environment first.

Node and Container Sizing#

Proper sizing of containers (in terms of memory, CPU cores, and possibly disks) is crucial. Oversized containers can lead to reduced cluster utilization, while undersized containers may cause excessive overhead or thrashing. To find the sweet spot:

• Start with a single container size that is a small fraction of total node resources (e.g., 1/8 of each node) and gradually modify based on observed performance.
• Monitor resource usage metrics (CPU, memory, I/O) to fine-tune container allocations.
• Avoid extremes – both tiny and significantly large containers can result in diminished returns.

Monitoring YARN Scheduler Performance#

YARN provides a web UI (default port 8088) to monitor cluster resource usage, queue allocations, and running/finished applications. Additionally, metrics can be exported to external systems like Prometheus or Grafana. Key metrics to track include:

Tuning for High Concurrency#

High concurrency environments require careful balancing, so that many small to medium tasks can run simultaneously without starving. Here are some tactics:

• Limit the maximum number of containers per queue to prevent any single queue/user from saturating the cluster.
• Utilize CPU and memory reservations in advanced scheduling rules for best concurrency.
• Leverage container re-use features (especially in Spark) to reduce the overhead of container allocation and teardown.

Isolating Workloads#

A straightforward technique for isolating workloads is to allocate separate queues (or pools) for each user group, setting up definitive capacity, ACLs, and resource limits. Additionally, you can use cgroups or Docker containers at the operating system level to add stronger isolation around CPU, memory, and I/O usage.

Access Controls and User Management#

YARN’s Access Control Lists (ACLs) allow you to define which users can submit applications to a queue, administer that queue’s configuration, or view the logs of particular applications. Integrated with Kerberos-based authentication (in a secure Hadoop cluster), ACLs help ensure each user or team operates solely within their authorized domain.

Interactive Queries vs. Batch Processing#

Tools like Hive on Tez or Spark SQL may host interactive or near-interactive queries, where users expect results within seconds or minutes. In such a scenario:

• A Fair Scheduler with preemption can handle bursts of interactive jobs.
• Specific high-priority queues for interactive queries can guarantee minimal waiting time.
• Lower priority batch-processing jobs may run in the background without impacting interactive SLAs significantly.

ETL Pipelines and Scheduling Policies#

Large-scale ETL pipelines typically run on a schedule (e.g., nightly or hourly). With a Capacity Scheduler setup:

• You could dedicate a queue with a high guaranteed capacity for these ETL jobs.
• Since the workload is predictable, preemption or a strict queue capacity helps maintain a stable environment.
• Extremely critical ETL tasks can be placed in high-priority sub-queues to ensure they run without interruption.

Machine Learning Workloads#

Machine learning model training can be resource-intensive. In some cases, GPU resources are also involved. For these workloads, you can:

• Configure specialized queues or pools that enforce GPU resource requests.
• Limit concurrency for GPU-bound ML tasks to ensure stable performance.
• Use advanced scheduling plugins to account for additional hardware constraints like GPU memory and compute cores.

Advanced Queue Design Patterns#

For large organizations with complex demands, queue hierarchies can become intricate. Some design patterns include:

• Functional Segmentation – Creating top-level queues for broad functional areas (e.g., production, testing, ad-hoc analytics).
• Departmental and Role-Based Sub-Queues – Within each functional queue, sub-queues can group resources by department, project, or user role.
• Priority-Driven Queues – Enforcing strict priority rules within sub-queues to ensure that mission-critical tasks preempt lower-priority tasks.

Here is an example structure combining functional and role-based segmentation:

1
root
2
 ├── production
3
 │    ├── marketing
4
 │    └── engineering
5
 └── testing
6
      └── general

Dynamic Container Resource Reservation#

Some workloads demand dynamic resource adjustments over time—especially long-running services or streaming applications. With advanced YARN features and custom ApplicationMasters, containers can request additional resources mid-application lifecycle. This dynamic resource reservation can be beneficial when:

• A streaming job’s throughput suddenly spikes, requiring more CPU and memory.
• A long-running Spark job transitions from reading data to an intensive compute phase that can utilize more resources.

Though powerful, dynamic resource reservation requires careful configuration and close monitoring to prevent cluster-wide resource instability.

Future Trends and Ecosystem Shifts#

With the rise of container orchestration systems such as Kubernetes, some organizations are migrating away from YARN for new analytics workloads. However, YARN remains robust and is actively updated to integrate modern cluster management capabilities. Extending YARN with technologies like Apache Slider or adopting YARN Federation can help scale clusters beyond a single cluster resource manager. Keep an eye on:

• Native Support for Docker – YARN increasingly supports Docker containers for better resource isolation.
• GPU and FPGA Resource Scheduling – Emerging features target specialized hardware to meet the demands of AI and other HPC workloads.
• Hybrid Orchestration – Integrations that allow YARN to coexist with Kubernetes for an even broader resource pool.