Troubleshooting YARN: Common Pitfalls and How to Avoid Them#

2.1 ResourceManager Internals#

The ResourceManager manages the entire cluster’s resource usage. It runs two main services:

• Scheduler: Allocates resources based on scheduling policies like CapacityScheduler or FairScheduler.
• ApplicationManager: Manages application submission, monitors existing applications, and restarts failed ApplicationMasters if needed.

2.2 NodeManager Responsibilities#

Each NodeManager takes directives from the ResourceManager. Responsibilities include:

• Container Lifecycle Management: Launches containers, monitors resource usage, and terminates containers once tasks finish.
• Log Management: Collects and stores logs from containers, typically shipping them to a distributed file system for centralized storage.
• Resource Isolation: Enforces resource limits (CPU, memory) using cgroups or other underlying mechanisms.

2.3 ApplicationMaster Role#

An ApplicationMaster is an application-specific entity responsible for:

• Negotiating Resources: Communicates with the ResourceManager to request containers and resources.
• Task Coordination: Oversees the application’s task scheduling, fault tolerance, and job progress.
• Progress Tracking: Reports job status back to the ResourceManager and eventually completes or fails.

Understanding these components lays the groundwork for diagnosing the root causes of YARN failures. Nearly all troubleshooting will involve verifying these roles function as intended.

3.1 Memory Misconfiguration#

A prevalent mistake is choosing incorrect memory settings. Each NodeManager has a finite memory pool, from which containers request memory. If you set the container memory too large, you can starve other applications. If it is too small, your tasks may fail with “OutOfMemoryError.”

1
<configuration>
2
  <!-- Physical memory on each NodeManager -->
3
  <property>
4
    <name>yarn.nodemanager.resource.memory-mb</name>
5
    <value>8192</value>
6
  </property>
7

8
  <!-- Maximum memory for a single container -->
9
  <property>
10
    <name>yarn.scheduler.maximum-allocation-mb</name>
11
    <value>4096</value>
12
  </property>
13

14
  <!-- Minimum memory for a single container -->
15
  <property>
16
    <name>yarn.scheduler.minimum-allocation-mb</name>
17
    <value>256</value>
18
  </property>
19
</configuration>

3.2 CPU Core Allocation#

Memory alone is insufficient for resource management. Failing to configure CPU cores properly can lead to bottlenecks and inefficient scheduling. Typically, these property names are analogous to those for memory but with “vcores” in place of “mb.” For instance:

• yarn.nodemanager.resource.cpu-vcores
• yarn.scheduler.minimum-allocation-vcores
• yarn.scheduler.maximum-allocation-vcores

You may have 16 CPU cores on each node, but not all of them are necessarily allocated to containers. Consider the overhead needed by system and cluster daemons.

3.3 Incorrect Scheduling Policies#

YARN’s Scheduler uses policies (such as CapacityScheduler, FairScheduler, or the deprecated FIFO) to allocate resources among multiple tenants or queues. A configuration mismatch here can cause some queues to hog resources while others starve.

Below is a simple table comparing schedulers:

Ensure that your chosen scheduler aligns with your performance and multi-tenancy requirements. For advanced configurations, specifically with the CapacityScheduler, define each queue’s capacities in the capacity-scheduler.xml file.

5.1 Log Analysis#

Logs are your most reliable sources of truth. Key log locations and tips:

• ResourceManager Logs: Typically located in $HADOOP_LOG_DIR/yarn. Look for exceptions related to queue capacity, application scheduling, or cluster resource constraints.
• NodeManager Logs: Typically found in $HADOOP_LOG_DIR/yarn. Review container launch failures, memory overuse messages, and other node-specific issues.
• ApplicationMaster Logs: Stored in the user logs directory (by default, YARN_NODEMANAGER_LOG_DIR/application_ID/). Here, you can see how your application requested resources and whether it encountered out-of-memory or classpath issues.
• Container Logs: Usually in the same directory as AM logs, with subdirectories for stderr, stdout, and syslog.

A recommended approach is to enable DEBUG level logs in yarn-site.xml during chronic issues; just remember to revert to a NORMAL or INFO level once you’re done to prevent log bloat.

5.2 Using the YARN Web UI#

The YARN ResourceManager web interface (by default on port 8088 if not overridden) provides real-time insights into queue usage, active/failed applications, resource distribution, and job states. For each job, you can view:

• Application Overview: Overall status, final state, progress.
• ApplicationMaster Tracking: Links to the ApplicationMaster’s web UI (if provided).
• Logs/Diagnostics: Summaries of container logs to spot failures.

By combining direct logs with the web UI’s summarized metrics, you can zero in on trouble spots more efficiently.

5.3 Command-Line Tools#

The Hadoop command-line interface can provide quick, text-based insights:

• yarn application -list: Shows running applications or all applications with additional flags.
• yarn application -status : Provides resource consumption, state, and diagnostic information.
• yarn logs -applicationId : Fetches container logs from HDFS or local, reducing the need to manually traverse directories.

Use these commands to quickly identify job states, container events, or locate relevant logs.

5.4 Resource Setup Checks#

Before suspecting complicated bugs, ensure your node-level resources are correctly recognized. For example, if a NodeManager can’t see all CPU cores or memory, it might be a host-level configuration issue (e.g., cgroups, Docker constraints, or virtualization).

5.5 Reproducing Issues in a Test Environment#

When feasible, replicate the job or workload on a smaller test cluster or a staging environment with identical configurations. This approach lessens the impact on production and helps you isolate the root cause under controlled conditions.

6.1 Container Overruns Memory#

A container that surpasses its allocated memory can trigger a kill event from the NodeManager. You might see logs like:

1
Container [pid=XXXX,containerID=container_e03_XXXX] is running beyond virtual memory limits.
2
Current usage: 1548MB of 1024MB physical memory used.

Causes:

• Insufficient container memory allocations.
• JVM settings such as Xmx exceeding container memory.
• Memory leaks in the application code.

Suggested Solutions:

1. Adjust Container Memory: Increase yarn.scheduler.maximum-allocation-mb if your tasks routinely require more memory than initially allocated.
2. Tune JVM Options: Ensure the -Xmx setting is within container memory limits. Often, clients inadvertently set large heap sizes that overshadow YARN’s container settings.
3. Inspect Code for Leaks: When suspecting memory leaks, use profilers like YourKit, VisualVM, or built-in Spark instrumentation to find unbounded data structures or unclosed file handles.

6.2 Unhealthy NodeManagers#

YARN NodeManagers can report themselves as “unhealthy.” Common triggers include:

• Disk usage exceeding thresholds.
• High system load or memory pressure.
• NodeManager local directories becoming full or inaccessible.

You can configure health checks in yarn-site.xml via properties such as yarn.nodemanager.disk-health-checker.min-healthy-disks or user-defined scripts that check node conditions.

Resolution Steps:

1. Inspect Node Logs: Identify the exact condition causing the node to be marked unhealthy.
2. Free Up Disk Space: Often, logs or temporary files fill up disks. Periodic cleanup or expanding disk capacities may be needed.
3. Adjust Disk Checker Thresholds: If nodes have large volumes, you may set thresholds too low. Increase them cautiously and monitor.
4. Check for Node Hardware Failures: Sometimes, underlying disk corruption or faulty memory can cause frequent node health issues.

6.3 Stuck Applications in ACCEPTED State#

You may notice your applications remain in the “ACCEPTED” state indefinitely. Possible reasons:

1. Resource Shortage: The cluster doesn’t have enough memory/CPU available.
2. Capacity Quota Exhaustion: If using the CapacityScheduler, the queue may have reached its maximum capacity.
3. Queue Configuration Constraints: If queue access policies or user limits are incorrectly configured, the application might never receive resources.

How to Diagnose:

• Check ResourceManager logs for queue capacity errors.
• Look at cluster metrics on the RM UI under the “Cluster” tab.
• Use the capacity-scheduler.xml to ensure the queue is configured with adequate resources for your application.

Fix:

• Increase queue capacity or run fewer concurrent jobs.
• Temporarily move certain applications to a less-loaded queue.
• Optimize container allocations so that each job uses only what it needs.

6.4 Classpath and Library Conflicts#

When containers fail to launch due to missing libraries or version conflicts, you’ll see messages such as “ClassNotFoundException” or “NoClassDefFoundError.” This usually stems from:

• Inconsistent versions across the cluster nodes.
• Missing dependencies in the application jar.
• Classpath Confusions if YarnConfiguration or other Hadoop libraries overshadow the application’s libraries.

Recommendations:

• Standardize Hadoop and YARN versions across all nodes.
• Bundle all necessary libraries in a fat jar or use distributed cache.
• Validate your classpath with debug logs or by using the —verbose option for the JVM.

6.5 Shuffle Failures#

For MapReduce or Spark applications, shuffle failures can occur when the NodeManager’s shuffle service is not running or is improperly configured. Typical error logs might say “Failed to retrieve shuffle data from .”

Remedies:

1. Enable Shuffle: In yarn-site.xml, ensure yarn.nodemanager.aux-services includes mapreduce_shuffle or spark_shuffle.
2. Check NM Aux Service Logs: Review NodeManager logs to see if the shuffle service started properly.
3. Network or Firewall Issues: Shuffle might be blocked by firewall rules. Ensure the relevant ports (often 13562 for MapReduce Shuffle in older releases, or ephemeral ports used by Spark) are open.

7.1 Container Reuse#

For MapReduce, turning on container reuse speeds up repeated tasks by reducing overhead in creating new containers. This is generally controlled by mapreduce.job.reduce.slowstart.completedmaps or similar properties.

7.2 Parallelism Tweaks#

Adjust the number of map and reduce tasks to avoid saturating or under-utilizing the cluster. If you have many small files, tasks might be created in large numbers, causing overhead. Consider using the CombineFileInputFormat or other consolidation strategies.

7.3 Speculative Execution#

Speculative execution helps mitigate slow-running tasks, but it can also waste resources if set incorrectly. If you see tasks running duplicates too often, reevaluate speculative execution settings like mapreduce.map.speculative and mapreduce.reduce.speculative.

7.4 Scheduler Configuration#

Improve scheduling throughput and fairness by fine-tuning:

• Maximum Container Allocation: If this is too large, it can starve smaller tasks.
• Queue Weights: In FairScheduler, adjust queue weights to reflect usage priorities.
• User Limits: Limit the maximum number of containers a single user can hold, preventing resource monopolization.

8.1 Node Labels#

YARN supports labeling nodes to allocate specific hardware resources (e.g., GPU nodes) to particular queues or applications. This is powerful but can lead to scheduling deadlocks if label capacities are not well-planned. For example, if an application requires GPU-labeled nodes but the cluster has no such nodes, the application will remain in ACCEPTED state forever.

Key Node Label Configurations:

• yarn.node-labels.enabled (boolean)
• yarn.node-labels.fs-store.root-dir (file path for label definitions)
• yarn.scheduler.capacity.root.default.accessible-node-labels

Regularly verify your node labels match actual hardware capacity and that your scheduling policies for those labels are realistic.

8.2 Multi-Tenancy and Security#

Complexities arise when multiple organizations share a cluster. Common issues include:

• Kerberos Misconfiguration: Jobs failing due to invalid tickets or expired tokens.
• Queue ACL Errors: Users receiving “AccessDenied” because of insufficient queue permissions.
• Delegation Token Issues: Temporary credentials not being renewed, leading to hdfs DFS read/write errors mid-job.

Best Practices for Secure Multi-Tenancy:

1. Enable Kerberos with all Hadoop components (HDFS, YARN, Hive, etc.).
2. Implement ACLs carefully in yarn-site.xml or capacity-scheduler.xml.
3. Use Delegation Token Renewal processes to avoid token expiration if your jobs run longer than the default period.

8.3 Federation and High Availability Configurations#

For very large clusters, you may opt for YARN Federation, which stitches multiple sub-clusters into a single logical resource. This helps scale horizontally but introduces complexity in routing application submissions and tracking job states across sub-clusters.

Common Pitfalls:

• Inconsistent Cluster Configurations: Sub-clusters might have different capacity-scheduler.xml or yarn-site.xml settings, leading to unexpected scheduling.
• Federation Router Failures: If the Federation Router is not configured to handle high request loads, the entire system can bottleneck.

For High Availability:

• ResourceManager HA involves using ZooKeeper to store application states. If ZK is not properly configured or is overloaded, failovers can be delayed or fail outright.
• Always monitor your ZK ensemble’s health and integrate it with cluster-level alerting.

11.1 Debugging Container Launch Failures#

Tools like jmap and jstack can help you diagnose memory usage and thread states. If a container is severely misbehaving, you can attach a profiler to its process ID. These steps, however, require NodeManager access and the appropriate OS privileges.

11.2 Optimizing for Low-Latency Apps#

For near real-time applications (Spark Streaming, Flink, Storm), you may need:

• Fine-Grained Resource Scheduling: Keep container sizes small for quick starts and stops.
• Priority Scheduling: Give streaming queues higher priority to reduce processing latency.
• Pre-Warmed Containers: In certain custom setups, maintain a pool of pre-warmed containers for immediate execution.

11.3 Large-Scale Cluster Diagnostics#

For clusters with thousands of nodes:

• Use Federation but be mindful of complex routing.
• Maintain a hierarchical scheduling structure.
• Segment the cluster physically (e.g., by department, region, or hardware type) and logically (by assigning node labels).
• Integrate with an enterprise-grade monitoring stack that can handle large data volumes.

12.1 Checking Node Status via CLI#

1
# List all active NodeManagers
2
yarn node -list
3

4
# Output example:
5
# Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers
6
# machine01:8041      RUNNING    machine01:8042          4
7
# machine02:8041      UNHEALTHY  machine02:8042          3

12.2 Quickly Gathering Logs for a Failed Job#

1
# Suppose your application ID is application_1633043785700_0001
2
yarn logs -applicationId application_1633043785700_0001 > app_logs.txt
3

4
# Then grep for memory errors
5
grep -i "memory" app_logs.txt

12.3 Resource Configuration Validation Script (Hypothetical Example)#

Below is a simple shell script that checks the total memory configured in yarn-site.xml matches the actual physical memory on the node.

1
#!/bin/bash
2

3
YARN_SITE_PATH="/etc/hadoop/conf/yarn-site.xml"
4
NM_MEM_PROP="yarn.nodemanager.resource.memory-mb"
5
ACTUAL_MEM_KB=$(grep MemTotal /proc/meminfo | awk '{print $2}')
6

7
# Convert KB to MB
8
ACTUAL_MEM_MB=$(( ACTUAL_MEM_KB / 1024 ))
9

10
CONFIGURED_MEM_MB=$(xmllint --xpath "string(//property[name='$NM_MEM_PROP']/value)" $YARN_SITE_PATH)
11

12
echo "Node physical memory (MB): $ACTUAL_MEM_MB"
13
echo "YARN configured memory (MB): $CONFIGURED_MEM_MB"
14

15
if [ $ACTUAL_MEM_MB -lt $CONFIGURED_MEM_MB ]; then
16
  echo "WARNING: YARN is configured to use more memory than physically available!"
17
else
18
  echo "OK: YARN memory configuration seems reasonable."
19
fi

While simplistic, small checks like these can prevent major issues where YARN is over-committing resources.