MLPerf Essentials: How to Compare AI Accelerators#

Core Principles#

1. Fairness: MLPerf ensures that tested systems follow a common set of rules and best practices.
2. Transparency: MLPerf submissions must be reproducible and undergo peer evaluation.
3. Diversity of Use Cases: Benchmarks range from computer vision tasks to natural language processing (NLP), recommender systems, speech recognition, and more.

MLPerf Training#

Focused on the performance of training large neural networks, the MLPerf Training benchmark covers tasks like:

• Image classification (ResNet-50)
• Object detection (Mask R-CNN, SSD-ResNet34)
• Natural language processing (Transformer, BERT)
• Recommendation systems (DLRM)
• Reinforcement learning (MiniGo)

In the training division, systems must be measured on time-to-train, maintaining specified accuracy thresholds. This time-to-train metric captures how quickly hardware and software stacks can converge the model to a state-of-the-art baseline accuracy.

MLPerf Inference#

Whereas training can be extremely time-consuming and compute-intensive, inference often needs to be hyper-optimized for latency and throughput. MLPerf Inference benchmarks:

• Test real-time scenarios such as speech recognition, object detection, and image classification.
• Evaluate throughput (samples processed per second) and latency (time to respond).
• Include both offline and online modes, simulating batch-based inference and interactive streaming use cases.

MLPerf HPC#

High-Performance Computing (HPC) merges machine learning with large-scale scientific computing. The MLPerf HPC benchmark includes workloads that involve:

• Climate modeling
• Computational fluid dynamics
• Physics simulations

These stress tests reveal how well accelerators handle tasks that require both physics-based calculations and machine learning inference/training, often in massive compute clusters.

MLPerf Tiny#

Designed for resource-constrained environments such as microcontrollers and embedded devices, MLPerf Tiny focuses on:

• Memory constraints
• Ultra-low power consumption
• Simple neural network architectures

This category is critical for edge AI solutions like IoT devices and real-time control systems, where both performance and energy efficiency are paramount.

Setting Up Your Environment#

1. Choose Your Benchmark Suite: Decide which MLPerf suite (Training, Inference, HPC, or Tiny) you want to focus on.
2. Get the Official MLPerf Repositories:
   • Visit the MLCommons GitHub to find the official MLPerf repository.
3. Install Dependencies:
   • Python (3.6+), Docker, and hardware-specific drivers.
4. Prepare Data:
   • Download or create datasets required for each benchmark (e.g., ImageNet for ResNet-50, COCO for object detection).

Running Benchmark Scripts#

After configuring your environment, you’ll run a set of scripts that compile or fetch the relevant model architectures and data. The steps usually include:

1. Cloning the MLPerf repository
2. Compiling or building Docker images
3. Loading or generating the dataset
4. Launching the benchmark with a command-line script

During each run, you can specify your hardware accelerator, batch sizes, or other configuration parameters. MLPerf standardizes many of these options to ensure fair comparisons.

Code Snippet: Sample Dockerfile#

Below is a simplified Dockerfile to illustrate how one might set up a container for a particular MLPerf benchmark (e.g., Inference on ResNet-50). Note that this snippet omits various details for brevity and is only illustrative:

1
FROM nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04
2

3
# Install dependencies
4
RUN apt-get update && \
5
    apt-get install -y python3 python3-pip git
6

7
# Install MLPerf Inference dependencies
8
RUN pip3 install --upgrade pip setuptools wheel
9
RUN pip3 install tensorflow==2.6.0
10

11
# Clone MLPerf repository (pseudo-example)
12
RUN git clone https://github.com/mlcommons/inference.git /mlperf_inference
13

14
WORKDIR /mlperf_inference
15
RUN chmod -R +x scripts/
16

17
# Entry point to run the benchmarks
18
ENTRYPOINT ["./scripts/run_inference.sh"]

To build the Docker image:

1
docker build -t mlperf-inference .

Once the container is built, you can run the benchmark in a controlled environment:

1
docker run --runtime=nvidia --rm mlperf-inference

Throughput#

• Definition: The number of processed samples or images per second.
• Interpretation: Higher throughput often indicates an accelerator’s capacity to handle bigger or more frequent batches, which is crucial in data centers or batch-processing scenarios.

Latency#

• Definition: The time taken to process one unit of data, typically measured in milliseconds.
• Interpretation: Low latency is crucial for interactive or real-time applications like autonomous vehicles or live video analytics.

Accuracy and Quality#

• Definition: The performance of the model on its target metric (e.g., accuracy for classification, mAP for object detection, BLEU for language translation, etc.).
• Interpretation: Even if an accelerator claims high throughput, it is important to confirm that it achieves the required accuracy threshold set by MLPerf to avoid subpar model quality.

Power Consumption#

• Definition: The amount of power in watts used by the accelerator during operation.
• Interpretation: Power efficiency (throughput per watt) is often critical for large-scale deployments and edge devices. Some MLPerf results include energy usage data to help you weigh cost vs. performance.

Benchmark Scores and Rankings#

1. Throughput Rankings: Systems are grouped by how many samples or images they can process per second.
2. Time-to-Train: For training tasks, the time to reach the target accuracy.
3. Latency: In many real-time tasks, the 99th percentile latency is a common measure.

Each of these metrics helps you judge how well a setup might serve your use case. Some MLPerf results highlight power consumption and cost trade-offs, if vendors have disclosed such information.

Comparative Scaling#

You will often see linear or near-linear scaling in MLPerf submissions when adding more accelerators. For example, if your system doubles the number of GPUs, you should see close to twice the throughput. Unexpectedly low scaling can reveal bottlenecks in networking, data loading, or memory bandwidth.

Usability and Ecosystem Fit#

While MLPerf offers vital performance data, real-world usability factors also come into play:

• Software Ecosystem: Does your preferred deep-learning framework (PyTorch, TensorFlow) run smoothly on the accelerator?
• Community Support: Is there vibrant forum or vendor documentation for debugging issues?
• Compatibility: Does the accelerator fit into your organization’s existing data center components or edge systems?

A top-ranked MLPerf solution may be less attractive if your team has to significantly alter its codebase or if driver support is immature.

Optimizations for MLPerf#

1. Graph Optimization: Convert your model’s computational graph to remove redundant operations or fuse compatible layers.
2. Mixed Precision Training/Inference: Lower floating-point precision (FP16 or even INT8) can speed up calculations on modern accelerators without significant accuracy loss.
3. Kernel Fusion: Combine multiple operations such as convolution and activation into a single GPU kernel.

Hyperparameter Tuning#

While MLPerf sets strict guidelines for certain hyperparameters (to ensure comparability), you still have some flexibility. Adjusting batch size, learning rates, or parallelization strategies can yield better performance on specific hardware.

Distributed Architectures#

For extremely large models or datasets, you may want to distribute training or inference across multiple nodes. MLPerf results can show how well an accelerator’s performance scales from one node to many:

• Interconnect: Performance across distributed clusters often hinges on interconnect speed (e.g., InfiniBand vs. Ethernet).
• Collective Operations: Distributed training solutions often require optimized collective operations like AllReduce or AllGather.

Robustness and Model Generalization#

MLPerf typically sets accuracy thresholds tailored to the model and task. Nevertheless, you might push your system to see if it can exceed those thresholds, possibly indicating better model generalization. In scenarios such as medical imaging, even small accuracy improvements can be vitally important.