Beyond Speed: What MLPerf Reveals About AI Hardware#

Early Benchmarks#

The first version of MLPerf focused on training tasks, covering key workloads like image classification (ResNet-50), object detection (SSD, Mask R-CNN), language modeling (Transformer, GNMT), and reinforcement learning (MiniGo). These initial tasks laid the groundwork for measuring training performance on widely accepted deep learning architectures. Over time, MLPerf expanded to include inference, HPC (high-performance computing) simulations, and more advanced models.

Key Contributors#

Big names in the hardware industry—NVIDIA, Intel, Google, and AMD—along with prominent research labs and startups, have contributed to MLPerf. Each submission to MLPerf must adhere to rigorous standards. This ensures reproducibility and fairness in results, making MLPerf a reliable source of performance data for customers and researchers alike.

Examples of Benchmarked Models#

• ResNet-50 for image classification (CIFAR, ImageNet, etc.)
• BERT for natural language processing (language understanding)
• SSD and Mask R-CNN for object detection
• GNMT for machine translation
• DLRM (Deep Learning Recommendation Model) for recommendation systems
• MiniGo for reinforcement learning tasks

Each model comes with specific quality thresholds (e.g., accuracy, BLEU score) that must be met to consider a run valid. This ensures that participants cannot cut corners at the expense of model quality.

Training#

For the training benchmark, one primary metric is the time-to-train, i.e., how long it takes for a given hardware-software stack to train a reference model to the required accuracy. Another metric is “throughput,�?often reported in examples per second, which indicates how many data samples can be processed per second during training.

Inference#

For inference, latency and throughput both matter. MLPerf Inference tasks often measure:

• Latency (ms/sample): The time it takes for the model to process a single input.
• Throughput (queries/second): How many samples can be processed in a second at scale.

Different scenarios include single-stream (one query at a time), multi-stream, server (simulating real-world requests), and offline modes. Each scenario tests a different aspect of the inference pipeline.

Additional Metrics#

• Power Efficiency (samples per joule): Some MLPerf submissions also track power usage. This metric is indispensable when evaluating large-scale data centers or battery-powered edge devices.
• Memory Usage: While not always a primary metric, memory constraints are monitored, especially in edge and HPC cases.

A Note on Deployment Geometries#

• Single Accelerator: Often used for inference or small-scale training where cost or simplicity is the primary factor.
• Multi-GPU/Accelerator: Common for advanced training scenarios (e.g., multi-node HPC clusters).
• CPU Offload/Task Splitting: Some systems rely on the CPU for certain operations like data preprocessing while the GPU or TPU handles matrix multiplications.

GPUs in a Typical Setup#

When you install a deep learning framework (e.g., PyTorch) on a machine with GPUs, you can offload tensor operations to the GPU by specifying device placement. For instance:

1
import torch
2

3
# Check if GPU is available
4
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
5

6
model = MyModel().to(device)
7
input_data = torch.randn(16, 3, 224, 224).to(device)  # Example image batch
8

9
# Forward pass
10
output = model(input_data)

In this example, the user explicitly moves the data and model to the GPU device, allowing the parallel architecture of the GPU to speed up matrix operations. MLPerf training benchmarks measure how quickly systems can perform this in a large-scale context, often with distributed data parallel (DDP) strategies.

TPUs in the Cloud#

Google’s TPUs are accessible through Google Cloud. Training on TPUs involves a slightly different API, often inside TensorFlow:

1
import tensorflow as tf
2

3
resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='your-tpu-address')
4
tf.config.experimental_connect_to_cluster(resolver)
5
tf.tpu.experimental.initialize_tpu_system(resolver)
6

7
strategy = tf.distribute.experimental.TPUStrategy(resolver)
8

9
with strategy.scope():
10
    model = create_model()
11
    model.compile(optimizer='adam', loss='categorical_crossentropy')
12

13
# Train
14
model.fit(training_dataset, epochs=10, validation_data=validation_dataset)

By structuring your code within a TPU-friendly distribution strategy, you efficiently leverage the TPU cores. MLPerf results that use TPUs highlight how specialized tensor cores can drastically accelerate certain tasks.

Custom ASIC Clusters#

Companies like Cerebras build wafer-scale engines that can fit entire neural networks on a single chip. These parallelize across tens or hundreds of thousands of processing elements. In MLPerf, you’ll see submissions from such specialized hardware that might show dramatic speed-ups on certain tasks, although cost, availability, and software ecosystem are also considerations.

Downloading the Benchmark#

To experiment with MLPerf benchmarks yourself:

1. Visit the MLPerf repository on GitHub.
2. Choose a benchmark suite (Training, Inference, HPC, Tiny, etc.).
3. Read the instructions for system setup, dataset downloads, and submission rules.

For a simple test, you can start with the MLPerf Inference repository, which might be easier to run on a single machine with GPU. If you have multiple GPUs or a sizable cluster, you might explore the Training suite.

Simplified Example#

Below is a pseudo-code snippet showing how you might run an MLPerf inference benchmark on the ResNet-50 model:

1
# Assuming you have Docker installed
2
git clone https://github.com/mlcommons/inference.git
3
cd inference
4

5
# Build Docker container for the ResNet-50 benchmark
6
make build RESNET50
7

8
# Run the benchmark
9
make run RESNET50

Within the Docker container, scripts handle data loading, model setup, and measurement of latency/throughput. Final results are recorded in standard formats for easy comparison.

Time to Train vs. Throughput#

When looking at training results, the “time to train�?metric is quite intuitive: how many minutes or hours are required to reach a certain accuracy threshold. Throughput might be measured in images (or tokens, sequences, etc.) per second. A higher throughput might correlate with lower time to train, but not always—factors like scaling overhead can keep throughput gains from translating evenly to time savings.

Latency vs. Offline Scenarios#

On the inference side, understanding latency is crucial for real-time applications (e.g., self-driving cars, live language translation). However, for batch processing tasks (e.g., applying your model to millions of images without tight real-time constraints), an offline scenario might be more relevant. MLPerf’s multiple scenarios help you distinguish these nuances.

Vendor Submissions and Real-World Relevance#

A top submission might be a custom-built system with exotic cooling, highly specialized hardware, and an optimized software stack that’s not widely available. While these results push the boundaries of what’s possible, you also want to look for more general-purpose or commercially available reference systems. MLPerf categorizes submissions to help you navigate these distinctions (e.g., “closed�?vs. “open�?categories).

Distributed Training and Scaling#

At the professional level, system architects look at how well performance scales across multiple servers. Linear scaling implies that doubling the number of accelerators nearly halves the training time. However, inter-node communication introduces overhead. MLPerf HPC benchmarks are specifically designed to test large-scale training, revealing which hardware-interconnect strategies are most effective.

Model Parallelism vs. Data Parallelism#

Neural networks like GPT-3 or Megatron-LM contain billions of parameters. Training these massive models can exceed the memory capacity of even high-end GPUs. Professionals often employ model parallelism, splitting layers or parameter tensors across multiple accelerators, in addition to the more common data parallelism. MLPerf HPC and advanced submissions showcase these strategies.

Multi-Precision Training (FP32, FP16, BF16)#

Modern hardware often supports lower-precision arithmetic (e.g., FP16, BF16) to speed up training. Some MLPerf benchmarks allow these optimizations, though you must still meet accuracy requirements. The difference between single-precision and lower-precision results can illustrate how well hardware handles numeric stability.

Quantization and Sparse Representations#

Inference, especially on edge devices, can benefit greatly from quantization (reducing weights to INT8 or even INT4). Some hardware architectures are specifically optimized for sparse matrix multiplication, awarding big performance gains when the model exhibits high sparsity. MLPerf Inference includes tasks that measure these optimizations.

Thermal and Power Considerations#

In data centers, you might find that the limiting factor is not raw speed but thermal design power (TDP). Achieving top performance might require specialized cooling. Similarly, at the edge, power constraints can be even more restrictive. MLPerf’s reported power metrics help highlight these issues.

Enterprise Data Center#

An enterprise might run an MLPerf-like benchmark to select hardware for large-scale training. Even if a wafer-scale solution is faster, the cost and operational complexity might outweigh the benefit compared to more standard GPU servers. MLPerf’s standardized tests allow direct comparisons to see if the speed gain is worth the incremental cost.

Edge Device Deployment#

For a mobile or embedded system, you might focus on MLPerf Tiny or a smaller subset of the MLPerf Inference suite. The emphasis is on power efficiency and memory usage rather than raw throughput. A high-power GPU system that delivers thousands of frames per second might be irrelevant for an IoT scenario.

Mixed Workload Scenarios#

Some organizations run a mix of HPC workloads—classical simulations (e.g., finite element analyses) combined with ML-based post-processing or analytics. MLPerf HPC helps evaluate how well a hardware solution handles this entire pipeline, ensuring that you’re not only accelerating ML tasks but also effectively performing HPC computations.