Most conversations about large-model training end at the GPU: how many were wired together, how many Gbps the interconnect runs at, what the MFU is. A paper that dug into what those GPUs are actually idling on — using production data — won the best paper award at OSDI 2026, the 20th USENIX Symposium on Operating Systems Design and Implementation, held in Seattle in July 2026.
The title is “Teaching The Old Dog New Tricks: Building Efficient Data Pipelines for Large-Scale LLM Pre-training.” The old dog is HDFS. The paper’s argument is about how far you can go with a distributed file system released in 2006, holding up exabyte-scale LLM training, without tearing the architecture down and starting over.
The authors are centered at the University of Science and Technology of China (USTC) and the ByteDance Seed team, with Tsinghua University and the Institute of Artificial Intelligence at the Hefei Comprehensive National Science Center participating. The analysis covers 30,000 training job traces collected over 90 days on ByteDance’s actual pre-training clusters. What makes the paper notable is not a flashy technique but the fact that every number in it is a measurement from a running production environment, not a benchmark.
The setup — pre-training happens in three phases
To understand the bottlenecks you first have to see how a training job touches storage. The paper splits pre-training execution into three phases.
- Initialization: parameters and optimizer states are read from the latest checkpoint and distributed across thousands of ranks. Nothing starts until everyone is done — a synchronous barrier.
- Iterative training: the dataloader prefetches and transforms data for upcoming steps, while the framework periodically persists checkpoints and, when enabled, writes logits every step.
- Companion evaluation: a separate pipeline that runs in parallel with training. It pulls a recent checkpoint, runs benchmark suites, and checks from the outside whether the model is falling apart.
The third one may be unfamiliar, and it is exactly where the paper’s first bottleneck lives. Operational experience says that watching the training loss alone does not catch a model going bad. The loss curve can look perfectly healthy while forgetting and collapse are already underway on downstream tasks. So benchmarks run continuously on a separate cluster.
Bottleneck 1. The cross-DC latency trap — round trips, not bandwidth
Companion evaluation clusters usually sit in a different datacenter. There are two reasons. Pre-training moves thousands of GPUs in lockstep under gang scheduling, so squeezing evaluation jobs into the same cluster fragments scheduling and cuts training throughput. And evaluation is throughput-oriented work that does not need the newest accelerators, so it goes wherever power and space are available. A sensible placement. The problem is the I/O that decision creates.
Over a 30-day window, 19 pre-training jobs ran 3,589 companion evaluations and caught 156 critical model regressions. Evaluation clearly earns its keep. But for a representative large multimodal model (MM-L), one evaluation cycle exceeds four hours, and about 56.6% of it is data I/O. Of that I/O time, 84.8% sits in the checkpoint merging stage, because the evaluation cluster has to pull roughly 2.6 TB across the wide area network.
Here is the interesting part. Bandwidth was not the problem.
- A transformer checkpoint is thousands of disjoint tensors, and in the MM-L checkpoint more than 60% of the tensors are smaller than 16 KB (LayerNorm parameters, biases, and the like).
- Merging reads only the tensors each modality submodel needs, so requests go out at tensor granularity. That is the worst possible match for an HDFS tuned for 128 MB blocks, and it produces about 1.5× read amplification.
- Add a WAN with a 100 ms round-trip time. Throughput is now bound by latency, not bandwidth. That is why the paper’s graph shows a 60 Gbps link idling in a sawtooth pattern.
- On top of that, cross-DC bandwidth is shared with 208 concurrent tasks on average (data migration, log aggregation, other evaluations). First-come-first-served cannot tell an urgent evaluation apart from an overnight archival job.
The cost works out like this. Across those 156 regressions, I/O-induced delay wasted roughly 2.6 million GPU hours. The evaluation system saved about 5.5 million GPU hours through early detection — meaning the I/O of evaluation was giving back nearly half of what evaluation earned.
There are two fixes, and neither of them buys new hardware.
Predictive checkpoint replication. Companion evaluation typically runs every 1,000 steps — a predictable schedule. So replication can start the moment the training side begins writing a checkpoint. Storage batches the required shards into large contiguous transfers, caches them in the evaluation cluster’s storage tier, and a lightweight namespace service called NNProxy steers evaluation jobs to the local copies. Once wide-area reads become local reads, both the RTT and the small-read fragmentation disappear.
Signal-driven prioritization. Not every evaluation is urgent. Unlike scheduled runs, evaluations triggered by a loss spike or abnormal gradients need an answer fast. A priority signal combining model scale, job importance, and anomaly severity lets the storage scheduler preempt background traffic and hand the bandwidth over.
The result is a 76.1% average reduction in merge latency (89.3% for the small text model T-S, and still 70.8% for MM-L). The compute wasted per regression fell from 16,800 GPU hours to 4,000, and across the observation window the optimizations recovered nearly 2 million GPU hours.
Bottleneck 2. The initialization I/O storm — a paradox created by deduplication
Large-scale training is not a picture of months of uninterrupted execution. Hardware failures, automated maintenance and, above all, debugging cycles during model development stop it and restart it constantly. And every restart has thousands of ranks reading the checkpoint at the same instant.
Here is the startup latency breakdown for one restart in the MM-S trace.
| Phase | Operation | Time (s) |
|---|---|---|
| Job initialization | Cluster startup | 23 |
| Job initialization | Distributed context init | 130 |
| Model construction | Model construction | 118 |
| Checkpoint recovery | Coordination | 76 |
| Checkpoint recovery | Downloading from HDFS | 300 |
| Checkpoint recovery | Parameter loading | 104 |
| Total | 751 |
One blocking download is 39.95% of the entire startup. Multiply that by how clustered restarts are: the paper describes a single 111-minute debugging window in which one job restarted four times in a row. At five minutes of download per run, 18.02% of that window’s wall-clock time is I/O. Extrapolated across the cluster, startup delay burns more than a million GPU hours a year.
A controlled experiment at 2,048 GPUs went looking for the cause, and it broke a common assumption. The bottleneck in a distributed file system is supposed to be metadata operations (open, getattr) — but measured open latency was low and stable. What spiked was read latency, and it lined up exactly with QPS surges. In other words, this is data contention, not metadata. The download latency distribution had a long tail, and a few straggler reads accounted for 67.97% of the total wait. Measuring peak per-file QPS in 0.5-second windows showed that the top 5% of files generated 38.8% of the total pressure.
Those hotspots come from two places. One is global metadata that every rank reads (.metadata, common_states). The other is more interesting: an access skew created by deduplication.
Checkpoint systems use parallel saving and redundancy elimination for write throughput and storage efficiency. A parameter that is replicated across ranks, such as an embedding table, is persisted only once. That is an excellent decision on the write path. But on the read path hundreds of ranks converge on that single file. The default 3-replica policy cannot absorb it and latency goes vertical. By contrast, sharded parameters such as MoE experts live in separate files per parallel group, so the load spreads itself out. Write optimization created read asymmetry — the single most reusable observation in this paper.
The traditional answer is reactive replication: detect heat, then add replicas. It fails twice here. The contention window lasts tens of seconds, so by the time the hotspot is detected and copies exist, loading is over; and injecting replication traffic into the middle of the storm pushes on an already saturated network and makes things worse.
So the paper inverts the direction. Tell the storage layer about the hotspots in advance. The parallelism strategy and world size are fixed at job submission, which means which files will get hot is deterministically computable. The framework passes the list of global metadata and replicated tensors through a SetReplicationHints interface; storage divides each file’s expected concurrency by the safe load a single replica can carry, sets a target replica count, and expands ahead of time. To keep storage from bloating, those ephemeral replicas carry a TTL and are reclaimed once the startup phase ends.
In the 2,048-GPU experiment, pre-expanding hot files to 128 replicas cut checkpoint loading from 38.48 seconds to 22.78 — a 40.8% improvement. In absolute terms that is a dozen-odd seconds, but the meaning is in the scaling. Contention worsens as ranks multiply, and pre-expansion turns a bottleneck that degraded unpredictably with scale into something close to constant time. The paper goes as far as codifying it as an operational guideline — 64 replicas for 10k-GPU clusters, 128 for 20k-GPU clusters.
Bottleneck 3. The transformation wall — CPU, not storage, is the wall
In text-only training, data loading hides behind GPU compute. Reading and tokenizing takes far less time than a step. Move to multimodal and that assumption collapses.
Here is how the 5.35 seconds of data loading in the MM-L trace breaks down.
| Stage | Average (s) | Max (s) |
|---|---|---|
| Metadata read | 0.007 | 0.506 |
| Data read | 0.013 | 0.266 |
| Transformation (decode, crop, …) | 5.05 | 41.52 |
| Other overhead | 0.279 | 0.428 |
| Total | 5.35 | 42.72 |
Transformation is 94.4% of it. The time spent actually reading bytes from storage averages 13.6 ms — negligible. This workload is not I/O-bound; it is compute-bound.
The variance is worse than the average. A multimodal batch mixes short text with ten-minute high-resolution video, and transformation cost scales with duration, codec, and sampled frame count. While one host spends 41.5 seconds on a single 161.9 MB sample, the others finish an average of 20 MB in 5.05 seconds. A 500 MB, ten-minute H.264 clip takes 2.18 minutes even with hardware-accelerated decoding and four threads. In synchronous training, one straggler like that stalls the other thousands of GPUs. Step time diverged across hosts by as much as 5 seconds, and that skew alone evaporates more than 10,000 GPU hours per day.
The obvious fix is to transform everything up front, and the paper cuts it down for two reasons. First, storage amplification. Unpacking JPEG and H.265 into float tensors inflates video by more than 40× and images by tens of times. Turning a petabyte dataset into an exabyte one just moves the bottleneck from CPU to capacity. Second, inflexibility. Crop size, frame count and resolution are hyperparameters that change from run to run. If every change means regenerating the entire pre-transformed dataset, model development velocity dies.
Instead the paper finds a resource that has already been paid for. Storage node CPUs were sitting at 20–30% utilization. While training nodes saturate on compute and media processing, storage nodes are pinned by disk seeks and NIC limits, leaving cores idle.
So the storage tier is redesigned as a disaggregated pre-processing engine. What makes this possible is that their dataloader is deterministic: an offline global execution plan already fixes which sample is consumed at which step, so storage knows the future access order exactly. Shuffling permutes index entries rather than moving physical data.
- Schedule synchronization: at startup the framework passes only the dataset identifier and step progress; storage nodes read the per-step info files and work out the upcoming blocks themselves. Millions of RPCs leave the critical path.
- JIT transformation: storage nodes keep a consumer queue, speculatively read raw bytes, and run decoding, cropping and normalization in the background. While the GPU computes step N, storage is preparing tensors for step N+1. For video, frame sampling also happens on the storage side so that the payload inflated by decoding never hits the network.
- Load-aware fallback: if a storage node’s CPU passes 80%, it aborts the transformation and returns the raw bytes instead. The training client detects whether it received a tensor or a binary and falls back to local processing. This is the guard that keeps shared storage from being taken down.
The results: P99 data loading latency down 85.7%, stall time from transformation stragglers down 63.2%, a 10.8% relative MFU improvement, and a 94% reduction in data-loading CPU usage on training hosts.
The three bottlenecks on one page
| Bottleneck | Where | Real cause | Fix | Effect |
|---|---|---|---|---|
| Cross-DC latency | Checkpoint merge in companion evaluation | Thousands of small tensors × WAN RTT | Predictive replication + priority signal | Merge latency −76.1% |
| Initialization I/O storm | Checkpoint loading on restart | Read skew created by deduplication | Proactive hot-file replication (TTL) | Loading −40.8% |
| Transformation wall | Multimodal data loading | Host-CPU decoding stragglers | JIT offload to storage nodes | Stalls −63.2% |
The roads not taken are the interesting part
The paper has a section called “What If?” listing the alternatives they considered and rejected, with reasons. For practitioners this may be the real body of the work.
Why not distribute checkpoints peer-to-peer? In theory it scales linearly with cluster size; they rejected it anyway. A large GPU cluster runs several training jobs at once, and P2P’s all-to-all traffic interferes with the NCCL collectives of the other jobs. And initialization is a synchronous barrier bound by the slowest node, while P2P performance is probabilistic — one throttled peer delays the whole gang’s start. A hierarchical client-server design instead gives a deterministic SLA.
Why not stand up a dedicated transformation cluster? That is the structure tf.data service and Ray Data chose. Rejected too. Decoded tensors are 50–100× larger than their compressed sources, and shipping them over the datacenter network merely moves the bottleneck from CPU to bandwidth. A standalone fleet also has to be sized for the multimodal peak and then sits idle through text-only runs.
Why not migrate to AI-native storage like 3FS or AIStore? The answer here ties straight back to the paper’s title. HDFS is not a warehouse used only by AI training; it is an exabyte-scale data lake shared across many business lines. Moving that much data is operationally impossible before it is expensive, and AI-native systems tend to require specific hardware (all-NVMe) or custom clients that break compatibility with the surrounding ecosystem, such as Spark-based cleaning pipelines. Data gravity is the heaviest force not only in vendor choice but in file system choice.
What is left for the rest of us
There are three takeaways even if you are not running tens of thousands of GPUs.
First, tell storage what is deterministic. All three fixes are variations on one sentence: the training job already knows what it will read and when, and storage does not, so it only reacts. Passing what you know in advance through an interface — evaluation interval, world size, sample order — unlocks exactly the cases a reactive cache can never catch. That design principle applies equally to an in-house file server and a managed service.
Second, suspect that write optimization is creating your read bottleneck. Deduplication, compression and parallel saving are all virtues at write time. But recovery creates an access pattern that is the mirror image of writing. If you design a backup, checkpoint or artifact store without drawing the fan-out at recovery time, you will step into the same trap.
Third, look at whether compute can move toward the data. This one is conditional. As the paper itself notes, storage-side offload rests on the assumption that storage nodes have spare CPU, which does not hold for object stores like S3 or Azure Blob, or for lean storage appliances. In that case the alternatives the paper offers are caching transformed samples (though with global shuffling in pre-training the hit rate is poor, so this suits reinforcement learning or repeated fine-tuning better) and predicting transformation cost from sample metadata to compose batches that mix heavy and light samples.
Read the numbers with care
The numbers are strong, but they come from one company’s one environment. Storage is HDFS, the dataloader is deterministic, storage nodes have spare CPU, and evaluation runs in a different datacenter. Change any one condition and the improvement changes with it. The 40.8% figure comes from a controlled 2,048-GPU experiment that cut 38.48 seconds to 22.78, and the point of it is stability under scaling rather than the absolute value.
Even so, it is clear enough why this won best paper. Without building new storage and without migrating anything, a few interfaces that expose application determinism to a twenty-year-old system removed production-scale bottlenecks. For anyone who runs infrastructure, conclusions rarely come more practical than that. The authors plan to release the traces after anonymization; when they do, they will serve as a baseline not only for follow-up research but for any team that wants to measure its own cluster.
⚠️ The figures in this article are values reported in “Teaching The Old Dog New Tricks: Building Efficient Data Pipelines for Large-Scale LLM Pre-training” (OSDI 2026, USENIX). They were measured in one specific ByteDance production environment (HDFS-based, deterministic dataloader) and do not transfer unchanged to other environments. The best paper award follows the designation on the official USENIX page, and part of the description of its significance is based on announcements from a participating institution.
Frequently asked questions
What kind of conference is OSDI?
It is a top-tier venue for operating systems and computer systems, organized by USENIX. 2026 was the 20th edition, held July 13–15 in Seattle, USA. Together with SOSP it is considered the most heavily cited venue in systems software, with a low acceptance rate and a strong bias toward papers about real systems at scale. Accepted papers are published openly on the USENIX site.
What exactly is companion evaluation?
It is a continuous validation pipeline that pulls a recent checkpoint and runs benchmarks on a separate cluster without pausing training. Watching the training loss alone can miss a model that is already degrading on particular tasks, so this provides an external measurement. In the environment described, 3,589 evaluations over 30 days caught 156 critical regressions, and a bad result can make the training control plane trigger a rollback.
Why is slow checkpoint loading not a metadata problem?
Conventional wisdom says metadata operations such as open and getattr are the bottleneck in distributed file systems, but when the authors measured at 2,048 GPUs, open latency was low and stable. What spiked was read latency, correlating exactly with QPS surges. The cause was data contention — hundreds of ranks converging on a few popular files — and the embedding file stored only once because of deduplication was the archetypal hotspot.
Is transforming data on storage nodes safe?
The paper adds two safeguards. One is load-aware fallback: if a storage node’s CPU exceeds a threshold (for example 80%), it stops transforming and returns raw bytes. The training-side dataloader inspects the format it received and falls back to local processing on its own. The other is finishing frame sampling on the storage side for video so that the data leaving over the network does not balloon. Not breaking the stability of shared storage was a hard precondition.
We use S3 — can we apply the same techniques?
The first two, predictive pre-replication or caching and smoothing the access skew at startup, can be carried over conceptually. Storage-side transformation offload cannot, because it depends on storage nodes having spare CPU, which object stores do not expose. The paper states this limitation and offers caching of transformed samples and cost-aware batch composition as alternatives. The latter is something a dataloader team can try immediately with nothing more than sample size, codec and duration metadata.
Leave a Reply