The Code Was Fine and the Result Was Wrong: Five Papers That Changed Troubleshooting in 2026

Anyone who has handled an outage knows the worst case is not a server dying. It is a server that stays up and keeps producing wrong values. The logs are clean, no alert fires, and no amount of code review turns up a bad line. Yet the results are wrong.

OSDI 2026, held in Seattle in July 2026, carried an unusual number of papers attacking exactly this problem. And many of them came from the conference’s separately labeled “Operational Systems” track — papers written by the engineers who actually run these systems, bringing their own incident history with them. ByteDance, Microsoft Azure, Huawei and Alibaba each opened up their production floor.

This article picks five of them to trace how troubleshooting practice is changing. What matters more than any individual technique is the way of thinking they share. In short: infer less, compare more, and say “I don’t know” when you don’t.

Scene 1. “The code was fine” — when a GPU quietly returns the wrong number

The first is “SDCs in the Wild” from Shanghai Jiao Tong University and ByteDance Seed, an autopsy of 23 SDC-defective GPUs harvested from a production cluster. SDC — Silent Data Corruption — is hardware producing an incorrect result without raising any error signal.

The most painful sentence in the paper is not a technique but a description of the job. What torments engineers is not the frequency of these errors but their ambiguity. A GPU SDC shows up as an abrupt crash or a loss spike, and that looks exactly like a software bug or numerical instability. So engineers instinctively suspect the code first and spend days or weeks bisecting the pipeline. The conclusion “it was hardware” usually arrives only after every other hypothesis has been exhausted.

Two real cases from the paper illustrate the trap.

• A shape-mismatch crash during MoE training. An SDC hit the token-count aggregation and silently corrupted a buffer size; the exception surfaced much later at a different line. The code at the crash site was correct. A textbook case of the error message not pointing at the real cause.
• Ten reruns from the same checkpoint. Run 1 spiked. Run 2 reproduced it. But runs 3 and 4 were clean. Runs 5 and 6 spiked again — at completely different steps this time. A code bug fails at a consistent logical point. The fact that the failure point moved was itself the clue pointing at hardware.

The scale numbers are striking too. Meta reported hardware failures roughly every 2.78 hours on a 16,000-GPU training run, with about 1.4% of them attributed to GPU SDC (the Llama 3 405B run logged 419 unexpected interruptions over 54 days on 16,384 GPUs, more than half from GPUs and HBM3 memory). Google observes an SDC every one to two weeks in its TPU clusters, and ByteDance recorded 6,096 implicit errors over three months.

And the industry-standard response — running synthetic stress tests across every GPU — missed more than 60% of the defective devices. The characterization explains why.

• They appear across the whole hardware lifecycle. Only 25% were caught during burn-in, while 40% surfaced around a year after deployment. This is cumulative degradation, not a manufacturing defect, and once it starts the error rate climbs.
• They depend on data and on specific units. A device passes general tests but fails on particular kernels and particular input ranges. Notably, FP32 and FP64 CUDA cores failed more often than Tensor Cores — the explanation being that high-precision units are physically larger and more complex, so a defect is more likely to manifest.
• ECC and thermal sensors do not catch them. These are logic-level bit errors in computation, not memory errors.

So the paper’s diagnosis system, SDCHunter, throws out synthetic benchmarks and replays the exact workload and the exact data that triggered the failure. Some defective GPUs reproduce the error 100% of the time; others do so about one time in a million, which is precisely why generic tests can never catch them.

It works in two phases. First it splits the cluster along the data-parallel axis into two replicas, feeds them the identical batch, and compares tensor hashes only at pipeline-parallel boundaries (3% overhead). The stage where the two signatures diverge is found within hours, that replica is removed, and training resumes. Then it deterministically replays the problematic iteration on the suspect group and a healthy reference group, comparing signatures for every intermediate tensor to find the first tensor and kernel that diverge, and maps that back to the defective device.

The key move is decoupling recovery from debugging. Training resumes within an hour, and the defective GPU is confirmed offline separately, also within an hour. Since deployment the system has handled 40 SDC incidents, and diagnosis time dropped from days to under an hour.

All of this rests on deterministic execution: locked RNG seeds, deterministic kernels, standardized communication order. The common belief is that determinism costs performance, but in this paper’s production measurements the step-time difference was within 0.01% while normalized debugging time fell by 70%.

Scene 2. Aligning two runs bit for bit

OpGuard, from the University of Michigan and ByteDance Seed, attacks the same problem from another angle. One incident in it compresses the difficulty of this field into a single story.

A vision-language model training run on 1,000 GPUs raised a gradient-norm alert after more than 3,000 steps. Experienced developers spent five days repeating experiments, toggling flags and swapping kernels, with no progress. The real cause was a tiny race condition in an embedding backward kernel. Under rare token patterns a few rows were perturbed, the next forward pass read the corrupted weights, and the error spread through the tensor-parallel group. By the time the loss visibly diverged, most GPUs were already holding bad tensors.

The problem was not a lack of tools. The developers were comparing a good run against the buggy one. But what they compared were aggregate signals like loss and gradient norm — values that fold millions of operations into one number, diluting the error and saying nothing about where the divergence began.

OpGuard’s proposal is simple: compare bit for bit, and much earlier. Treat both runs as sequences of model-level computation boundaries and compute the longest prefix over which corresponding tensors are bitwise identical. The first boundary that fails is the earliest point where the runs diverged, and that becomes the pivot for debugging. In the case above, the loss difference was exactly zero through step 3080 and became -1.0014e-05 at step 3081 — noise to the human eye, an unambiguous boundary to a bitwise comparison.

The hard part is stripping out benign nondeterminism. The system finds semantically stable points across differing schedules and kernel implementations, fingerprints them, and keeps harmless differences from being mistaken for errors — so that the first mismatch really is evidence of a fault. Deployed across ByteDance’s pre-training and post-training workloads, it diagnosed more than twenty production issues, including kernel races and silent corruptions that existing checks had missed, cutting debugging time from days to minutes.

Scene 3. Sense cheaply, confirm in the pipeline bubbles

AEGIS, from Tsinghua University and ByteDance, takes on the same SDC problem as continuous online detection rather than post-hoc diagnosis.

None of the three existing options was satisfactory. Offline diagnostic suites stop training and validate only a limited set of workloads. Rerun-based confirmation is definitive but doubles the computation, which is unaffordable at this scale. Algorithm-based online detection (checksums) is attractive in principle, but in low-precision arithmetic a checksum mismatch drowns in ordinary floating-point error.

AEGIS splits detection into two stages, the cSensor–cVerifier design. cSensor runs inline with training, senses “something looks suspicious” very cheaply, and captures the minimum context needed for later confirmation — before the framework overwrites that state. cVerifier does the definitive confirmation, and it schedules verification into naturally idle periods such as pipeline bubbles. In other words, sensing sits on the critical path and confirmation sits off it.

The low-precision problem was sidestepped through hardware characteristics. Because tensor units in modern GPUs accumulate at high precision, a mixed-precision checksum can separate real SDCs from ordinary numerical error.

In a production deployment spanning 35 million GPU hours, it found 18 SDC incidents and 13 faulty GPUs with a performance overhead of 0.86%. The practical message is that always-on detection can be bought for less than 1%.

Scene 4. The profiler was lying

The fourth has a different flavor: “When Sampling Lies,” from Huawei’s compiler team with the University of Toronto and YScope. The target is a smartphone OS rather than a cloud, but the lesson applies to anyone who uses a profiler.

It starts like this. Engineers added an optimization using ARMv8.1’s LSE instructions to reduce instruction count, then ran perf as a sanity check. perf reported that the instruction count had increased by 6%, about 90 million. The exact percentage moved run to run, but the direction was consistent. Trusting it, the engineers spent weeks hunting for the cause.

The culprit was not the optimization but the measuring instrument: the combination of skid and the shadow effect. Modern processors cannot deliver a PMU interrupt precisely at the instruction that overflowed the counter, so samples land on later instructions (skid). By itself that is random noise. But when a long-latency instruction blocks the reorder buffer, the skid window narrows and samples are biased toward that instruction. This optimization had replaced several short instructions with one long-latency atomic. So perf produced results that were consistent and completely wrong.

The paper’s own words are memorable. The team learned through painful experience that perf’s output could not be trusted, and says they spent months chasing non-issues. It also names a scarier scenario: an optimization that actually degrades performance could have been shipped to production on the strength of a bad measurement.

Nor was that the only problem. On a flat profile — thousands of short-lived functions with no dominant bottleneck — perf covered only 62% of functions even at its highest sampling rate. And high-frequency sampling perturbed the system enough to increase L1 data cache misses by up to 46×.

Their alternative, Blink, drops interrupts and instead reads PMU counters directly at instrumentation points such as function entry and exit, eliminating skid and shadow effects by construction. Self-patching binary rewriting keeps the disabled path down to a single jump instruction. The result is 99.999% accuracy on instruction counts and under 1% increase in user-visible frame drops. Most importantly, measurements that needed more than 50 repeated runs under perf to stabilize now take one or two.

The lesson to carry away is not a tool name. It is that every observability tool has an error model, and it can be systematically wrong where that model meets your workload. Random noise washes out with repetition; systematic bias does not.

Scene 5. The right to say “I don’t know” — RCA that abstains

The fifth is the one I find most interesting: “The Abstention Protocol,” on Azure’s network RCA system.

A large cloud network is always broken somewhere. In a cluster with thousands of switches there is always a cable throwing CRC errors, a link flapping, a top-of-rack switch mid-upgrade. Clos topologies have enough path diversity to absorb this and keep forwarding. The trouble comes when an incident is raised: dozens of entities show faults, and most of them are background noise unrelated to this incident.

One sentence lands hard for anyone who has carried a pager. A fresh on-call engineer, looking only at counters and probes, cannot tell whether a given signal belongs to the incident under investigation or to one of the dozens of unrelated faults always quietly happening somewhere in the fabric.

Most of these failures are not fail-stop but gray failures: a loose optical module dropping 2% of packets in one direction of one link, a linecard bit-flip corrupting headers only for flows that hash to a particular ECMP path, a firmware bug rebooting a switch and recovering before health monitors notice.

The previous system scored each signal and blamed the entity with the highest weighted sum. It failed in predictable ways. Background faults always contributed some score, so a culprit was named even when the network was innocent, and tuning weights down for one failure mode raised false positives for another. The false positive rate hovered uncontrollably between 18 and 22%.

The insight behind the new system, CoreSec, is to treat this as a composition problem rather than a scoring problem. No telemetry source is reliable across all failure modes: active probes catch link failures fast but miss software defects, device counters catch hardware degradation but are noisy, traffic-derived signals reflect customer impact but cover sparsely.

So CoreSec borrows the structure of Linux’s Pluggable Authentication Modules (PAM). Just as PAM composes independent checks — password, biometrics, hardware token — by tagging each required, sufficient or optional, CoreSec tags each telemetry agent requisite / required / sufficient / optional. And crucially, it admits a third outcome alongside healthy and unhealthy.

When evidence is missing or conflicting, the system abstains.

The effect of that one design decision shows up in the numbers. Over three years and more than 700,000 incidents at Azure, the false positive rate fell from 18–22% to under 1%. The team did not treat abstention as free — they counted it as an explicit false negative, and that rate too fell from 10% initially to 1.5% over the most recent six months. Manual RCA work worth three full-time engineers disappeared.

Automated diagnosis usually loses trust not by failing to answer but by answering wrongly with confidence. Once the unknown cases can be marked unknown, the remaining answers become more trustworthy — and only then do people actually use them.

Bonus. Shallow by default, deep only when something is wrong

StriaTrace, from Alibaba and Shanghai Jiao Tong University, addresses tracing for LLM inference services. Unlike training, inference is latency-sensitive, so a single sporadic anomaly is an SLO violation. Yet existing profilers cost 10–20% overhead, too much to leave on.

Three principles, stated crisply: trace only synchronization points (cutting CPU instrumentation sites from around a thousand functions to about ten), trace only the critical path, and trace in detail only during anomalies. The result was a 97.8% reduction in tracing overhead while diagnosing hundreds of anomalies across 19 distinct root causes.

The point is to spend the observability budget adaptively rather than evenly. Shallow in normal times, deep when something looks off.

Three principles running through all five

The techniques differ, but the underlying principles converge on three.

First, comparison instead of inference. Three of the five share the same skeleton: put two runs side by side and find where they diverge. That requires execution to be reproducible, which is why determinism is no longer a research luxury but a precondition of debugging infrastructure.

Second, catch signals early and raw. Loss curves and average latencies have already flattened millions of operations into one number, diluting the error and destroying its location. Bitwise fingerprints, tensor signatures and synchronization points are the signal before it gets flattened.

Third, model the limits of the tool. Profilers go wrong through skid; score-based RCA goes wrong through background noise. Both papers, rather than making the tool more sophisticated, removed the failure mode from the design — one by eliminating interrupts, the other by making “I don’t know” a legitimate output.

What you can do in your own organization

Some of this transfers even without tens of thousands of GPUs.

1. Build the reproducibility switch in advance. Fixed seeds, deterministic kernel and library paths, a pinned communication and processing order — arranged so you can turn them on at will. Building it after an incident is already too late. Check the performance cost by measuring; in the paper above the step-time difference was within 0.01%.
2. Keep a reference run available. Simply being able to run “the last version that was fine” on the same input gets you halfway to comparison-based debugging.
3. Raise the resolution of your comparisons. Do not compare averages and totals; leave values that do not flatten — checksums, hashes, fingerprints — at each boundary. The goal is to find the first point of divergence.
4. Document your tools’ error modes. One wiki page on when your profiler or APM is systematically wrong will save weeks the next time someone sees a strange number.
5. Give automated diagnosis an “I don’t know.” In alerting, RCA, anomaly detection — anywhere — stop it from naming a culprit on thin evidence. And track the abstention rate as a metric, which is exactly why the Azure team counted abstentions as false negatives.
6. Use observability adaptively. Always-on tracing shallow; on an anomaly signal, deep on that window only.

Read with care

These papers mostly describe very large environments. Trade-offs that pay off across thousands of GPUs, tens of thousands of switches and hundreds of millions of requests do not automatically hold at smaller scale. The cost of deterministic execution varies by workload, and the 0.01% figure belongs to that environment.

The headline results are also self-evaluations of a company’s own system in its own environment. What generalizes is the structure of the failure, not the absolute numbers — why synthetic tests miss defects, why aggregate metrics dilute errors, why score-based verdicts collapse under background noise. Those structures recur regardless of scale.

One last thing. None of the five claims that automation replaces people. SDCHunter separated training recovery from hardware confirmation to shorten the time humans spend waiting; CoreSec invented abstention precisely to hand ambiguous cases back to humans. The goal of a good troubleshooting tool is less to produce the answer than to let people work on the certain parts first.

⚠️ Every paper discussed here was presented at OSDI 2026 (the 20th USENIX Symposium on Operating Systems Design and Implementation, Seattle, July 2026), and the figures are those the papers report. Most are self-evaluations of a company’s own system in its own production environment and do not transfer unchanged elsewhere. The Meta and Google failure statistics are re-cited from public reports referenced by those papers.

Frequently asked questions

What exactly is Silent Data Corruption (SDC)?

It is hardware producing an incorrect computation result without raising any error signal. Unlike a crash, or an error the hardware detects and reports (a DUE), there is no warning at all, so it surfaces late as a loss spike or an out-of-place exception. Causes include environmental factors such as cosmic rays, design and manufacturing defects, and circuit degradation over time. According to the OSDI 2026 paper, GPU SDCs appeared more often around a year after deployment than in early life.

Doesn’t turning on deterministic execution slow things down a lot?

Contrary to the common belief, in that paper’s production measurements the step-time difference was within 0.01% while normalized debugging time fell by 70%. That said, this is a figure from a specific environment — large-scale LLM training with a fixed parallelization setup. The cost of determinism depends on which sources of nondeterminism you remove, so measure step time on your own workload before adopting it. The important part is being ready to switch it on when you need it.

Does this mean we should stop using sampling profilers like perf?

No. For workloads with a clear bottleneck they remain excellent tools. The trouble arises on flat profiles where thousands of short functions share the time evenly, especially when long-latency instructions are involved and skid combines with the shadow effect to bias results systematically. Repeated runs do not erase that kind of error. If what you are measuring is a 1–2% improvement, cross-check with an instrumentation-based tool.

Isn’t an RCA that abstains just an RCA that gives no answer?

Close to the opposite. After abstention was introduced at Azure, the false positive rate fell from 18–22% to under 1%, and as the answers it did give became trustworthy, teams actually started using them. The key is not making abstention free: that team counted abstentions as explicit false negatives and tracked them, driving the rate from 10% down to 1.5%. This design stays healthy only when “I don’t know” is part of the scorecard.

Is any of this applicable to a small service?

Three things, in order of lowest cost. First, keep the last known-good version reproducible on the same input. Second, instead of averages and totals, leave values that do not flatten — a checksum or hash — at each boundary. Third, stop your alerting and automated diagnosis from naming a culprit when the evidence is thin. None of the three needs large infrastructure, and the time they save during an incident does not depend on scale.