NVIDIA Attribution: Root Cause for Distributed AI Failures

Distributed AI infrastructure · ax-trend issue #3781 · Released November 2, 2026

NVIDIA Attribution title card

NVIDIA open sourced Attribution on November 2, 2026, the system it uses internally to find the root cause of distributed AI workload failures. Attribution gathers logs, metrics, telemetry, and scheduler state for a single workload and returns one structured diagnosis: an outcome, a root cause, the affected nodes, the symptoms that support the call, and the evidence records behind it.

The notable design choice is what the system does not use. There is no LLM in the judgment step. Attribution applies deterministic rules to the evidence, so the same input always produces the same diagnosis, and an operator can read, edit, and redeploy the rules that produced a verdict. NVIDIA says it has run the system on its own AI infrastructure for years, analyzing millions of workloads per week across hundreds of thousands of GPUs.

Workloads analyzed internallyMillions per week
GPU fleet covered internallyHundreds of thousands
Diagnosis methodDeterministic rules, no LLM
License and sample dataApache 2.0, samples included

Release Overview

The repository, dsx-ai-factory/attribution, ships under the Apache 2.0 license and includes sample workloads, so a team can run diagnoses against sample evidence before connecting a real cluster. The release splits into two components. The Attribution Service serves diagnoses through a REST API. The Attribution Blueprint is a Python orchestrator that collects evidence from schedulers and telemetry sources, calls the Service, and writes the result out.

The problem Attribution addresses is structural to distributed training. When one node fails, every rank in the job tends to log an error, and the exception in the log marks where the workload stopped rather than where the failure started. Root cause analysis has usually meant an engineer lining up logs from many nodes by hand. Attribution is NVIDIA publishing that comparison work as a fixed set of rules, positioned alongside readiness checks, continuous health monitoring, and recovery systems: those layers watch and repair, while Attribution owns the question of why a workload failed and what evidence supports the answer.

Attribution worked example showing NCCL errors on three nodes and an XID 149 fault on node-02
The worked example from the NVIDIA post: NCCL errors surface on all three nodes, while the node-02 syslog holds the XID 149 entry that the extraction step turns into the originating event. (Photo: NVIDIA developer blog)

How Attribution Works

How Attribution diagnoses a failure in four steps
Evidence is collected per workload, deterministic rules judge it, and the diagnosis separates the root cause from the symptoms it caused.

The Blueprint ships with connectors on both sides of the pipeline. For execution context it connects to Slurm, Kueue, and static input. For logs and metrics it connects to Loki, local files, Kubernetes pod logs, and PromQL metrics. Results go to stdout and SQLite. That connector list matters for adoption: most training clusters already run one of these schedulers and one of these log stacks, so the evidence collection layer is configuration work rather than a new instrumentation project.

Inside a diagnosis, the separation between cause and symptom is explicit. Symptoms are the errors the failure spread to other ranks. The root cause is the single originating fault the rules identify, and the evidence records let an operator audit how the system got there. For postmortem automation, that audit trail is the valuable part: a diagnosis that cannot show its evidence is just another alert, while a diagnosis with records attached can feed runbooks, ticketing, and node quarantine decisions without a human re-deriving it first.

The XID 149 Worked Example

The blog walks through one failure in detail. A single GPU raised an XID 149 hard fault, and the error then spread to multiple ranks as NCCL errors. Read naively, the logs point everywhere at once, because every rank reports a communication failure. Attribution decoded the XID 149 entry as an NVLink signal integrity problem and named node-02 as the root cause. The NCCL errors stayed in the diagnosis, but in their proper place: as symptoms, not as the cause.

This example shows why a rule system can beat a general model at this specific job. The mapping from an XID code to a hardware fault class is a lookup, not a reasoning problem. Encoding it as a rule makes the answer repeatable and lets the rule be corrected once, in one place, when it is wrong. The same logic applies across a fleet: a failure pattern identified on one cluster becomes a rule that every later workload is checked against automatically.

Limits Stated in the Release

NVIDIA published no accuracy benchmark for Attribution. The internal scale figures describe how much the system runs, not how often its root cause call is right.

These limits frame the honest expectation for outside adopters. NVIDIA's internal results were earned on NVIDIA's own telemetry discipline, and a cluster with sparse logging will get sparse diagnoses. The sample workloads in the repository are the right first step: they show what evidence a good diagnosis needs, which is also a checklist for what a cluster should be recording before it expects the tool to help.

Practical Significance for Cluster Operators

For a team running training clusters, failure time is GPU idle time. A tool that names the starting node instead of the loudest error shortens the path from a dead job to a quarantined node and a restarted run. Because the judgment is rules, the improvement loop is concrete: every failure the system misses or mislabels becomes a candidate rule, reviewed once and then applied to every future workload.

The release is also a useful counterweight in a market rushing to put a model in every operations loop. Diagnosis over well structured evidence, with a bounded set of known fault classes, is exactly the kind of problem where deterministic rules are cheaper, testable, and auditable. Attribution's shape, evidence in, structured diagnosis with records out, is a reference design for teams automating postmortems, whether or not they adopt the tool itself.

Related Reading

For more NVIDIA open releases covered here, see NVIDIA open models from GTC Berlin, six model families released with open weights. For the verification side of AI infrastructure, see FlashProve, Anthropic's prototype for proving which model produced an answer. Attribution covers the third side of the same operational problem: when the run itself fails, proving where it started.

FAQ

What is NVIDIA Attribution?

NVIDIA Attribution is an open source system that NVIDIA released on November 2, 2026 under the Apache 2.0 license. It collects logs, metrics, telemetry, and scheduler state for a distributed AI workload and returns one structured diagnosis with an outcome, a root cause, affected nodes, supporting symptoms, and evidence records.

Does Attribution use an LLM to diagnose failures?

No. Attribution judges evidence with deterministic rules, so the same input always produces the same diagnosis. Operators can edit the rules to fit their own environment, although changed rules have to be deployed together with the Attribution Service rather than uploaded through an HTTP API.

What does Attribution connect to?

The Attribution Blueprint connects to Slurm, Kueue, and static input for execution context, and to Loki, local files, and Kubernetes pod logs plus PromQL metrics for evidence. Diagnoses are written to stdout and SQLite.

How accurate is Attribution?

NVIDIA has not published a benchmark for diagnostic accuracy. The release states the internal operating scale, millions of workloads per week across hundreds of thousands of GPUs, but not the share of diagnoses that identified the correct root cause. When evidence is thin the system returns no specific cause, and when several causes match, the sub_domain field is marked ambiguous.

Sources

AI InfraPerformanceAX Trend