
Nvidia Attribution
NVIDIA Attribution is a service-plus-orchestrator that turns the raw symptoms of a failed GPU workload into one structured diagnosis with the symptom chain that produced it.
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
NVIDIA Attribution is a service-plus-orchestrator that turns the raw symptoms of a failed GPU workload into one structured diagnosis with the symptom chain that produced it.
Modern AI training and inference workloads fail in dozens of ways. A multi-rank workload at 10k-GPU scale typically produces hundreds to tens of thousands of log lines across application logs, host syslogs, scheduler database entries, GPU driver dmesg, fabric manager logs, and metric backends. Application logs in particular scale with rank count, so volume grows with the size of the workload. Triaging them by hand breaks down at production-cluster volume. NVIDIA Attribution automates the cross-correlation. Each failed workload becomes one structured diagnosis, with the cascade rendered as the symptom chain that produced it.
Status. NVIDIA Attribution is open-source software. The Service exposes a versioned HTTP API. The Library and Blueprint interfaces can change. See SUPPORT.md for compatibility commitments and CONTRIBUTING.md to propose improvements.
NVIDIA Attribution ships in two components that share one rules engine. Pick the form that matches how you want to integrate.
Both use the Apache License 2.0. The Blueprint is the orchestrator that wires the Service into a real cluster. It implements a modular connector architecture designed to be extended and customized for your production environment. Run it as-is or adapt it to your topology, and propose improvements through the contribution workflow.
NVIDIA Attribution identifies and correlates symptoms from XIDs, NCCL events, application logs from training frameworks (PyTorch, NeMo), kernel messages, scheduler state, and metric counters into one diagnosis per failed workload. The diagnosis carries the failure class, the responsible node when the cascade points there, the specific cause when the symptoms support it, and the chain of events that led to it.
A workload finishes. The engine looks back at what happened.
For more detail, read How Attribution Works or step through the interactive attribution process walkthrough.
The example below shows what that collapse looks like for a single workload.
applog Connection closed 800 occurrences across 128 nodes applog ncclRemoteError 500 occurrences across 128 nodes applog ncclInternalError 3 occurrences across 1 node applog NCCL invalid NIC selection 1 occurrence across 1 node scheduler Cancelled by user 2 occurrences scheduler Cancelled due to task failure 1 occurrence
The engine collapses these transport symptoms to one root cause. For this workload NVIDIA Attribution returns:
terminal_state: fail sub_domain: system_software
At NVIDIA scale
Three cases the rules engine handles in production:
NVIDIA Attribution doesn't probe. It is a log correlation engine. It does not run diagnostic commands or probe hardware state. The output describes the cascade and names the rule that fired. It does not make binary hardware-call decisions (replace, return to service). Those calls are the operator's.
NVIDIA Attribution doesn't replace your monitoring or telemetry stack. It consumes the signals your stack already produces (XIDs, scheduler state, log lines, Prometheus counters). It does not ingest, route, or alert on healthy workloads.
NVIDIA Attribution doesn't claim definitive attribution where the symptoms are ambiguous. Output quality scales with the quantity and coverage of the logs the Service sees. When multiple causes fit, sub_domain is ambiguous; when the evidence is insufficient to attribute, sub_domain is unattributed. Those are honest narrowings, not false confidence.
Prerequisites
Python. >=3.14.6,<3.15 on the host running the Attribution Service. 3.11+ on the host running the Blueprint orchestrator. The two can run on the same host or different hosts.
uv package manager. Install with curl -LsSf https://astral.sh/uv/install.sh | sh. Every uv run command resolves and installs dependencies on first invocation, no separate install step.
Scheduler. Slurm 25.11+ (SLUID required), or Kueue v0.6+ with JobSet v0.5+ for JobSet-shaped distributed training. NVIDIA KAI Scheduler support is pending: the Service API accepts kai as a scheduler value, but attribution for it is not implemented yet.
Telemetry backends (any subset of): Loki for application or system logs, a PromQL-compatible Prometheus or Mimir instance for node metrics, kubectl access for Kueue pod log collection, or local files for sacct-resolved StdOut/StdErr paths.
Network reach. The Blueprint must reach the Attribution Service over HTTP. The Service does not need outbound network access.
Supported GPU architectures. A100, H100, L40S, B200, GB200, GB300. GB200 and GB300 rule coverage continues to mature as coverage expands.
Quick start
Run the attribution service and web UI in Docker, attribute sample AI factory evidence, then view the results.
git clone https://github.com/dsx-ai-factory/attribution.git nvidia-attribution cd nvidia-attribution
The bundle contains the attribution service under service/service-app/ and the blueprint reference implementation under blueprint/.
This quickstart assumes you are running commands from the root directory of the attribution service bundle.
export ATTRIBUTION_IMAGE="attribution-blueprint-web:quickstart" export ATTRIBUTION_CONTAINER="attribution-blueprint-web-local" export ATTRIBUTION_SERVICE_PORT="8000" export ATTRIBUTION_WEB_PORT="8080" export ATTRIBUTION_SERVICE_URL="http://127.0.0.1:$ATTRIBUTION_SERVICE_PORT" export ATTRIBUTION_DB=".attribution-sample/attributions.db"
Start Docker
docker build
-f blueprint/docker/Dockerfile
-t "$ATTRIBUTION_IMAGE"
"$PWD"
mkdir -p "$(dirname "$ATTRIBUTION_DB")"
docker run -d --rm
--name "$ATTRIBUTION_CONTAINER"
-e ATTRIBUTION_DB=/data/attributions.db
-p "127.0.0.1:$ATTRIBUTION_SERVICE_PORT:8000"
-p "127.0.0.1:$ATTRIBUTION_WEB_PORT:8080"
-v "$PWD/$(dirname "$ATTRIBUTION_DB"):/data"
"$ATTRIBUTION_IMAGE"
Run Attribution
This run uses the committed per-job TOML files under blueprint/examples/static/configs.
uv run --project blueprint --frozen nvidia-attribution-blueprint
--configs blueprint/examples/static/configs
--connector attribution service_url "$ATTRIBUTION_SERVICE_URL"
--connector output db sqlite path "$ATTRIBUTION_DB"
The command should finish with:
Attribution complete: 118/118 execution contexts succeeded.
The web UI shows the outcomes produced by the attribution rules in the service version you are running.
If ATTRIBUTION_DB already exists, the command appends a new attribution run for each sample job. Use a different database path if you want a separate run history.
Open http://127.0.0.1:8080, or the port you set in ATTRIBUTION_WEB_PORT.
To run the web UI from a source checkout without Docker, use blueprint/scripts/webui.sh. It builds the React assets and serves them from the FastAPI backend against a SQLite database. Pass your own database and port:
blueprint/scripts/webui.sh --db "$ATTRIBUTION_DB" --port "$ATTRIBUTION_WEB_PORT"
Without --db, the helper seeds and serves a demo database. Run it with --help for all options, and see Web UI Flags for the underlying nvidia-attribution-blueprint-web command.
Start here for operators: Overview
How the engine reaches a diagnosis: How Attribution Works
System architecture: docs/ARCHITECTURE.md
Running the Service, web UI, and database: Service Operator Guide
Common questions: docs/FAQ.md
Writing rules: Add a New Extractor Rule
Runnable release-bundle quickstart: docs/quickstart.md
Blueprint configuration reference: docs/CONFIGURATION.md
Kueue example: examples/kueue
Attribution Service v2 REST walkthrough: docs/rest-api.md
Static sample configs used by the quickstart: examples/static/configs
Static sample evidence used by those configs: docs/sample-data/static
Maintainers
The repository maintainer team is listed in CODEOWNERS.
Contributions are welcome. Read CONTRIBUTING.md for the proposal, testing, sign-off, and review process, and follow the Code of Conduct.
Reproducible upstream bugs: GitHub Issues.
Usage questions and feature requests: GitHub Discussions.
Support scope, response targets, and compatibility: SUPPORT.md.
Security vulnerabilities: use the private reporting route in SECURITY.md.
Versions and release process: RELEASE.md.
License
NVIDIA Attribution project code is licensed under the Apache License 2.0. See LICENSE.
The components retain their own Apache-2.0 license files so their source and package distributions include the license when distributed separately:
The openapi/extras/ license is copied into the client package during generation.
The Code of Conduct retains its CC BY-SA 4.0 terms. OpenAPI Generator templates retain their upstream license. Bundled font files and dependencies retain their respective notices and terms. See THIRD_PARTY_NOTICES.md for these license exceptions.
The Blueprint WebUI and walkthrough builds also write assets/THIRD_PARTY_NOTICES.txt with the license and notice texts of the packages they bundle.
For task-focused workflows, see the Attribution Blueprint use-case guides.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。