返回资讯中心
外部精选
软件工程
#开发者实时流

TorchNeuron: Run PyTorch on AWS Trainium with minimal code changes

torch-neuronx is the native PyTorch backend for AWS Trainium. It adds a neuron device to upstream PyTorch through the PrivateUse1 mechanism. You use the PyTorch APIs that you already know. Move a model to neuron, then t…

Hacker News 最新ilreb2026/10/19 · 27 天前15 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

torch-neuronx is the native PyTorch backend for AWS Trainium. It adds a neuron device to upstream PyTorch through the PrivateUse1 mechanism. You use the PyTorch APIs that you already know. Move a model to neuron, then train or run it.

If you use the XLA-based torch-neuronx (PyTorch/XLA), see PyTorch support on Neuron. This repository is the PyTorch-native backend.

You need a Trn2 or Trn3 instance, and Python 3.12, 3.13, or 3.14.

Neuron DLAMI or Deep Learning Container (recommended). torch-neuronx is already installed. Activate the Python environment that comes with it.

Other Linux instance. Install the Neuron driver and runtime for your OS, and then install torch-neuronx:

. /etc/os-release wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | sudo gpg --dearmor -o /usr/share/keyrings/neuron.gpg echo "deb [signed-by=/usr/share/keyrings/neuron.gpg] https://apt.repos.neuron.amazonaws.com ${VERSION_CODENAME} main" | sudo tee /etc/apt/sources.list.d/neuron.list sudo apt-get update -y sudo apt-get install -y linux-headers-$(uname -r) aws-neuronx-dkms aws-neuronx-collectives aws-neuronx-runtime-lib aws-neuronx-tools export PATH=/opt/aws/neuron/bin:$PATH

Amazon Linux 2023 sudo tee /etc/yum.repos.d/neuron.repo > /dev/null <<EOT [neuron] name=Neuron YUM Repository baseurl=https://yum.repos.neuron.amazonaws.com enabled=1 metadata_expire=0 EOT sudo rpm --import https://yum.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB sudo dnf install -y "kernel-devel-uname-r = $(uname -r)" aws-neuronx-dkms aws-neuronx-collectives aws-neuronx-runtime-lib aws-neuronx-tools export PATH=/opt/aws/neuron/bin:$PATH

EFA driver, for multi-node training curl -O https://efa-installer.amazonaws.com/aws-efa-installer-latest.tar.gz tar -xf aws-efa-installer-latest.tar.gz cd aws-efa-installer && sudo bash efa_installer.sh --yes

python3 -m venv ~/neuron_venv && source ~/neuron_venv/bin/activate pip install torch==2.14.0 --index-url https://download.pytorch.org/whl/cpu pip install "torch-neuronx==2.14." "neuronx-cc==2." "nki>=0.6" --extra-index-url https://pip.repos.neuron.amazonaws.com

Keep the version pins. Earlier torch-neuronx versions are the XLA-based product.

To check the installation, run python -m torch_neuronx.collect_env. The device usable from torch line must show yes.

To build from source, see CONTRIBUTING.md.

After you install torch-neuronx, run this script on a Trainium instance. It runs a model in eager mode, and then compiles the same model with the neuron backend:

import torch # PyTorch loads torch-neuronx automatically.

model = torch.nn.Linear(16, 8).to("neuron").eval() inputs = torch.randn(4, 16, device="neuron")

with torch.no_grad(): eager_output = model(inputs) compiled_model = torch.compile(model, backend="neuron", dynamic=False) compiled_output = compiled_model(inputs)

print(tuple(eager_output.cpu().shape), tuple(compiled_output.cpu().shape))

The script prints (4, 8) (4, 8). The first run compiles the operations and the graph, so it takes longer than later runs.

To run your own model, move the model and its inputs to the neuron device. For training, distributed training, and NKI kernels, see the TorchNeuron tutorials and the examples.

The API compatibility guide lists each PyTorch API as supported, supported with CPU fallback, or not supported.

Experimental: adaptive eager execution fuses eligible eager operations into one compiled unit. It is off by default. To try it, set TORCH_NEURONX_ENABLE_CONCATENATION=1 before you start Python.

Define the kernel with nki.jit. Then register the kernel as a PyTorch operator with nki_op, and call the operator like any other PyTorch operation:

import nki import nki.isa as nisa import nki.language as nl import torch from torch_neuronx import nki_op

@nki.jit def add_kernel(a, b): a_sbuf = nl.ndarray(a.shape, dtype=a.dtype, buffer=nl.sbuf) nisa.dma_copy(dst=a_sbuf, src=a) b_sbuf = nl.ndarray(b.shape, dtype=b.dtype, buffer=nl.sbuf) nisa.dma_copy(dst=b_sbuf, src=b) c_sbuf = nl.ndarray(a.shape, dtype=a.dtype, buffer=nl.sbuf) nisa.tensor_tensor(dst=c_sbuf, data1=a_sbuf, data2=b_sbuf, op=nl.add) out = nl.ndarray(a.shape, dtype=a.dtype, buffer=nl.shared_hbm) nisa.dma_copy(dst=out, src=c_sbuf) return out

@nki_op("example::add", mutates_args={}) def nki_add(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: return add_kernel(a, b)

a = torch.rand(128, 128, device="neuron") b = torch.rand(128, 128, device="neuron") print(torch.allclose(nki_add(a, b).cpu(), (a + b).cpu()))

The script prints True. An nki_op also works inside torch.compile. The backend puts the kernel into the compiled graph, so the graph does not break. To train through the kernel, register a backward pass with torch.library.register_autograd. For in-place kernels and wrap_nki, see How to work with NKI kernels.

Example Description

compile/QUICKSTART.md Getting started with torch.compile on Neuron

llama Llama-style model: training step and text generation

torch_compile/whisper Whisper speech-to-text

torch_compile/qwen2 Qwen2 model

fused_attention Scaled dot-product and multi-head attention

simple_moe Mixture of Experts

gpt2-train-loop GPT-2 training loop

simple_mlp_tpxfsdp Tensor parallel and FSDP

multi_node_tests Multi-node distributed training

simple_asr Conformer ASR training

Documentation

The full documentation for torch-neuronx is in the AWS Neuron documentation.

The torch-neuronx version has the format <PyTorch major.minor>.<torch-neuronx major.minor>.<build>. For example, 2.14.3.0.4527 is torch-neuronx 3.0, build 4527, for PyTorch 2.14.

For the features and known gaps of this release, see release-notes/2.33.0.md.

Report bugs or request features: GitHub Issues. To report a security issue, follow SECURITY.md.

This project follows the Amazon Open Source Code of Conduct. For more information, see the Code of Conduct FAQ or contact opensource-codeofconduct@amazon.com.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →