返回资讯中心
外部精选
软件工程
#开发者实时流

Triton Notes

Triton is a programming language and compiler originally developed at OpenAI that makes custom GPU kernels easier to write. It makes efficient GPU code easier to express than in CUDA, especially for deep learning worklo…

Hacker News 最新qwertyforce2026/10/20 · 28 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Triton is a programming language and compiler originally developed at OpenAI that makes custom GPU kernels easier to write. It makes efficient GPU code easier to express than in CUDA, especially for deep learning workloads.

In CUDA, you usually decide what each individual thread does and manually manage many low-level details. Triton provides a higher-level abstraction: you describe what one program does to a block, or tile, of data. Triton's compiler then maps those block operations onto GPU threads and generates the GPU code. We will see what this means later in this note.

It is also worth noting that Triton uses the same style of array programming found in PyTorch and NumPy. If that way of thinking is unfamiliar, these two introductions are useful refreshers to think in shapes:

This note is an introduction to Triton's abstractions. We will use two kernels—vector addition and matrix multiplication—throughout the note to build intuition.

At a high level, GPU programming has two sides regardless of the DSL:

We will divide this into two steps and look at each:

As in CUDA, the host (CPU) launches the kernel before any device-side work begins. A single launch creates many instances of the kernel, called programs in Triton.

Because kernels provide considerable freedom over variables and memory, it is helpful to fail fast and loud when something is wrong. We therefore validate shapes, strides, dtypes, and device-specific features, such as those introduced with Blackwell.

These assertions largely serve two purposes: ensuring correctness and enabling specialized fast paths. Many kernels branch into completely different implementations based on the results of these checks.

def vector_add(a: torch.Tensor, b: torch.Tensor): assert a.is_cuda and b.is_cuda assert a.device == b.device assert a.ndim == b.ndim == 1 assert a.shape == b.shape assert a.dtype == b.dtype

...

#Setting Grid Size

A launch grid is a 1D, 2D, or 3D tuple specifying how many program instances run along each axis. Inside the kernel, tl.program_id(axis) identifies the current program along that axis.

Poor grid mapping can reduce parallelism or cache locality, so it's important to tune it.

So how do we set the grid size? Most kernels use one of two launch styles:

In a non-persistent tiled launch, we divide the output into tiles of BN elements and launch one program for each tile. Program 0 processes the first BN elements, program 1 processes the next BN elements, and so on. Therefore, the grid size is the number of tiles required to cover all N elements.

...

grid = lambda meta: (triton.cdiv(N, meta["BN"]),) _vec_add_kernel[grid](

...

)

What is happening here?

triton.cdiv(a, b) computes ceil(a / b). This ensures that a final partial tile still receives a program. The kernel must mask any lanes that fall outside the logical tensor bounds.

When grid is a function, Triton passes it a dictionary containing the kernel's compile-time meta-parameters, such as BN. This lets the grid size depend on the selected block sizes, including values selected by autotuning.

A persistent kernel launches a limited number of programs, often bounded by the number of output tiles and chosen relative to the GPU's SM count. Each program loops over tiles inside the kernel. See when one program processes multiple tiles for how this works in the kernel.

We pass the input tensors that the kernel reads and preallocated output tensors that it writes.

The number of elements in a tensor (e.g., 1,000) may not be divisible by its tile size (e.g., 32). The kernel receives the logical dimension and creates a mask such as offs_n < N, preventing lanes in a partial tile from accessing memory outside the vectors.

Stride is the number of elements between adjacent positions along one tensor dimension. For vector addition, the stride is usually 1 because vectors are commonly contiguous, but a non-contiguous vector may have a different stride. We will talk more about strides in the two-dimensional case.

An argument annotated with tl.constexpr is known at compile time. Triton can use it to specialize generated code, size block-shaped tensors, or remove compile-time branches. Tile sizes are common autotuning targets, while tl.constexpr is also used for algorithm variants and feature flags.

def vector_add(a: torch.Tensor, b: torch.Tensor):

...

_vec_add_kernel[grid]( a, b, c, # a, b as input tensor, c as output tensor N, a.stride(0), b.stride(0), c.stride(0), # all 1 BN=1024, # known at compile time; we can autotune it )

#End-to-end example: launching a vector add kernel

pythonCopy

12345678910111213141516171819

def vector_add(a: torch.Tensor, b: torch.Tensor): assert a.is_cuda and b.is_cuda assert a.device == b.device assert a.ndim == b.ndim == 1 assert a.shape == b.shape assert a.dtype == b.dtype

c = torch.empty_like(a) N = c.numel()

grid = lambda meta: (triton.cdiv(N, meta["BN"]),)

_vec_add_kernel[grid]( a, b, c, # a, b as input tensor, c as output tensor N, a.stride(0), b.stride(0), c.stride(0), # all 1 BN=1024, # known at compile time; we can autotune it ) return c

Note

Q. Is separating the kernel launch from the kernel definition the best approach?

A. Repeated kernel launches can add overhead. Techniques such as CUDA graphs and MegaKernel can mitigate that overhead, but most kernels are still launched from the CPU.

Until now, we have seen how the host launches a kernel. Before looking at how a Triton kernel is defined, let's first go over the two most important abstractions in Triton: tile and program.

If the vectors contain one million elements, a Triton kernel usually does not process all one million elements in a single program. Instead, we divide the output into smaller regions and launch multiple instances of the kernel.

TL;DR: A tile is a region of the data, while a program is an executing kernel instance that processes that region.

A tile is a logical region of the overall problem that is processed together.

For example, one program might process a block of 1024 consecutive elements of the vector addition output C.

tile 0: C[ 0 : 1024] tile 1: C[1024 : 2048] tile 2: C[2048 : 3072] ...

A tile is represented by block-shaped offsets, pointers, and values inside the program.

This differs from CUDA, where the unit of code is a thread: values are scalars, and you index one element at a time.

In Triton, tile and block are often used almost interchangeably. Both commonly refer to a group of tensor elements processed together by one program.

For example, a program may be responsible for the tile C[1024:2048]. Inside the program, that tile is represented by blocks of offsets and values containing up to 1024 elements.

In this note, we use them interchangeably.

A Triton program is one executing instance of a compiled kernel. It is roughly analogous to a CUDA thread block, also called a CTA.

The CPU kernel launch shown above creates as many program instances as specified by the grid. For example, if N = 4,096 and BN = 1024, four programs are launched.

program 0 → tile 0 → C[ 0 : 1024] program 1 → tile 1 → C[1024 : 2048] program 2 → tile 2 → C[2048 : 3072] program 3 → tile 3 → C[3072 : 4096]

Each program receives a program ID (pid) and usually processes a different input or output tile. Triton launch grids have at most three dimensions, matching the grid defined by the host.

All four programs execute the same kernel code but operate on different data. In fancy terms, we call this SPMD.

Now that we've covered the two central abstractions, let's see how a kernel is defined and run on the GPU.

At a high level, each program follows these steps:

Triton kernels receive runtime arguments and configuration metadata from the host code described in the previous section, including:

@triton.jit def _vec_add_kernel( a_ptr, b_ptr, c_ptr, N, stride_a, stride_b, stride_c, BN: tl.constexpr, ):

...

#1. Read the program ID and determine which output tile the program owns

#Program ID

A program ID, or pid, identifies a program along one launch-grid axis through tl.program_id(axis). The kernel uses it to determine which tile that program processes.

What does it mean by "along one launch-grid axis"?

Like CUDA, Triton's launch grid can be 1D, 2D, or 3D. The grid shape determines the range of program IDs along each axis.

_matmul_kernel(num_pid_m, num_pid_n)

If the grid is launched with two dimensions, tl.program_id(axis=0) selects an M tile and tl.program_id(axis=1) selects an N tile. The matmul kernel in these notes uses this 2D mapping directly and reads both axes.

A 1D grid is convenient when a kernel flattens its logical work into one tile index. A 2D or 3D grid can instead map independent dimensions directly, such as M/N tiles for matmul or batch/head/tile dimensions for attention.

@triton.jit def _vec_add_kernel(

...

): pid = tl.program_id(axis=0)

Here we use axis=0 because the host launched a one-dimensional grid.

Local offsets are indices within the tile, ranging from 0 to tile_size - 1. We simply use tl.arange (just like np.arange or torch.arange) to create the array.

Block sizes used as the endpoint of tl.arange must satisfy its power-of-two constraints. For the common form tl.arange(0, BN), choose a power-of-two BN, such as 128, 256, or 1024, rather than an arbitrary value such as 1000.

local_offs_n = tl.arange(0, BN)

To locate the tile within the full tensor, we incorporate the program ID. In practice, kernels commonly construct global offsets directly:

offs_n = pid * BN + tl.arange(0, BN)

What does this mean? Since each program processes a tile of size BN, we multiply it by pid to locate that program's block. So offs_n ranges from pid * BN to (pid + 1) * BN - 1.

For example, if BN = 1024 and pid = 3, offs_n is:

3 * 1024 + tl.arange(0, 1024) == [3072, 3073, 3074, 3075, ..., 4095]

These lines construct block-shaped tensors of output coordinates. The coordinates are later combined with base address and strides to construct pointers for A, B, and C.

As explained, masks allow a regular block-shaped program to safely handle the final partial tile. A mask is usually applied to the global offsets.

@triton.jit def _vec_add_kernel(

...

):

...

offs_n = pid * BN + tl.arange(0, BN) n_mask = offs_n < N

This way, even if N is not a multiple of BN, the final partial tile can be masked correctly.

Once we have the global offsets, we map the logical indices to memory pointers. These are the final memory addresses we will use to load the tensor elements.

@triton.jit def _vec_add_kernel( a_ptr, b_ptr, c_ptr,

...

):

...

a_ptrs = a_ptr + offs_n * stride_a b_ptrs = b_ptr + offs_n * stride_b c_ptrs = c_ptr + offs_n * stride_c

#4. Load the input blocks

Input tensors normally already reside in GPU global memory before the kernel begins. A Triton kernel reads values with tl.load, computes on them using on-chip resources (typically registers), and writes results back with tl.store. The compiler and hardware manage lower-level movement through the cache hierarchy.

We use the mask we've created here as well to mask off elements outside the bounds.

The other argument supplies a value for masked-out loads. We usually use 0.0 or -float("inf") depending on the context. For example, vector addition can simply use 0.0.

@triton.jit def _vec_add_kernel( a_ptr, b_ptr,

...

):

...

a_ptrs = a_ptr + offs_n * stride_a b_ptrs = b_ptr + offs_n * stride_b a = tl.load(a_ptrs, mask=n_mask, other=0.0) b = tl.load(b_ptrs, mask=n_mask, other=0.0)

#5. Compute the output block

In Triton, computation is pretty high-level. More complex kernels may require additional operations, but many computations are expressed using basic arithmetic or matrix multiplication (tl.dot). Vector addition is dead simple:

@triton.jit def _vec_add_kernel( a_ptr, b_ptr,

...

):

...

a_ptrs = a_ptr + offs_n * stride_a b_ptrs = b_p

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →