Deep dive into Azure Maia 300 architecture

1x

Share

Link copied to clipboard!

Categories

AI + machine learning

DevOps

Tags

AI

Large language models (LLMs)

Audience

Business decision makers

Developers

IT decision makers

IT implementors

Content types

Thought leadership

Products

GitHub Copilot

Microsoft Foundry

From its inception, Azure Maia has been built around a single architectural philosophy: optimize the entire system for LLM inference so that every watt and every dollar delivers more intelligence. Now in its third generation, Maia 300 is the fullest expression of that philosophy. Maia 300 is built as a complete system, with software, silicon, physical rack infrastructure, networking, and datacenters co-designed to operate as a single platform to meet the demands of cloud-scale agentic inference.

That system-level approach is increasingly necessary as agentic workloads, including multi-step execution, long-context decoding, and tool use, change the computational profile and arithmetic intensity of inference. Meeting these demands requires sustaining locality, supplying compute with enough memory capacity and bandwidth, minimizing data movement, scaling communication efficiently, and fitting the most capability within a datacenter’s fixed power and cooling envelope.

At Microsoft, we see this transformation firsthand through the products and platforms our customers use every day, from Microsoft Copilot and GitHub Copilot to Microsoft Foundry. Operating these workloads at scale gives us a unique view into how AI is being adopted across industries and the infrastructure needed to support the next generation of AI applications and agents. Maia 300 is designed to maximize useful tokens per watt and useful tokens per dollar, bringing more capacity, capability, and performance to these products and Microsoft’s broader AI portfolio for our customers.

Maia 200 introduced a new class of accelerator architecture called Software Defined Locally Accessed Dataflow Architecture, or SDLA. It begins with the observation that efficient data management, including storing, moving and converting data, is one of the central challenges in modern AI systems. The Maia programmable dataflow architecture separates control path programming from data path execution and allows software to explicitly coordinate data placement, movement, conversion and synchronization across specialized local memories, DMA engines, compute engines, and the network.

Maia 300 system-on-chip architecture

Maia 300 evolves this architecture for long-context, reasoning-intensive, and decode-heavy inference. First, memory moves from the uniform memory access (UMA) organization of the SoC to a locality-aware non-uniform (NUMA) organization across chiplets, so software can place data near the compute that uses it. Second, new narrow-precision formats with hardware-accelerated data type conversion reduce the bytes moved for each token. Third, the rack doubles to 96 accelerators and the scale-up domain grows to a superpod of 12,288 accelerators, enabling more accelerators to work effectively as a single system.

Rather than scaling performance through increased power and silicon area alone, Maia 300 improves performance per watt through a set of architectural innovations that optimize data movement, locality, precision, and memory efficiency across the system:

Maia 300 brings 2.5 times the memory bandwidth, 33% more capacity, and approximately 2 times the HBM bandwidth per watt compared with Maia 200

One scale-up fabric: From pod to superpod

Maia 300 makes the network a native extension of its software-defined dataflow architecture. Maia’s AI network controller implements custom AI transport based on Ethernet. A two-tier network builds a high-performance superpod with up to 12,288 accelerators. Maia presents one communication model to software—the same semantics to move data directly to any accelerator within a superpod.

One fabric from the accelerators in a pod to the superpod

One fabric rather than separate scale-up and scale-out domains Integrated network controllers and specialized AI transport use the same communication semantics inside the pod and across the superpod. Maia accelerators can directly initiate communication to any accelerator in the superpod, without involving the host or a discrete scale-out NIC.

Low-hop communication paths There are three latency domains: A direct connected accelerator pair is the lowest latency domain, then accelerators within a pod are one switch hop away, and any accelerator across pods is three switch hops away. Software can keep latency-sensitive tensor-parallel traffic local within the lowest latency domains.

Bandwidth allocated to the workload The bandwidth split between in-rack and cross-rack bandwidth per accelerator is not fixed at the system design time. It becomes a parameter that is set based on job packing. An accelerator can drive up to 750 GB/s to any peer within the superpod, when only a subset of accelerators are going cross-rack. Refer to the spare tray example below.

Flexible job placement Tensor parallelism can remain within the 96-accelerator rack, while wider expert and pipeline parallelism span racks. Jobs can use capacity across rack boundaries, reducing stranded accelerators when workload sizes do not align with the physical rack configuration.

A 64-accelerator job can run as four 16-accelerator shards, one in each of four racks

Disaggregated inference within the scale-up domain Prefill and decode workers can be sized and placed independently. After prefill computation completes, KV state moves directly to the selected decode workers over the same scale-up fabric rather than leaving the domain through a separate host-mediated scale-out path.

Resilience across racks When accelerators become unavailable, schedulers can use hot-spare capacity in another rack within the superpod transparently to software kernels. The workload remains on the scale-up fabric, trading additional path latency for improved availability and resource utilization.

A spare tray in one rack can offset a failed tray in another rack

Rack-scale architecture for deployment efficiency

At datacenter scale, efficiency is determined by how much useful infrastructure can be deployed and operated within a fixed power and cooling envelope. Building on the foundational energy efficiency of the Maia SoC architecture, the Maia 300 rack co-designs server layout and density, network topology and switching, power and cooling design, and advanced telemetry to deliver up to 2.5x more performance per watt compared to Maia 200.

Consolidated switch architecture Six high-radix switch ASICs are integrated into three liquid-cooled switch trays, providing the rack-local connectivity required by all 96 accelerators while minimizing the space and power consumed by switching.

Rack-level power delivery Dual power feeds, high-capacity power shelves, and a 50 V busbar architecture distribute power efficiently across accelerator and switch trays while providing operational redundancy. The integrated rack automatic transfer switch maintains high availability even when a single power feed is offline for servicing.

Flexible cooling architecture Liquid cooling removes more than 80% of the rack’s thermal load from accelerators, voltage regulators, and switch ASICs, while air cooling supports memory, storage, and other ancillary components. Maia 300 can connect directly to facility water where it is available or use a heat-exchanger unit to reject heat in air-cooled datacenter environments. This provides greater flexibility in where and how the system can be deployed at scale.

Datacenter-integrated telemetry We have incorporated deployment learnings to strengthen reliability, availability, and serviceability (RAS), observability, debuggability, fault handling, and redundancy, improving robustness as we deploy additional volume. Rack and row level power telemetry connect with datacenter operations, enabling operating points to be managed against available power and cooling capacity. This data informs algorithms that dynamically allocate power and optimize workload placement and deployment density, improving utilization and tokens delivered per watt.

What Maia 300 delivers

Maia 300 is the next step in a multigenerational program to deploy efficient AI systems at scale. The design is optimized end to end across software, silicon, physical rack infrastructure, networking, and datacenters to deliver a leap in performance and efficiency for cloud-scale agentic inference.

Relative to Maia 200, Maia 300 delivers up to 7 times greater rack-scale token throughput in its full 96-accelerator configuration and more than 2.5 times the performance per watt. It brings more memory bandwidth per watt and twice the accelerators per rack, while building on SDLA, our software-defined dataflow architecture and a two-tier scale-up network to support a single programming model spanning kernel to data center scale. Together, these advances enable us to deliver more useful intelligence, more efficiently, to more people and organizations.

Maia 300 delivers up to 7 times greater rack-scale token throughput and more than 2.5 times the performance per watt compared with Maia 200

The opportunity ahead is not simply to build more compute. It is to make advanced AI capabilities more accessible, affordable, and useful. Maia 300 is an important step toward that goal.

Next month at Microsoft Ignite 2026 in San Francisco and online, we’re looking forward to sharing more information on our AI strategy, technology adoption trends, and success in Frontier Transformation.

Read about the latest Azure Maia 300’s news here.

Learn more about Azure’s silicon to systems approach here.

President, Azure Hardware Systems and Infrastructure

Corporate Vice President, Systems Planning and Architecture