Microsoft recently announced its second-generation custom AI accelerator, the Maia 200:

• Microsoft’s second-generation custom AI accelerator focuses exclusively on inference workloads using industry-standard FP4/FP8 precision
• Built on TSMC 3nm process with 144 billion transistors, delivering vendor-claimed 30% better performance per dollar than existing Azure hardware
• 10.15 petaflops FP4 performance with 216 GB HBM3e memory at 7 TB/sec bandwidth in 750-watt envelope
• AI Transport Layer interconnect supports clusters up to 6,144 accelerators using eight-rail Ethernet fabric
• Abandons proprietary MX data formats for ecosystem compatibility; includes Maia SDK with PyTorch integration
• Initially deployed in US Central and US West 3 regions supporting OpenAI GPT-5.2 and Microsoft synthetic data

---

Microsoft recently announced its second-generation custom AI accelerator, the Maia 200. The new chip is an inference-optimized alternative to third-party GPUs in its Azure infrastructure. The company says the accelerator delivers 30% better performance per dollar than existing Azure hardware while supporting OpenAI’s GPT-5.2 models and Microsoft’s own synthetic data generation workloads.

Unlike its predecessor, the Maia 100, which targeted both training and inference, the Maia 200 focuses exclusively on inference tasks using standard FP4 and FP8 precision formats rather than Microsoft’s proprietary MX data types.

Technical Details

The Maia 200 is a significant architectural departure from its predecessor, fabricated on TSMC’s N3P performance variant of its 3-nanometer process. The chip contains approximately 144 billion transistors across an estimated 836 mm² die area, approaching the 858 mm² reticle limit of current lithography methods.

This is a 37% increase in transistor count over the Maia 100’s 105 billion transistors, achieved primarily through process shrinkage rather than die expansion.

Microsoft’s architectural choices reveal clear prioritization decisions that favor memory capacity and bandwidth over on-chip cache:

• Compute cores: The Maia 200 contains an estimated 96 cores arranged in clusters, a 50% increase over the Maia 100’s 64 cores. Each core includes separate tensor and vector units, with Microsoft suggesting approximately 92% yield (translating to 88 usable cores in production parts).
• Clock speed: The N3P process allows 3.1 GHz operation, an 8% increase over the Maia 100’s 2.86 GHz clock.
• On-chip SRAM: Microsoft specifies 272 MB of SRAM partitioned into multi-tier Cluster-level SRAM (CSRAM) and Tile-level SRAM (TSRAM). This is a reduction from the Maia 100’s 500 MB total cache, with aggregate SRAM bandwidth decreasing by about 60% despite the core count increase.

Memory Subsystem

The most substantial improvements appear in the off-chip memory configuration. Microsoft equipped the Maia 200 with six stacks of twelve-high HBM3e memory from sk hynix, substantially exceeding Maia 100’s capabilities:

• Capacity: 216 GB total (36 GB per stack), representing a 3.4X increase over the Maia 100’s 64 GB HBM2E configuration
• Bandwidth: 7 TB/sec aggregate, a 3.9X improvement over the previous generation’s 1.8 TB/sec
• Memory technology: HBM3e rather than HBM2E, contributing both to capacity and bandwidth gains

Its memory configuration allows the Maia 200 to handle larger model deployments (though practitioners should note that the reduction in on-chip SRAM relative to core count may create data movement bottlenecks for certain workload patterns).

Compute Performance

Microsoft abandoned its proprietary MX6 and MX9 data formats in favor of industry-standard precision types:

• FP4 performance: 10.15 petaflops (vendor specification)
• FP8 performance: 5.07 petaflops on tensor units
• BF16 performance: 1.27 petaflops on vector units
• Thermal envelope: 750 watts TDP

The shift to standard data formats improves model portability but eliminates any potential advantages Microsoft claimed for its microexponent-based precision types.

Its 750-watt TDP is a 50% increase over the Maia 100’s advertised specifications, though Microsoft reportedly ran the previous generation at 500 watts in production environments.

Interconnect

The Maia 200 introduces what Microsoft calls the AI Transport Layer (ATL), an evolution of the Maia 100’s RoCE Ethernet-based interconnect. This architecture employs 56 lanes of 400 Gb/sec SerDes delivering 2.8 TB/sec bidirectional bandwidth, promising to deliver a 2.33X improvement over the previous generation:

• Intra-quad connectivity: Nine lanes create all-to-all connections between four Maia 200 accelerators on a single blade server.
• Inter-rack connectivity: The remaining 47 lanes implement eight separate rails for packet spraying across the ATL fabric.
• Scale-up domain: Microsoft says it supports coherent clusters of up to 1,536 nodes containing 6,144 accelerators (compared to the Maia 100’s 576-node, 2,304-accelerator domain).

The ATL relies on a two-tier Ethernet topology rather than proprietary fabrics like NVIDIA’s NVLink. Its integrated NIC design eliminates discrete network adapters, but couples network evolution to silicon refresh cycles.

System Integration

Each Maia 200 blade server houses four accelerators alongside a single Cobalt 200 Arm-based CPU (Microsoft’s second-generation custom Arm-based processor.) The company employs second-generation closed-loop liquid cooling through Heat Exchanger Units (HXU), providing thermal density that air cooling can’t address at rack scale.

Impact to IT & AI Practitioners

Organizations evaluating Azure AI infrastructure face several operational considerations driven by the Maia 200’s deployment:

• Workload suitability: The Maia 200’s focus on FP4 and FP8 inference optimizes it for specific deployment patterns. Organizations running quantized large language models for token generation should see cost benefits from Microsoft’s claimed 30% performance-per-dollar improvement, assuming workloads align with the accelerator’s architectural strengths. However, teams requiring BF16 or higher precision for inference quality reasons may find the Maia 200’s 1.27 petaflops BF16 performance insufficient compared to alternatives, forcing them toward more expensive GPU instances.
• Development and migration costs: Microsoft’s Maia SDK preview offers PyTorch integration, a Triton compiler, and optimized kernel libraries intended to simplify model porting. Despite these tools, organizations should anticipate non-trivial engineering investment to validate performance, optimize memory layouts for the Maia 200’s SRAM hierarchy, and potentially restructure inference pipelines to leverage the ATL interconnect’s characteristics.
• Lock-in considerations: Migrating workloads to Maia 200-optimized implementations creates vendor dependency that extends beyond typical cloud lock-in. Organizations optimizing for Maia-specific SRAM hierarchies, ATL communication patterns, or precision characteristics may find later migrations to alternative infrastructure require substantial re-engineering. This risk compounds for teams building on Microsoft’s Maia-specific programming interfaces beyond standard PyTorch abstractions.

Analysis

Microsoft’s Maia 200 follows broader hyperscaler strategies to reduce dependence on external chip suppliers while maintaining heterogeneous infrastructure that preserves customer choice. The company’s approach, however, differs materially from competitors in several aspects that shape its market impact:

• Differentiation through vertical integration: The Maia 200’s integration with Cobalt 200 CPUs and Azure’s control plane demonstrates Microsoft’s advantage in coordinating silicon, system design, and datacenter infrastructure.
• Economic model transparency: Microsoft’s stated “30% performance-per-dollar” improvement raises questions about the comparison baseline. If measured against Maia 100 instances that never reached general availability, the metric provides limited practical guidance. If compared to current-generation NVIIDA instances, the improvement magnitude depends on Microsoft’s instance pricing strategy, which hasn’t yet been disclosed.
• Ecosystem implications: The abandonment of MX6 and MX9 data formats signals Microsoft’s recognition that proprietary precision types hinder adoption regardless of theoretical advantages. This pragmatic shift toward FP4 and FP8 industry standards improves Maia 200’s ecosystem compatibility.

Overall, the new Maia 200 is Microsoft’s calculated approach to reducing its dependence on third-party AI accelerators. The accelerator’s focus on inference optimization, abandonment of proprietary data formats, and substantial memory subsystem improvements address real deployment requirements for organizations running quantized large language models at scale.

The broader significance of the Maia 200 lies in its demonstration that hyperscalers possess the technical capability to develop competitive AI accelerators, even if ecosystem maturity and software tooling lag established alternatives. For the AI infrastructure market, this validates the viability of custom silicon approaches while highlighting the substantial engineering investment required to challenge incumbents.

Competitive Impact & Advice to IT Buyers

The Maia 200 enters a continuously evolving landscape where multiple hyperscalers and chip vendors compete across different optimization points and go-to-market strategies.

Microsoft’s play with the Maia 200 presents both competitive advantages and structural challenges…

These sections are only available to NAND Research clients and IT Advisory Members. Please reach out to [email protected] to learn more.