IBM, Red Hat, Google Cloud, CoreWeave and NVIDIA release the first stable version
The llm-d project recently released version 1.0, the first stable release of an open-source, Kubernetes-native framework for distributing large language model inference across fleets of accelerators. llm-d sits above inference engines such as vLLM and SGLang, adding routing, cache management, and scheduling across model servers.
IBM, Red Hat, Google Cloud, CoreWeave, and NVIDIA founded the project in 2025. The Cloud Native Computing Foundation (CNCF) hosts it as a sandbox project, and Red Hat, IBM, and Google serve as lead maintainers. The project reports more than 500 contributors from over 25 organizations.
Earlier releases proved individual techniques, including prefix-cache-aware routing, KV-cache offloading and prefill/decode disaggregation. Version 1.0 adds the operating layer that production teams require, with multi-tenant quality of service, router high availability, coordinated rolling upgrades for disaggregated deployments, a v1 configuration API and session-aware routing for AI agents.
The release targets the cost structure of agentic workloads, where long and repeatedly reused contexts make input processing the largest expense.
The release also provides the industry with a vendor-neutral reference implementation of an inference control plane for open-weight models. One backed by an accelerator vendor, a hyperscaler, a neocloud, and IBM through its Red Hat subsidiary.
Technical Details
llm-d is a serving layer that runs on Kubernetes. It works something like this:
- A router, built on the Kubernetes Gateway API Inference Extension, selects the model server for each request.
- The model servers run vLLM, SGLang or AMD’s ATOM engine.
- Pools of prefill replicas and decode replicas handle the two phases of generation separately, with NVIDIA’s NIXL library moving KV-cache data between them. LMCache supports KV-cache offload.
Techniques Carried Into 1.0
The announcement describes five techniques as its foundation, with version 1.0 integrating them and hardening the request path:
- Prefix-cache and load-aware routing: Requests are directed to the replica that already holds the relevant context, avoiding recomputation.
- KV-cache offload: Cached context spills from GPU memory to CPU DRAM and storage, ensuring useful prefixes survive memory pressure.
- Prefill/decode disaggregation: Prompt processing and token generation run in separate pools that scale independently.
- Wide expert parallelism: Large MoE models span many accelerators with data-parallel attention.
- Batch and asynchronous inference: Grouped work increases utilization for latency-tolerant traffic.
New in Version 1.0
The release introduces four capability groups for day-two operations:
- Multi-tenant quality of service: Each request carries a flow key that combines a fairness identifier with a priority band, and the router maintains a separate queue per flow. Dispatch prioritizes by band, then cycles fairly across tenants within a band. Under saturation, the router holds requests while backend queues drain, and the probabilistic admission controller sheds load when demand exceeds capacity. By default, behavior remains first-come-first-served until operators enable the policies, so existing deployments do not change on upgrade.
- Zero-downtime operations: The endpoint picker runs with a hot standby that assumes scheduling if the active instance fails or is replaced. A DisaggregatedSet resource manages prefill and decode replicas as a single logical unit, gates readiness on a pod snapshot, and routes by revision. Requests reach a prefill/decode pair only when both sides run compatible versions, enabling lockstep in-place upgrades of split deployments.
- Stable configuration API: The configuration API graduates to v1, and flow control is consolidated under a single configuration surface. The release materials list no migration steps and no breaking changes.
- Agent-aware serving: A session-state plugin tags requests, and the router routes follow-up requests to the same backend until that replica saturates. Turn-priority scheduling considers a request’s position in an ongoing exchange, and the native Responses API state, including the previous response identifier, carries context across turns.
Two capabilities remain on the roadmap. Active-active router configurations and request-level failover or migration are not in 1.0, so the hot standby covers router replacement and does not recover in-flight requests.
Platform Support and Licensing
The release explicity supports the following accelerators, with NVIDIA, AMD, and Google each contributing vendor-specific optimization and enablement:
- NVIDIA H100, H200, B200, GB200 NVL72 and GB300 systems, with Vera Rubin support in progress.
- AMD support centers on the Instinct MI355X.
- Google supports TPU 7x through native PyTorch serving.
Performance & Cost
The performance figures below are from IBM Research, Red Hat, and the llm-d project, which published a large-model benchmark coinciding with the 1.0 release:
- Large-model throughput: IBM Research and Red Hat deployed GLM-5.2, an approximately 753-billion-parameter mixture-of-experts model with about 39 billion active parameters, across 544 NVIDIA H100 GPUs in a disaggregated topology. The deployment delivered over 6.6 million output tokens per minute at peak, with up to 3,000 concurrent coding agents and zero preemptions. At 3,000 agents, the system approached its serving limit, while 2,500 agents was the comfortable operating point at 7,612 requests per minute.
- Cost: At current cloud rental rates, IBM said the deployment costs “5 to 10 times less” per token than equivalent commercial API pricing, with the largest savings for input-heavy traffic.
- Routing gains: On the CyberGym agentic benchmark, moving from approximate to precise prefix routing increased throughput by 79 percent and reduced time to first token by 67 percent. The local prefix hit rate rose from 44.46 percent to 73.18 percent. Across the benchmark program, 85.2 percent of input tokens were served from cache.
- Chip design workload: IBM Research used open frontier models with llm-d for quantum error correction hardware design at about $0.11 per million tokens blended, with a prefix-cache hit rate near 98 percent. Red Hat cites an 11.5 times cost advantage over an all-proprietary configuration.
- Production routing effect: The project’s April 2026 case study of a Tesla deployment reports a 3x increase in output tokens per second and a 2x reduction in time to first token after enabling prefix-cache-aware routing. The model was Llama 3.1 70B, running on four AMD MI300X GPUs using KServe, llm-d, and vLLM. The comparison is a single deployment with routing off and on.
Analysis
llm-d turns a set of proven serving techniques into a shared standard governed by the CNCF, and each founder gains something different from that:
- IBM and Red Hat extend the enterprise AI platform story they sell, and Red Hat already offers AI platform products, including OpenShift AI and Red Hat AI Inference.
- Google Cloud integrates llm-d 1.0 into Google Kubernetes Engine across its TPU and NVIDIA GPU portfolios.
- CoreWeave offers llm-d on the CoreWeave Kubernetes Service alongside NVIDIA Dynamo, letting customers choose.
- NVIDIA brings llm-d to Vera Rubin and contributes components from the open-source Dynamo.
- AMD adds support for Instinct accelerators.
Practitioner Impact
The primary users are platform engineering and MLOps teams at enterprises, neoclouds and model providers that serve open-weight models on Kubernetes.
A stable v1 API gives these teams a version to standardize on, and flow control lets one fleet serve interactive users and bursty agents without separate clusters.
Competitive Impact
llm-d competes with other open-source inference control planes and with commercial model APIs. The following table summarizes the main alternatives.
| Alternative | Model and Approach | Where It Leads | Comparison to llm-d |
| NVIDIA Dynamo | Open-source orchestration layer above vLLM, SGLang and TensorRT-LLM with disaggregated serving, KV-aware routing, multi-tier cache and autoscaling | NVIDIA-led development with TensorRT-LLM support | Covers the same technique set. The 1.0 materials include no comparison with Dynamo, and NVIDIA contributes to both projects. |
| AIBrix | Open-source Kubernetes building blocks with an LLM gateway, autoscaling, LoRA management and KV-cache offloading | Adapter management and autoscaling components | Overlaps on gateway routing and cache offload. llm-d adds flow control, DisaggregatedSet and five founding organizations. |
| KServe | Kubernetes-native inference platform with scale to zero, canary rollouts and pluggable runtimes | Multi-framework serving and rollout features | Complements llm-d. The Tesla case study runs KServe with llm-d and vLLM. |
| vLLM or SGLang alone | Single-engine serving with built-in batching and prefix caching | Lowest operational complexity | llm-d adds fleet-level routing, disaggregation and tenancy. Single-replica deployments gain little from it. |
| Commercial model APIs | Per-token managed endpoints from proprietary and open-weight model hosts | No infrastructure to operate and access to proprietary frontier models | The 5 to 10 times cost claim targets this option. Savings depend on GPU utilization and on open-weight model quality for the task. |
llm-d’s differentiation is strongest in three areas:
- It supports accelerators from NVIDIA, AMD, Google, and Intel under a single API, a breadth that vendor-led stacks have less incentive to match.
- CNCF governance, with five named founders, gives buyers a neutral home.
- Production adopters with named workloads demonstrate use beyond the founders’ own labs.
Final Thoughts
llm-d 1.0 does what a first stable release should do: it freezes the configuration API and adds the operational features that distinguish a performance demonstration from a service a team can run on Monday morning.
The founders have integrated routing, caching, disaggregation and tenancy into a single coordinated stack, and the early adopter list shows production use at AWS, DigitalOcean, Google Cloud, JetBrains, Oracle Cloud, Mistral, Prime Intellect, and Roblox.
As with all benchmark claims, organizations evaluating inference infrastructure should treat the 5 to 10 times cost claim as a hypothesis to test against their own traffic. The outcome depends on context reuse, GPU utilization, and the quality of open-weight models for their tasks.
Organizations running open-weight models on Kubernetes at meaningful scale have a clear reason to pilot llm-d now, because the API is stable and the project’s direction is shared by vendors selling competing hardware.
With a stable API, CNCF governance, and production adopters, llm-d 1.0 sets the open baseline for inference cost and operations against which every competing serving stack, proprietary or open, will now be measured.
