llm-d 1.0 Open Distributed Inference for Production AI
The llm-d project recently released version 1.0, the first stable release of an open-source, Kubernetes-native framework for distributing large language model inference across fleets of accelerators. llm-d sits above inference engines such as vLLM and SGLang, adding routing, cache management, and scheduling across model servers.
NAND Insider Newsletter: March 30 2025
Every week NAND Research puts out a newsletter for our industry customers taking a look at what’s driving the week, and what happened last week that caught our attention. Below is a excerpt from this week’s, March 30, 2025.