#USA #Supermicro #San Jose #NVIDIA #IBM

Introduction

On November 5, 2026, Supermicro, Inc. (NASDAQ: SMCI) announced the launch of a groundbreaking Key-Value (KV) cache solution in partnership with IBM and NVIDIA. This innovative solution is designed to deliver substantial performance enhancements in AI inference, crucial for businesses increasingly relying on AI technologies.

What is KV Cache?

The KV cache solution integrates Supermicro’s petascale storage servers with IBM's Storage Scale software and NVIDIA's Dynamo inference software, all supported by NVIDIA Spectrum-X Ethernet switches. This integration aims to reduce the time it takes for AI models to generate responses, an essential feature given the rise of agentic AI workflows and long-context AI applications in various industries.

Performance Improvements

During internal tests, it was revealed that the KV cache solution significantly decreased the Time-to-First-Token (TTFT) when handling long context windows, compared to traditional systems lacking KV cache support. By allowing a higher utilization of GPU resources, greater throughput, and a more scalable infrastructure, customers can expect substantial performance boosts in their AI inference tasks.

Strategic Benefits

Charles Liang, the President and CEO of Supermicro, emphasized that as AI workloads transition from training to inference, enterprises need to remove bottlenecks that hinder GPU efficiency. He notes, "Our solution with IBM and NVIDIA clearly demonstrates these performance advantages."

The performance of the KV cache becomes especially relevant during the intricate phases of AI inference. These phases typically consist of a prefill stage, where input text is converted into tokens for processing by a Large Language Model (LLM), followed by decoding. By storing the intermediate KV states generated during this phase in a persistent KV cache, the system can quickly leverage these states for future requests, thus saving time and computational resources.

Scalability and Reliability

The KV cache solution not only enhances latency but also allows companies to scale their dedicated storage capacity independent of their compute resources. This results in a more robust and resilient architecture capable of delivering faster, responsive AI experiences. Key highlights of the solution include:

• - Joint Validation: The integrated solution has undergone extensive cluster-level testing for performance evaluation by Supermicro, IBM, and NVIDIA.
• - Drastic Reductions in TTFT: Initial tests showed that utilizing a permanent KV cache reduced TTFT by up to 56 times under specific conditions, including processing prompts of up to 130,000 tokens.
• - Higher GPU Utilization Rates: By maintaining frequently-used context outside the GPU's memory, the system can effectively reduce idle GPU time, thus increasing throughput and accommodating numerous simultaneous requests.
• - Robust AI Operations: The permanent KV cache offers high availability for stored inference states while allowing shared access across multiple GPU nodes and clusters, which is essential for robust and scalable AI services.

Accelerated Deployment Services

Enabled by Supermicro's factory rack integration and on-site deployment services, the time taken to bring this solution into operation can be shortened from weeks to days, potentially transforming the operational paradigms for companies adopting AI solutions.

Sam Werner, General Manager at IBM Storage, stated, "Businesses need fundamentally better ways to scale AI inference. Together, IBM, Supermicro, and NVIDIA deliver a powerful and robust KV cache solution that maximizes GPU investments, provides scalable storage, and significantly enhances infrastructure efficiency."

NVIDIA also supported this initiative, with Shruti Koparkar, AI Inference Product Lead, highlighting that NVIDIA Dynamo aids in coordinating distributed inference, allowing businesses to maximize throughput from their GPUs.

Conclusion

As the demand for AI continues to escalate, particularly with the rise of agentic AI applications and workflows involving long contexts, the KV cache solution from Supermicro, in collaboration with IBM and NVIDIA, sets a new standard for AI inference performance. To learn more about these innovative solutions, interested readers can access resources from Supermicro and IBM.

About Supermicro

Founded in San Jose, California, Supermicro is recognized as a leader in optimized, application-specific IT solutions. The company focuses on delivering cutting-edge technology for enterprise, cloud, AI, and 5G/edge, aiming to improve TCO while reducing environmental impact. Supermicro’s Server Building Block Solutions® provide tailored options for a wide array of IT needs globally.