Nvidia proposes an AI memory full-stack collaborative design solution, with the Vera Rubin platform increasing per-GPU memory capacity by only about 3%, but inference throughput improves by up to 30 times.

NVIDIA states that as AI model sizes expand, context windows lengthen, and agent applications become more widespread, GPU memory pressure continues to rise, with HBM supply tightening and prices increasing. NVIDIA proposes improving memory usage efficiency through collaborative design of GPU, CPU, networking, storage, and software, instead of simply increasing GPU memory capacity. Specifically, NVIDIA compresses model weights and KV Cache using the NVFP4 low-precision format, offloads temporarily inactive contexts to lower-cost storage layers, and frees up HBM space through multi-GPU distributed deployment. NVIDIA revealed that in the SemiAnalysis AgentX test, the GB300 NVL72 achieved up to 10 times the unit cost inference throughput of the H200 platform under certain workloads; compared to the GB300 NVL72, the Vera Rubin NVL72 achieved up to a 30-fold increase in unit cost inference throughput, while the HBM capacity per GPU only increased by approximately 3%.