Nvidia, Georgia Tech, Stanford Boost LLM Inference Throughput
Researchers at Georgia Tech, Nvidia, and Stanford have developed BOOST, a runtime system that improves large language model (LLM) inference throughput by allowing concurrent access to both GPU host memory and HBM.
According to SemiEngineering, BOOST is a novel runtime system that enables concurrent and proportional access to both GPU memory tiers, extracting the combined bandwidth of host memory and HBM for LLM inference without kernel changes. The system integrates into vLLM and is evaluated on a Grace Hopper system, showing significant improvements in throughput.
BOOST uses kernel access patterns to make page allocation and runtime data management wave-aware. For static model weights, it applies modulo-based page placement to eliminate access-ratio variance. For dynamically provisioned attention key-value pairs, it makes the free KV page pool wave-aware, improving efficiency.
Compared to the previous generation, BOOST improves Time-per-Output-Token (TPOT) by 4.3% over HBM-only serving at iso-batch size. In high-throughput serving, it improves throughput by 31% on average, outperforming prefetching by 15%.
The source does not provide specific details on future versions of BOOST or its potential integration into other systems beyond the Grace Hopper evaluation.
Engineers should watch for further developments in memory management techniques that enhance LLM inference efficiency, particularly as model sizes and inference demands continue to grow.
Desk take
The introduction of concurrent access to both GPU memory tiers represents a significant advancement in managing large-scale inference workloads, potentially reducing inference times and improving system efficiency.
This innovation in memory management can significantly enhance the performance of large-scale language models, reducing inference times and improving overall system efficiency.
Source dispatch
Researchers at Georgia Tech, Nvidia Research, and Stanford University published a technical paper titled “BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference.” Abstract Excerpt: “This paper presents BOOST, the first runtime system that provides concurrent and proportional access to both GPU memory tiers, extracting the combined bandwidth of host memory and... » read more The post Concurrent HBM And Host Memory Access Improves LLM Inference Throughput (Georgia Tech, Nvidia, Stanford) appeared first on Semiconductor Engineering .
