An Empirical Study of LLM Serving in Confidential GPUs

Abstract

Large language models (LLMs) are increasingly deployed to serve user prompts, raising significant concerns regarding the security and privacy of user data. Confidential computing (CC) offers a practical approach for securing workloads within untrusted cloud environments. While CPU technologies such as Intel TDX and AMD SEV-SNP enable confidential virtual machines (CVMs) that isolate and attest software execution, recent advancements in NVIDIA GPUs have extended these security guarantees to accelerators. LLM serving frameworks, notably vLLM and SGLang, have emerged as the de facto infrastructure for large-scale deployment, orchestrating complex operations including batching, scheduling, KV-cache management, and kernel execution across heterogeneous CPU-GPU architectures. Although CC in GPU is known to impose overhead through encrypted CPU–GPU communication, its actual impact on these end-to-end LLM serving has not been evaluated in depth, considering the scheduling and optimizations in the serving frameworks, motivating a systematic characterization of how CC reshapes end-to-end LLM serving performance. This paper presents a comprehensive performance study of two popular serving frameworks, vLLM and SGLang, across machines utilizing AMD SEV and Intel TDX. Our evaluation demonstrates that offline throughput overhead varies notably by framework: vLLM exhibits a 1–13% overhead across models, whereas SGLang frequently exceeds 20%, with a range of 3–66%. In online scenarios, queuing delays further amplify CC penalties. Even when large batch sizes provide sufficient throughput, the CC overhead for end-to-end latency is often 2× larger than CC overhead observed for offline throughput. Furthermore, optimizations designed for standard environments do not always yield performance gains under CC. While kernel fusion via torch.compile improves throughput for both frameworks in non-confidential settings, SGLang's static-graph and max-autotune design result in a performance degradation under CC. Additionally, chunked prefill increases CC overhead due to the requirement for additional encrypted communication rounds. Similarly, KV-cache swapping can increase end-to-end latency by as much as 50% compared to recomputation, and model offloading incurs a throughput overhead exceeding 150% under CC.

Related Papers