LLM Serving & Inference Engines

High-throughput LLM serving engine with PagedAttention; the industry default for open-model inference.

Fast serving framework with RadixAttention prefix caching, strong on DeepSeek-style MoE and structured output.

NVIDIA's library for optimized LLM inference on NVIDIA GPUs with in-flight batching and FP8/FP4 kernels.

Cross-platform inference accelerator for ONNX models on CPU, GPU, NPU, mobile and the web.

Datacenter-scale inference orchestration with disaggregated prefill/decode over vLLM, SGLang or TensorRT-LLM.

Kubernetes-native distributed inference stack on vLLM with cache-aware routing, backed by Red Hat and Google.

Distributed compute framework for scaling Python AI workloads: data processing, training, tuning and serving.

NVIDIA's multi-framework model server for PyTorch, ONNX, TensorRT and Python backends at scale.

Toolkit for compressing, deploying and serving LLMs with the high-performance TurboMind engine.

Hugging Face's LLM server; archived in 2026 after maintenance mode, with vLLM and SGLang recommended instead.

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer