ONNX Runtime

vLLM

vllm-project/vllm

Popular

High-throughput LLM serving engine with PagedAttention; the industry default for open-model inference.

SGLang

sgl-project/sglang

Trending

Fast serving framework with RadixAttention prefix caching, strong on DeepSeek-style MoE and structured output.

TensorRT-LLM

NVIDIA/TensorRT-LLM

NVIDIA's library for optimized LLM inference on NVIDIA GPUs with in-flight batching and FP8/FP4 kernels.

NVIDIA Dynamo

ai-dynamo/dynamo

New

Datacenter-scale inference orchestration with disaggregated prefill/decode over vLLM, SGLang or TensorRT-LLM.

llm-d

llm-d/llm-d

New

Kubernetes-native distributed inference stack on vLLM with cache-aware routing, backed by Red Hat and Google.

Ray

ray-project/ray

Popular

Distributed compute framework for scaling Python AI workloads: data processing, training, tuning and serving.

Triton Inference Server

triton-inference-server/server

NVIDIA's multi-framework model server for PyTorch, ONNX, TensorRT and Python backends at scale.

LMDeploy

InternLM/lmdeploy

Toolkit for compressing, deploying and serving LLMs with the high-performance TurboMind engine.