Local LLMs & open models

Stories from vLLM Blog. Show all stories · All sources

• vvLLM Blog2026-10-15✓ Read

  Announcing Day 0 Support for NVIDIA Nemotron 3.5 Super VL on vLLM

  Run NVIDIA Nemotron 3.5 Super VL on vLLM for multimodal agent workflows with image, document, and video understanding, OpenAI-compatible serving, and multi-token prediction.

  v
• vvLLM Blog2026-10-14✓ Read

  Finish RL Training Faster: Bitwise-Consistent, High-Performance, Agent-Driven

  We make vLLM rollout and the training forward give bitwise identical logprobs on DeepSeek-V4-Flash and Nemotron-3.5-Lightning-30B-A3B (NVFP4), keep 91% of rollout throughput on DeepSeek-V4-Flash, and open-source the agent skill.

  v
• vvLLM Blog2026-10-14✓ Read

  Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM

  A push-based NIXL connector for vLLM disaggregated serving: the prefill instance writes KV directly into decode memory over RDMA, taking the router relay, decode-side allocation, and transfer setup off the TTFT critical path.

  v
• vvLLM Blog2026-10-14✓ Read

  Adaptive Verification Without a Confidence Head

  An online-fitted acceptance estimator that lets every draft-model speculator use Adaptive Verification, by predicting the target's acceptance probability from the drafter's own logits and refitting continuously during serving.

  v
• vvLLM Blog2026-10-12✓ Read

  Improve TTFT for Multi-Turn Agentic Workloads with Bi-Directional KV Transfer in vLLM

  Bi-directional KV transfer lets vLLM prefill nodes reuse previously computed KV from decode nodes on later conversation turns, cutting redundant prefill recompute and reducing TTFT by up to 3.3x on long multi-turn prompts.

  v
• vvLLM Blog2026-10-09✓ Read

  vLLM Support for NVIDIA Vera Rubin NVL72: 7.8x Throughput over GB200 NVL72

  vLLM now runs on NVIDIA Vera Rubin NVL72 with day-0 model support, Rubin-tuned FlashInfer kernels, and locality-aware MoE, delivering 7.8x the per-GPU throughput of GB200 NVL72 on AgentX.

  v
• vvLLM Blog2026-10-07✓ Read

  DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0

  Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions.

  v
• vvLLM Blog2026-09-29✓ Read

  Taking vLLM Apart: A Practical Guide to Disaggregated Serving

  What disaggregated serving actually buys you, how to run it end to end in vLLM today with prefill/decode plus the new GPU-less frontend and the things we're still working on.

  v
• vvLLM Blog2026-09-24✓ Read

  Watermarking in vLLM

  How vLLM implements distribution-preserving Gumbel-max text watermarking with efficient GPU kernels, statistical detection, speculative decoding, and repeated-context safeguards.

  v
• vvLLM Blog2026-09-22✓ Read

  Announcing vllm-metal: Concurrent Serving on Apple Silicon

  vllm-metal brings vLLM's paged, continuously batched serving stack to Apple Silicon, with flatter TTFT under concurrent agent load, batched MTP, and automatic M5 prefill acceleration.

  v
• vvLLM Blog2026-09-21✓ Read

  PD Serving of Qwen3.8-2.4T

  How vLLM reaches 5K throughput and 180 interactivity on Qwen3.8-2.4T with GB300 NVL72 PD serving and how to reproduce results yourself.

  v
• vvLLM Blog2026-09-15✓ Read

  vLLM x Novita AI: Chord, Faster INT4 MoE for Kimi K2.x. Up to 1.3x on H200, 2.15x on Untuned B300

  Novita AI has open-sourced Chord, a high-performance W4A16 MoE CUDA kernel for Kimi K2.x serving shapes, with a Humming-compatible indexed path and grouped SM90 operators.

  v
• vvLLM Blog2026-09-15✓ Read

  How we trained the fastest DSpark for Kimi-K3 using GB300 NVL72

  How Speculators and Mooncake enabled multi-node DSpark training for Kimi K3.

  v

Know a source we’re missing? Tell us about it