Vision Encoder (ViT) CUDA Graphs

The CUDA Graphs infrastructure in vLLM primarily targets the decoder (language model) forward pass. vLLM also supports capturing the encoder (vision transformer) forward pass as CUDA Graphs, independently from the decoder. This is based on Pull Request #35963.

For two-tower vision encoders (e.g., DeepSeek-OCR's SAM + CLIP with dynamic tiling), a dual-path graph mode captures two independent sets of CUDA graphs — one for the global image path and one for the local patch path — enabling independent budget selection and partial eager fallback per path. This is based on Pull Request #43586.

Note

Encoder CUDA Graphs are orthogonal to decoder CUDA Graphs — both can be enabled simultaneously. Encoder graphs capture the vision encoder execution (e.g., ViT in Qwen3-VL), while decoder graphs capture the language model execution as described in the CUDA Graphs design document.

Supported Models

Architecture	Models	CG for Image	CG for Video	Multi-Path Graph
DeepseekOCRForCausalLM	DeepSeek-OCR	✅︎	❌︎	✅︎
DeepseekVLV2ForCausalLM	DeepSeek-VL2	✅︎	❌︎	❌︎
DeepseekV41ForCausalLM	DeepSeek-V4.1-Flash	✅︎	❌︎	❌︎
Ernie4_5_VLMoeForConditionalGeneration	ERNIE-4.5-VL	✅︎	❌︎	❌︎
Gemma3ForConditionalGeneration	Gemma3	✅︎	❌︎	❌︎
Glm4vForConditionalGeneration	GLM-4.1V, GLM-4.6V-Flash	✅︎	✅︎	❌︎
Gemma4ForConditionalGeneration	Gemma-4	✅︎	✅︎	❌︎
InternVLChatModel	InternVL3.5, InternVL3, InternVL2.5, InternVL2	✅︎	✅︎	❌︎
KimiVLForConditionalGeneration	Kimi-VL	✅︎	❌︎	❌︎
KimiK25ForConditionalGeneration	Kimi-K2.5, Kimi-K2.6	✅︎	❌︎	❌︎
KimiK3ForConditionalGeneration	Kimi-K3	✅︎	❌︎	❌︎
Llama4ForConditionalGeneration	Llama 4	✅︎	❌︎	❌︎
Mistral3ForConditionalGeneration	Mistral 3	✅︎	❌︎	❌︎
PixtralForConditionalGeneration	Pixtral, Mistral-format Mistral 3	✅︎	❌︎	❌︎
Qwen2VLForConditionalGeneration	Qwen2-VL	✅︎	✅︎	❌︎
Qwen2_5_VLForConditionalGeneration	Qwen2.5-VL	✅︎	✅︎	❌︎
Qwen3VLForConditionalGeneration	Qwen3-VL	✅︎	✅︎	❌︎
Qwen3_5ForConditionalGeneration	Qwen3.5, Qwen3.6	✅︎	✅︎	❌︎
Qwen3_5MoeForConditionalGeneration	Qwen3.5-MoE, Qwen3.6-MoE	✅︎	✅︎	❌︎
Step3VLForConditionalGeneration	Step3-VL	✅︎	❌︎	✅︎
MiniCPMV	MiniCPMV2.5	✅︎	❌︎	❌︎
MiniCPMV	MiniCPMV2.6, MiniCPMV4.0	✅︎	✅︎	❌︎

Note

Encoder CUDA Graph has currently been tested with --mm-encoder-attn-backend=FLASH_ATTN and --mm-encoder-attn-backend=FLASHINFER on Blackwell GPUs. For Qwen2-VL and Qwen2.5-VL only FA2 and FA3 has been tested. Encoder CUDA Graph has also been tested with AMD MI350X (gfx950) used --mm-encoder-attn-backend=FLASH_ATTN (the ROCm default).

Motivation

Vision encoder inference incurs CUDA kernel launch overhead on the host side. The overhead is more significant when the batch size is small or image size is small.

Encoder CUDA Graphs eliminate this overhead by pre-capturing the full encoder forward pass at multiple token budget levels during model initialization, then replaying the appropriate graph at runtime.

For two-tower vision encoders such as DeepSeek-OCR (SAM + CLIP with dynamic tiling), the global image path and local patch path have independent token profiles (272 tokens per global image vs. 100 tokens per local patch). Capturing a single monolithic graph for both paths would significantly increase padding. The dual-path graph mode captures each path as a separate set of budgets, allowing the manager to select and replay a budget for each path independently.

Design

The encoder CUDA Graph system uses a budget-based capture/replay strategy, managed by EncoderCudaGraphManager. The system contains the following core components:

• EncoderCudaGraphManager: orchestrates capture, replay, dispatch, and data-parallel execution for encoder CUDA Graphs.
• SupportsEncoderCudaGraph: a runtime-checkable protocol that models implement to opt-in to encoder CUDA Graphs.
• EncoderItemSpec: describes a single encoder input item (image or video) with its input size and output token count.
• BudgetGraphMetadata: holds the captured CUDA Graph and its associated I/O buffers for a single token budget level.

Budget-Based Graph Capture

Multiple CUDA Graphs are pre-captured at different token budget levels (e.g., [2048, 4096, 8192, 13824]). Each budget defines a fixed token capacity, and all budgets share the same maximum batch size (number of images). The BudgetGraphMetadata for each level stores the graph along with pre-allocated input, metadata, and output buffers:

@dataclass
class BudgetGraphMetadata:
    token_budget: int
    max_batch_size: int
    max_frames_per_batch: int
    graph: torch.cuda.CUDAGraph
    input_buffers: dict[str, torch.Tensor]  # e.g. pixel_values, embeddings, seq metadata
    output_buffer: torch.Tensor      # encoder hidden states

Budgets are auto-generated over the model-provided range of get_encoder_cudagraph_budget_range(): quarter-octave steps up to 1024 tokens, steps of 128 up to 2560 tokens, then 1/8-octave steps rounded up to a multiple of 128 (e.g., [64, 80, ..., 1024, 1152, ..., 2560, 2816, 3072, 3456, 3840, 4224, 4608, 5120], 36 levels for the range [64, 16384] with the default maximum). The model-provided minimum and maximum are always included even when they do not fall on a grid point. The largest generated budget is encoder_cudagraph_max_token_budget in CompilationConfig (default 5120; None captures up to the model's largest budget), like the LLM's maximum capture size. The default balances graph memory against replay coverage; the point at which eager becomes faster depends on the model, batch shape, and hardware. Workloads that benefit from larger graphs can raise the maximum. Each batch replays at most one graph, the smallest budget that holds it; a larger batch runs eagerly in one forward. Budgets can also be explicitly specified by the user via encoder_cudagraph_token_budgets in CompilationConfig, which overrides both.

The dense default table is affordable for every model because capture buffers are shared across budgets automatically, with no model opt-in: budgets are captured largest-first, the first capture of each (path, capture-axis) group provides one set of max-size input buffers and one max-size output buffer, and every later capture of the group records its graph into prefix views of them (the same pattern as decoder input_ids[:num_tokens]). At capture time the manager checks each probed buffer for prefix compatibility (same trailing dims and dtype, no more rows than the shared buffer) and copies it into the shared buffer; buffers that do not fit this pattern, including 0-dim scalars, keep their own per-budget allocation, so models need no code changes. Buffer memory therefore stays at ~1× the largest budget instead of scaling with the sum of all budgets.

Because later captures overwrite a shared buffer, its capture-time contents may belong to another budget. Replay must therefore provide every shared buffer (the manager raises if one is missing), and a custom padding_logics entry must write its whole destination buffer.

Each entry in EncoderCudaGraphConfig.paths defines an independently captured encoder path. A path can provide its own minimum token budget and opt into zero-token batches; the manager generates and stores a separate budget graph set for every configured path.

Multi-Path Graph Capture

EncoderCudaGraphConfig.paths maps path names to EncoderCudaGraphPathConfig capture policies. For example, DeepSeek-OCR configures a global image path and a local patch path, which are captured independently under budget_graphs["global"] and budget_graphs["local"].

Budget generation. Each path gets a separate budget list. For DeepSeek-OCR:

• the global path — multiples of its fixed item size (min_token_budget): every count of 1–16 items, then 1/8-octave steps in item count (e.g., [272, 544, 816, ...] for DeepSeek-OCR).
• the local path — likewise multiples of the local patch size (e.g., [0, 100, 200, 300, ...] for DeepSeek-OCR). A budget of 0 is included when allow_zero_tokens=True to handle images with no local patches (images ≤ 640×640 that produce only global features).

Both lists are capped at the same max_budget.

Multi-path dispatch. Each EncoderItemSpec provides path_output_tokens, mapping each path to its contribution for that item. The whole batch is dispatched once per path: each non-empty path replays its smallest budget that holds the path's total tokens.

Partial graph fallback. For each non-empty path, the manager replays the smallest fitting graph or runs only that path eagerly when no graph fits. Paths with zero tokens are skipped; a 0-budget graph is never captured or replayed.

Buffer keys per path. Global and local paths use different buffer keys. For DeepSeek-OCR, the global path uses pixel_values (full images, shape [B, 3, 1280, 1280]) while the local path uses images_crop (patches, shape [P, 3, 1024, 1024]). The manager iterates over each captured graph's own input_buffers.keys() rather than a shared buffer_keys list, so both paths can use different buffers.

Post-processing. The postprocess_encoder_output method receives an outputs dictionary keyed by path name. The model is responsible for assembling global and local features into the final per-image embedding. For DeepSeek-OCR, it reshapes the global output into [B, 272, n_embed], reshapes the local output into [P, 100, n_embed], assembles patch grids with newline tokens, and concatenating [patches_grid, global, view_separator] for each image.

Note

The dual-path design enables partial CUDA graph coverage — one path can hit while the other falls back to eager. This avoids wasted compute on zero-padded patch buffers for untiled images and avoids graph invalidation caused by variable crop_shape per image.

One Replay per Batch

A graph only saves eager's host launch time. Eager latency is the larger of that launch time and the GPU time, while a replay costs the GPU time of its whole captured shape, padding included. A graph therefore helps only while eager is launch-bound, and splitting a batch into several replays only adds padding and fixed costs.

When a batch of images arrives, the manager replays the smallest budget that holds the whole batch, once per path. A batch that exceeds the largest captured budget, or has more images than the maximum batch size, runs as one eager forward.

For each graph replay:

1. Call prepare_encoder_cudagraph_replay_buffers() to compute buffer values (including pixel_values and precomputed metadata) from actual batch inputs.
2. Zero the pre-allocated input_buffers, then slice-copy the replay values into them.
3. Replay the CUDA Graph.
4. Clone outputs from output_buffer (cloning is necessary since the buffer is reused across replays of every budget on the path).

ViT DP Mode Support

When mm_encoder_tp_mode="data", the manager distributes images across TP ranks using load-balanced assignment via get_load_balance_assignment, executes locally on each rank, then gathers results back in the original order via tensor_model_parallel_all_gather.

Video Inference Support

Following Pull Request #35963 (ViT full CUDA graph support for image inference), Pull Request #38061 extends the encoder CUDA graph framework to support video inference for Qwen3-VL. Previously, the CUDA graph capture/replay path only handled image inputs (pixel_values + image_grid_thw). Video inputs use different keys (pixel_values_videos + video_grid_thw) and require larger cu_seqlens buffers because each video item contributes multiple frames (T attention sequences). This PR generalizes the protocol and manager to handle both modalities through a single shared graph manager.

Note

Video CUDA graphs are automatically disabled when video token pruning (EVS or VidCom2) is enabled, since pruning makes the token count data-dependent and incompatible with CUDA graph capture.

Mixed inputs (image+video) per prompt are also supported now.

Model Integration via SupportsEncoderCudaGraph

Models opt-in to encoder CUDA Graphs by implementing the SupportsEncoderCudaGraph protocol. This protocol encapsulates all model-specific logic so that the manager remains model-agnostic. The protocol defines the following methods:

• get_encoder_cudagraph_config() — returns static configuration (supported modalities, buffer keys, output hidden size, padding logics, max frames per video).
• get_encoder_cudagraph_budget_range(vllm_config) — returns (min_budget, max_budget) for auto-inference of token budgets.
• get_encoder_cudagraph_item_specs(mm_kwargs) — returns list[EncoderItemSpec] describing each item with its input size, total output token count (output_tokens), and per-path token counts (path_output_tokens) for multi-path models.
• select_encoder_cudagraph_items(mm_kwargs, indices) — extracts a sub-batch of items by index, used during dispatch and DP sharding.
• prepare_encoder_cudagraph_capture_inputs(..., path="default") — creates dummy inputs for graph capture. The path parameter ("global" or "local") tells the model which path to generate dummy inputs for. Returns EncoderCudaGraphCaptureInputs with a single values: dict[str, torch.Tensor] that contains all buffers to be recorded into the graph.
• prepare_encoder_cudagraph_replay_buffers(mm_kwargs, max_batch_size, max_frames_per_batch, path="default") — computes buffer values from actual batch inputs. The path parameter selects which modality keys to extract from mm_kwargs. Returns EncoderCudaGraphReplayBuffers with a values dict whose keys match the captured graph's input_buffers.keys().
• encoder_cudagraph_forward(inputs: dict[str, torch.Tensor], path="default") — forward pass accepting only fixed-shaped input tensors (the captured values dict). Called during both capture and replay. The path parameter dispatches to the correct encoder sub-module (e.g., global vs. local path for DeepSeek-OCR).
• encoder_eager_forward(mm_kwargs, path="default") — fallback eager forward when no graph fits. When path is "global" or "local", runs only that encoder path without graph capture.
• postprocess_encoder_output(outputs, ...) — post-process encoder outputs keyed by path name, enabling multi-path models to assemble their path-specific features into the final per-item embedding. With clone=True, the per-item results must not be views of outputs, since the next replay overwrites the shared output buffer.

Note

The SupportsEncoderCudaGraph protocol is designed to be model-agnostic. New vision encoder models can opt-in by implementing the protocol methods without modifying the manager.

Configuration

Five fields in CompilationConfig control encoder CUDA Graphs:

• cudagraph_mm_encoder (bool, default False) — enable CUDA Graph capture for multimodal encoder. When enabled, captures the full encoder forward as a CUDA Graph for each token budget level.
• encoder_cudagraph_token_budgets (list[int], default []) — token budget levels for capture. If empty (default), auto-inferred from model architecture on the subdivided grid (see above). User-provided values override auto-inference.
• encoder_cudagraph_max_token_budget (int | None, default 5120) — largest auto-generated token budget. Must be positive or None. A batch with more tokens runs eagerly. None captures up to the model's largest budget. Ignored when encoder_cudagraph_token_budgets is set.
• encoder_cudagraph_max_vision_items_per_batch (int, default 0) — maximum number of images/videos per batch during capture. If 0 (default), auto-inferred as max_budget // min_budget. A batch with more items runs eagerly in one forward.
• encoder_cudagraph_max_frames_per_batch (int, default None) — maximum number of video frames per batch during capture. If None (default), auto-inferred as encoder_cudagraph_max_vision_items_per_batch * max_frames_per_video (max_frames_per_video is a model-specific value from EncoderCudaGraphConfig, computed by get_max_frames_per_video() on the model). If we limit the video count per prompt to 0, it will also be set to 0 (i.e., fall back to image-only mode).

Multi-path mode is configured at the model level through EncoderCudaGraphConfig.paths. Each EncoderCudaGraphPathConfig can set a path-specific minimum budget and whether zero-token batches are allowed. The manager automatically generates separate budget lists and uses the same execution loop for single- and multi-path models.

Usage Guide

Image Inference

Enable encoder CUDA Graphs via compilation_config:

vllm serve Qwen/Qwen3-VL-32B \
  --compilation-config '{"cudagraph_mm_encoder": true}'

For Llama 4 (image only):

vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --limit-mm-per-prompt '{"image": 1}' \
  --compilation-config '{"cudagraph_mm_encoder": true}'

With explicit budgets:

vllm serve Qwen/Qwen3-VL-32B \
  --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [2048, 4096, 8192, 13824], "encoder_cudagraph_max_vision_items_per_batch": 8}'

Python example:

import vllm

compilation_config = {
    "cudagraph_mm_encoder": True,
    # Optional: override auto-inferred budgets
    # "encoder_cudagraph_token_budgets": [2048, 4096, 8192, 13824],
    # "encoder_cudagraph_max_vision_items_per_batch": 8,
}

model = vllm.LLM(
    model="Qwen/Qwen3-VL-32B",
    compilation_config=compilation_config,
)

The manager tracks hit/miss statistics and logs them periodically. A "hit" means an image was processed via CUDA Graph replay; a "miss" means eager fallback (image exceeded all budgets).

Video Inference

Enable encoder CUDA Graphs via compilation_config:

vllm serve Qwen/Qwen3-VL-32B \
  --compilation-config '{"cudagraph_mm_encoder": true}'

With explicit budgets:

vllm serve Qwen/Qwen3-VL-32B \
  --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [2048, 4096, 8192, 13824], "encoder_cudagraph_max_vision_items_per_batch": 8, "encoder_cudagraph_max_frames_per_batch": 64}'

Python example:

import vllm

compilation_config = {
    "cudagraph_mm_encoder": True,
    # Optional: override auto-inferred budgets
    # "encoder_cudagraph_token_budgets": [2048, 4096, 8192, 13824],
    # "encoder_cudagraph_max_vision_items_per_batch": 8,
    # "encoder_cudagraph_max_frames_per_batch": 64,
}

model = vllm.LLM(
    model="Qwen/Qwen3-VL-32B",
    compilation_config=compilation_config,
)

Benchmark Results

The following benchmarks were run on Blackwell GPUs (GB200) using vllm bench mm-processor. See #35963 for full details.

Single GPU (1x GB200)

Model: Qwen/Qwen3-VL-30B-A3B-Instruct, dataset: lmarena-ai/VisionArena-Chat (3000 prompts, 300 warmup), max_model_len=32768.

Backend	Mean latency improvement	P99 latency improvement
FLASH_ATTN	+11.8% (5.13→4.52ms)	+31.6% (9.16→6.26ms)
FLASHINFER	+19.6% (5.42→4.36ms)	+40.3% (10.87→6.49ms)

To reproduce:

vllm bench mm-processor \
  --model Qwen/Qwen3-VL-30B-A3B-Instruct \
  --dataset-name hf --dataset-path lmarena-ai/VisionArena-Chat \
  --num-prompts 3000 --num-warmups 300 \
  --max-model-len 32768 --seed 42 \
  --mm-encoder-attn-backend FLASH_ATTN \
  --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [512, 1024, 1536, 2048, 2560, 3072, 3584, 4096, 4864], "encoder_cudagraph_max_vision_items_per_batch": 8}'

Multi-GPU (4x GB200, TP=4, DP=4)

Model: Qwen/Qwen3-VL-32B-Instruct, dataset: random-mm (1000 prompts, 200 warmup, 20 images/request at 336x336), max_model_len=8192.

Backend	Mean latency improvement	P99 latency improvement
FLASH_ATTN	+18.4% (28.39→23.16ms)	+14.0% (238.78→205.28ms)
FLASHINFER	+44.4% (23.24→12.91ms)	+84.9% (172.41→26.05ms)

To reproduce:

vllm bench mm-processor \
  --model Qwen/Qwen3-VL-32B-Instruct \
  --dataset-name random-mm \
  --random-mm-base-items-per-request 20 \
  --random-mm-num-mm-items-range-ratio 0.0 \
  --random-mm-bucket-config '{"(336,336,1)": 1.0}' \
  --num-prompts 1000 --num-warmups 200 \
  --max-model-len 8192 --seed 42 \
  --mm-encoder-attn-backend FLASHINFER \
  --tensor-parallel-size 4 --mm-encoder-tp-mode data \
  --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [512, 1024, 1536, 2048, 2560, 3072, 3584, 4096, 4864], "encoder_cudagraph_max_vision_items_per_batch": 8}'

Note

Find more details about benchmarks on GPUs (A100) for video inference at #38061.