Skip to content

vllm.benchmarks.forward.main

Benchmark the model forward of exact batches as one-shot CUDA graphs.

Each case is a list of requests (query_len, kv_len, num_draft_tokens) per DP rank. The bench builds the batch without the scheduler, runs it through the real execute_model (input prep, attention metadata, DP sync), and times the target forward, target sampling (rejection sampling for verify rows) and configured drafter captured as a CUDA graph. Requests use the model's default sampling parameters (--generation-config). CPU step overhead isn't measured.

Single GPU: vllm bench forward --model M --case decode=32*1@8192. Multi-GPU: torchrun --nproc-per-node=N -m vllm.entrypoints.cli.main bench forward ..., one process per GPU.

_sampling_params(runner)

The model's default sampling parameters, as LLM uses them.

Source code in vllm/benchmarks/forward/main.py 
151
152
153
154
155
156
def _sampling_params(runner) -> SamplingParams:
    """The model's default sampling parameters, as `LLM` uses them."""
    defaults = runner.model_config.get_diff_sampling_param()
    if defaults:
        return SamplingParams.from_optional(**defaults)
    return SamplingParams()