EAGLE Draft Models¶

The following code configures vLLM to use speculative decoding where proposals are generated by an EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) based draft model. A more detailed example for offline mode, including how to extract request level acceptance rate, can be found in examples/features/speculative_decoding/spec_decode_offline.py

Eagle Drafter Example¶

from vllm import LLM, SamplingParams

prompts = ["The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    tensor_parallel_size=4,
    speculative_config={
        "model": "yuhuili/EAGLE-LLaMA3-Instruct-8B",
        "draft_tensor_parallel_size": 1,
        "num_speculative_tokens": 2,
        "method": "eagle",
    },
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Eagle3 Drafter Example¶

from vllm import LLM, SamplingParams

prompts = ["The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    tensor_parallel_size=2,
    speculative_config={
        "model": "RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3",
        "draft_tensor_parallel_size": 2,
        "num_speculative_tokens": 2,
        "method": "eagle3",
    },
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Pipeline Parallelism¶

EAGLE3, DFlash, and DSpark support pipeline parallelism on Model Runner V2 with compatible target models. Auxiliary hidden states are collected from the target model and forwarded across pipeline stages to the final stage, where the draft model runs.

Supported targets include Llama, Qwen2, dense Qwen3, Qwen3-VL (dense and MoE), gpt-oss, Gemma 4, and MiniMax-M3. DeepSeek-V4 and Kimi-K3 are also supported on NVIDIA GPUs. Other targets are rejected at startup with <architecture> does not support <method> with pipeline parallelism.

For example, to run gpt-oss with an EAGLE3 draft model across two pipeline stages:

vllm serve openai/gpt-oss-20b \
    --pipeline-parallel-size 2 \
    --speculative-config '{
        "method": "eagle3",
        "model": "RedHatAI/gpt-oss-20b-speculator.eagle3",
        "num_speculative_tokens": 3
    }'

Pre-Trained Eagle Draft Models¶

A variety of EAGLE draft models are available on the Hugging Face hub:

Warning

If you are using vllm<0.7.0, please use this script to convert the speculative model and specify "model": "path/to/modified/eagle/model" in speculative_config.