Aarch64 GB200 container perf regression from vllm 0.17 to vllm 0.18

[kaixih]

System Info

vLLM 0.18 Performance Regression on GB200 (aarch64)

Summary

Compared vLLM 0.17 (stock) vs vLLM 0.18 (pip install vllm==0.18.0) on GB200 nodes.
Training reward curves (critic/rewards/mean) match closely between the two versions,
confirming correctness. However, vLLM 0.18 shows a ~10% throughput regression.

Observations

• perf/throughput: vLLM 0.18 is ~10% slower than vLLM 0.17.
• perf/mfu/actor (training): nearly identical between versions — no regression in the training phase.
• perf/mfu/actor_infer (rollout): vLLM 0.18 is noticeably lower than vLLM 0.17 — clear gap.

Conclusion

The regression is isolated to the rollout/inference phase and does not affect training.
This points to a performance issue introduced in the vLLM 0.18 rollout backend on aarch64/GB200.

Screenshots

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Reproduction Steps

Environment

• Hardware: NVIDIA GB200 NVL4 (aarch64), 4 GPUs per node
• verl commit: 516657fa
• vLLM 0.17: stock (as shipped in docker/Dockerfile.stable.vllm)
• vLLM 0.18: pip install vllm==0.18.0 on top of the same base image

1. Build the container

One image is built and used for both runs. The vLLM 0.18 variant upgrades vLLM inside the container at runtime (step 3).

# Clone verl at the tested commit
git clone https://github.com/volcengine/verl.git
cd verl
git checkout 516657fa

# Build the aarch64 image (runs on a GB200 node or via buildx)
docker build -f docker/Dockerfile.stable.vllm \
    -t verl:vllm-arm64 .

2. Prepare the dataset

docker run --rm \
    -v /path/to/models:/models \
    -v $(pwd):/workspace/verl \
    -w /workspace/verl \
    verl:vllm-arm64 \
    python3 examples/data_preprocess/gsm8k.py \
        --local_save_dir /models/verl-datasets/gsm8k

Dataset is saved to /models/verl-datasets/gsm8k/{train,test}.parquet.

3. Run training

vLLM 0.17 (baseline) — use the image as-is:

docker run --rm --gpus all --privileged --shm-size=128g --network=host \
    -v /path/to/models:/models \
    -v $(pwd):/workspace/verl \
    -w /workspace/verl \
    verl:vllm-arm64 \
    python3 -m verl.trainer.main_ppo \
        algorithm.adv_estimator=grpo \
        data.train_files=/models/verl-datasets/gsm8k/train.parquet \
        data.val_files=/models/verl-datasets/gsm8k/test.parquet \
        data.train_batch_size=1024 \
        data.max_prompt_length=512 \
        data.max_response_length=1024 \
        data.filter_overlong_prompts=True \
        data.truncation='error' \
        actor_rollout_ref.model.path=deepseek-ai/deepseek-llm-7b-chat \
        actor_rollout_ref.actor.optim.lr=1e-6 \
        actor_rollout_ref.model.use_remove_padding=False \
        actor_rollout_ref.actor.ppo_mini_batch_size=512 \
        actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=128 \
        actor_rollout_ref.actor.use_kl_loss=True \
        actor_rollout_ref.actor.kl_loss_coef=0.001 \
        actor_rollout_ref.actor.kl_loss_type=low_var_kl \
        actor_rollout_ref.actor.entropy_coeff=0 \
        actor_rollout_ref.model.enable_gradient_checkpointing=True \
        actor_rollout_ref.actor.fsdp_config.param_offload=False \
        actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
        actor_rollout_ref.actor.fsdp_config.model_dtype=bfloat16 \
        actor_rollout_ref.actor.fsdp_config.forward_prefetch=True \
        actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=80 \
        actor_rollout_ref.rollout.name=vllm \
        actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
        actor_rollout_ref.rollout.gpu_memory_utilization=0.5 \
        actor_rollout_ref.rollout.enforce_eager=False \
        actor_rollout_ref.rollout.enable_chunked_prefill=True \
        actor_rollout_ref.rollout.n=5 \
        actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=160 \
        actor_rollout_ref.ref.fsdp_config.param_offload=True \
        actor_rollout_ref.ref.fsdp_config.forward_prefetch=True \
        algorithm.use_kl_in_reward=False \
        trainer.critic_warmup=0 \
        trainer.logger='["console","wandb"]' \
        trainer.project_name='verl_grpo_example_gsm8k' \
        trainer.experiment_name=vllm_vllm-0.17 \
        trainer.n_gpus_per_node=4 \
        +ray_kwargs.ray_init.num_gpus=4 \
        trainer.nnodes=1 \
        trainer.save_freq=20 \
        trainer.test_freq=1000 \
        trainer.total_epochs=15

vLLM 0.18 — launch the same image, upgrade vLLM, then run training:

docker run --rm --gpus all --privileged --shm-size=128g --network=host \
    -v /path/to/models:/models \
    -v $(pwd):/workspace/verl \
    -w /workspace/verl \
    verl:vllm-arm64 \
    bash -c "pip install vllm==0.18.0 && python3 -m verl.trainer.main_ppo \
        ... (same args as above) \
        trainer.experiment_name=vllm_vllm-0.18"

Expected behavior

We will first profile the rollout backend to root cause the regression. @borisfom

[kaixih]

cc. @jqwang2373 @wuxibin89 @ETOgaosion

[Kirrito-k423]

Thank you for the detailed performance analysis with clear reproduction steps and metrics comparison.

The ~10% throughput regression on aarch64 GB200 with vLLM 0.18 is a valuable finding. Your observation that the regression is isolated to the rollout/inference phase (perf/mfu/actor_infer) while training (perf/mfu/actor) remains unchanged provides a clear direction for investigation.

Potential causes for vLLM 0.18 rollout regression on GB200/aarch64:

1. Attention backend changes: vLLM 0.18 may have changed default attention implementation or kernel optimizations that are less optimized for aarch64.

2. CUDA graph compatibility: GB200 (aarch64) might have different CUDA graph behavior compared to x86_64. Check if enforce_eager=True vs enforce_eager=False makes a difference.

3. Chunked prefill: You're using enable_chunked_prefill=True. vLLM 0.18 may have changed chunked prefill scheduling that affects GB200.

4. FlashInfer backend: vLLM 0.18 may use different FlashInfer kernels. Check flashinfer version compatibility.

Debugging suggestions:

1. Profile rollout: Use py-spy or NVIDIA Nsight Systems to profile the rollout phase:

   py-spy record -o profile.svg --pid <rollout_pid>

2. Compare attention kernels: Check if vLLM 0.18 uses a different attention backend:

   # In vLLM 0.17 vs 0.18
   from vllm.attention import get_attn_backend
   print(get_attn_backend())

3. **Test with `enforce_eager=True``: This disables CUDA graphs which might behave differently on aarch64.

4. Check memory allocation: GB200 has unified memory architecture. vLLM 0.18's memory allocator changes might affect this.

Related: The VeRL team is aware of aarch64 specific issues. PR #5961 addresses vLLM compatibility issues for NPU/Ascend which may have overlapping concerns.

Could you share:

• The exact attention backend being used in both versions?
• Whether enforce_eager=True improves the regression?
• Memory usage comparison between vLLM 0.17 and 0.18?

This finding should be reported to the vLLM team as well, since it appears to be a vLLM backend issue rather than VeRL-specific.

[borisfom]

I think it's important to isolate vllm-0.18 upgrade from any potential dependency updates which may have caused regression.

I started with verl-vllm-0.17dev3 container, did pip install from the top of PR branch.  With 2048 batch size, I actually got even better throughput :
       perf/time_per_step:115.70092637099151 - perf/throughput:3744.0668245903494
 2.  Upgraded vllm package to vllm-0.18.1: pip install ‘vllm<0.19.0’ --upgrade
      This brought in quite a few major upgrades:
      Successfully installed flashinfer-python-0.6.6 huggingface-hub-0.36.2 numpy-2.2.6 nvidia-cudnn-frontend-1.18.0 transformers-4.57.6 vllm-0.18.1 xgrammar-0.1.33
     I did observe ~10% perf drop after that:
     perf/time_per_step:103.20219689499936 - perf/throughput:3423.438508382367
Note this is on B200, not GB200.

 3.  Downgraded vllm back to 0.17.1, leaving most of the rest intact, no change in perf :
      pip install ‘vllm<0.18.0’
       Successfully installed flashinfer-python-0.6.4 vllm-0.17.1 xgrammar-0.1.29
      perf/time_per_step:102.33666555100353 - perf/throughput:3340.706365204094

So it seems, the 10% drop we observe, is due to transformers-4.57.6 ? Note the version in the container was higher (Successfully uninstalled transformers-5.3.0).

Now, pip install “transformers>=5.3.0”, (could not find 5.3.0 version any longer) and the performance is restored:
Successfully installed hf-xet-1.4.3 huggingface-hub-1.10.1 transformers-5.5.3
perf/time_per_step:115.88194165000459 - perf/throughput:3744.46608178646

Please let me know if you can reproduce the same on GB200 and if we should close the GH issue.