RuntimeError: Engine core initialization failed.

[Chennu]

vllm.log

(EngineCore pid=1057) INFO 04-16 02:33:00 [core.py:107] Initializing a V1 LLM engine (v0.19.1rc1.dev241+g4d042ed85.d20260413) with config: model='Qwen/Qwen2.5-1.5B-Instruct-AWQ', speculative_config=None, tokenizer='Qwen/Qwen2.5-1.5B-Instruct-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir='/root/.ainode/models', load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=awq_marlin, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen2.5-1.5B-Instruct-AWQ, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto')
(EngineCore pid=1057) INFO 04-16 02:33:00 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.6.162:49119 backend=nccl
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] EngineCore failed to start.
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] Traceback (most recent call last):
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1084, in run_engine_core
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] return func(*args, **kwargs)
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 850, in init
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] super().init(
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 116, in init
(EngineCore pid=1057) ERROR 04-16 02:33:00 [core.py:1110] self.model_executor = executor_class(vllm_config)
(EngineCore pid=1057) ERROR 04-16 02:33

[webdevtodayjason]

Root cause identified and fixed. Fixed in commit a1869cf.

---

What happened

You used Qwen/Qwen2.5-1.5B-Instruct-AWQ (an AWQ-quantized model). vLLM automatically upgrades AWQ → awq_marlin — a fused Marlin kernel that's faster but requires separate CUDA kernel builds per GPU architecture.

The eugr spark-vllm-docker base image (which AINode runs on) is patched and compiled for sm_12.1 (GB10/Blackwell), but the awq_marlin Marlin CUDA kernels were not compiled for sm_12.1 in that build. Result: EngineCore initialization fails with a RuntimeError.

---

Fix applied

docker_engine.py now detects AWQ models by name and forces --quantization awq to pin the plain AWQ implementation, preventing the auto-upgrade to awq_marlin:

elif self.config.model and "awq" in self.config.model.lower():
    quant = "awq"   # prevent vLLM upgrading to awq_marlin

---

To get the fix today

Option A — switch to the non-AWQ model (instant, works now):

ainode config --model Qwen/Qwen2.5-1.5B-Instruct
sudo systemctl restart ainode

The plain (bf16) model works perfectly on GB10's 128 GB unified memory.

Option B — wait for 0.4.4 image (coming soon):

ainode update

The fix will be in the next image build and AWQ models will work correctly.

---

Great bug report — exact logs, hardware context, clear reproduction steps. This would have affected every GB10 user who tried an AWQ model.

[Chennu]

Applied Temp Fix:

(APIServer pid=61) WARNING 04-16 02:50:12 [envs.py:1796**] Unknown vLLM environment variable detected: VLLM_BASE_DIR**
(APIServer pid=61) INFO 04-16 02:50:21 [model.py:554] Resolved architecture: Qwen2ForCausalLM
(APIServer pid=61) INFO 04-16 02:50:21 [model.py:1684] Using max model len 32768
(APIServer pid=61) INFO 04-16 02:50:21 [vllm.py:819] Asynchronous scheduling is enabled.
(APIServer pid=61) INFO 04-16 02:50:21 [kernel.py:199] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
...
...
..
..
(APIServer pid=61) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

vllm.log.04152026.0955PMCST.log

[webdevtodayjason]

Thanks for the updated log. You're running the old image — v0.4.4 with the AWQ fix isn't published yet (building now).

A few things to sort out the immediate issue:

1. What model are you trying to load?

Please run this and share the output:

cat ~/.ainode/config.json | grep model

2. The VLLM_BASE_DIR warning is harmless — ignore that one.

3. The log cuts off before the root cause. The full error is earlier in the file with ERROR or Traceback. Can you share the full vllm.log from the beginning (or search for Traceback in it)?

4. Immediate workarounds while we wait for 0.4.4:

If it's an AWQ model:

ainode config --model Qwen/Qwen2.5-1.5B-Instruct
sudo systemctl restart ainode

If it's a non-AWQ model that's still failing, it may be an OOM or a different kernel issue. Check:

nvidia-smi   # confirm GPU is visible
docker exec ainode cat /root/.ainode/logs/vllm.log | grep -A5 "Traceback"

5. v0.4.4 is building now. Once it's published, run:

ainode update

That permanently fixes the AWQ/Marlin issue. ETA ~30 minutes.

[Chennu]

cat ~/.ainode/config.json | grep model

"model": "Qwen/Qwen2.5-1.5B-Instruct",
"models_dir": "/root/.ainode/models",
"max_model_len": null,

Observation: Models are not under root user. ".ainode" not even existed under /root

Here is the structure: Disabled root user. All the installation done under a dedicated user

tree -d ~/.ainode
/home/user/.ainode
├── datasets
├── logs
├── models
│   ├── google--gemma-4-26b-a4b-it:free
│   ├── hub
│   │   ├── models--Qwen--Qwen2.5-1.5B-Instruct
│   │   │   ├── blobs
│   │   │   ├── refs
│   │   │   └── snapshots
│   │   │   └── 989aa7980e4cf806f80c7fef2b1adb7bc71aa306
│   │   └── models--Qwen--Qwen2.5-1.5B-Instruct-AWQ
│   │   ├── blobs
│   │   ├── refs
│   │   └── snapshots
│   │   └── 3ecffa0ceb27851800f45519bab9c457a04405e1
│   ├── microsoft--Phi-3.5-vision-instruct
│   └── Qwen--Qwen2.5-1.5B-Instruct-AWQ
└── training

[Chennu]

Initial Issue fixed by updating config.json file.

Fix1: Updated models Directory instead of "/root/.ainode/models" : "models_dir": "/home/user/.ainode/models"

Root Cause for Engine core initialization failure: looking for the cluster interface:
"cluster_interface": "eno1"

Fix2: Updated Cluster Information: Verified all the available NIC's : ip -br addr
"eno1" is not available : Changed to : "cluster_interface": "enp1s0f0np0"
Updated on both nodes: cluster_role, distributed_mode, ssh_user, peer_ips(only in master),

Node 2 Started without any failure:
journalctl -u ainode -f
Apr 16 17:43:24 node-2 docker[38522]: │ │
Apr 16 17:43:24 node-2 docker[38522]: ╰─────────────────────────── Powered by argentos.ai ───────────────────────────╯
Apr 16 17:43:24 node-2 docker[38522]: GPU NVIDIA GB10 | 122 GB (unified memory)
Apr 16 17:43:24 node-2 docker[38522]: CUDA 13.2 | Driver 580.142
Apr 16 17:43:24 node-2 docker[38522]: Compute SM 12.1
Apr 16 17:43:24 node-2 docker[38522]: Model Qwen/Qwen2.5-1.5B-Instruct
Apr 16 17:43:24 node-2 docker[38522]: API http://localhost:8000/v1
Apr 16 17:43:24 node-2 docker[38522]: Web http://localhost:3000
Apr 16 17:43:24 node-2 docker[38522]: Node d0d93405
Apr 16 17:43:24 node-2 docker[38522]: Member mode — no local inference engine. Awaiting work from the cluster head.

Node 1: Started Web

journalctl -u ainode -f
Apr 16 17:44:11 riai360-brain-1 docker[109268]: ╰─────────────────────────── Powered by argentos.ai ───────────────────────────╯
Apr 16 17:44:11 node-1 docker[109268]: GPU NVIDIA GB10 | 122 GB (unified memory)
Apr 16 17:44:11 node-1 docker[109268]: CUDA 13.2 | Driver 580.142
Apr 16 17:44:11 node-1 docker[109268]: Compute SM 12.1
Apr 16 17:44:11 node-1 docker[109268]: Model Qwen/Qwen2.5-1.5B-Instruct
Apr 16 17:44:11 node-1 docker[109268]: API http://localhost:8000/v1
Apr 16 17:44:11 node-1 docker[109268]: Web http://localhost:3000
Apr 16 17:44:11 node-1 docker[109268]: Node adfe7a03
Apr 16 17:44:11 node-1 docker[109268]: Engine starting in background — web UI is ready now.
Apr 16 17:44:11 node-1 docker[109268]: Open http://localhost:3000 to get started.

But distribution log showing error: Please find the attached file:

Issue 1 : Error response from daemon: No such container: vllm_node
Issue 2: Starting Worker Node on 192.168.100.11...
mkdir: cannot create directory ‘/root’: Permission denied

THINKING : Can we completely remove the dependency of root directory : Instead of "/root" can we use "~/"

distributed.log.04162026.0545PMCST.log