inclusionAI/Ling-3.0-flash-FP8

Overview

inclusionAI/Ling-3.0-flash-FP8 is the serialized block-FP8 checkpoint of Ling-3.0-flash. It uses the BailingMoeV3ForCausalLM architecture with a hybrid MLA/KDA attention stack, 512 routed experts (8 active per token), one shared expert, and a native multi-token prediction head.

Prerequisites

• vLLM: 0.26.0 or newer with Ling 3 block-FP8 support
• Validated hardware: 2x NVIDIA H200 (default); 4x H200 with expert parallelism is also supported
• Weights: serialized block FP8
• Compute dtype: BF16
• Context length: 131,072 tokens

Launching the Server

NCCL_DEBUG=WARN vllm serve inclusionAI/Ling-3.0-flash-FP8 \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.9 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3

The validated configuration keeps CUDA graphs enabled; --enforce-eager is not required.

To use TP4 with expert parallelism instead, set --tensor-parallel-size 4 and add --enable-expert-parallel.

To enable the native MTP head, add:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Thinking Mode

Thinking is selected per request through the chat template:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="inclusionAI/Ling-3.0-flash-FP8",
    messages=[{"role": "user", "content": "Solve the problem step by step."}],
    temperature=0.0,
    max_tokens=128000,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content)
print(response.choices[0].message.content)

Validation

Both the default TP2 path and the TP4+EP path were validated on H200.

References

• Ling-3.0-flash-FP8 on Hugging Face