[Question] 如何正常运行LLaDA1.5，现在出来的都是MASK

[WuNein]

Required prerequisites

• I have read the documentation https://tilelang.com.
• I have searched the Issue Tracker that this hasn't already been reported. (comment there if it has.)

Questions

from diffulex import Diffulex, SamplingParams
from transformers import AutoTokenizer



# Initialize the Diffulex engine
model_path = "/root/models/LLaDA-1.5"
llm = Diffulex(
    model_path,
    model_name="llada",  # or "dream", "llada", etc.
    tensor_parallel_size=1,
    data_parallel_size=1,
    gpu_memory_utilization=0.7,
    max_model_len=2048,
    decoding_strategy="d2f",  # or "d2f", "fast_dllm"
    mask_token_id=126336,  # model-specific mask token ID
enforce_eager = True,
kv_cache_layout= "unified",
use_lora = False,
    add_block_threshold=0.1,
    semi_complete_threshold=0.9,
decoding_threshold=0.95,
block_size=32,
buffer_size=4,
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Set sampling parameters
sampling_params = SamplingParams(
    temperature=0.0,
    max_tokens=128,
)



# Prepare prompts
prompts = [
    "Question: What is the capital of France? Answer:",
    "Question: Explain quantum computing in simple terms. Answer:",
]

# Generate responses
outputs = llm.generate(prompts, sampling_params)
#print(outputs.convert_to_text(tokenizer))
print(outputs.to_benchmark_format())
# Process results
#for output in outputs:
#    print(f"Generated text: {output['text']}")
#    print(f"Number of diffusion steps: {output['n_diff_steps']}")
#    print(f"Token IDs: {output['token_ids']}")

如果按照Readme来[rank0]: TypeError: 'GenerationOutputs' object is not iterable

print(outputs.to_benchmark_format()) 可以输出内容，但是

Diffulex Generating: 100%|███████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:11<00:00,  5.89s/it, tpf=1.09tok/step, ttft=1.12s, tpot=0.04s, ptps=114.47tok/s, dtps=52.07tok/s, tps=47.28tok/s]
[08:57:57] INFO     --------------------------------
           INFO     Generation Outputs Summary:
           INFO     --------------------------------
           INFO     Total Tokens: 512 toks
           INFO     Total NFEs: 254 nfes (steps)
           INFO     Total Time: 11.718297283630818 sec
           INFO     TPF: 1.09 tok/step
           INFO     TTFT: 1.12 sec
           INFO     TPOT: 0.04 sec
           INFO     Throughput: 47.28 tok/sec
           INFO     Prefill Throughput: 114.47 tok/sec
           INFO     Decode Throughput: 52.07 tok/sec
           INFO     --------------------------------
None
[{'text': '', 'full_text': '<|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|>', 'token_ids': [126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463], 'nfe': 254}, {'text': '', 'full_text': '<|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|>', 'token_ids': [126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463], 'nfe': 254}]

结果都是MASK的空值，我已经根据benchmark yaml改了llada的配置参数，请问能否提供llada的运行配置。

[WuNein]

[rank0]: │ /root/models/Diffulex/diffulex/model/llada.py:195 in __init__                                                                                                                                                                            │
[rank0]: │                                                                                                                                                                                                                                          │
[rank0]: │   192 │   ) -> None:                                                                                                                                                                                                                     │
[rank0]: │   193 │   │   super().__init__()                                                                                                                                                                                                         │
[rank0]: │   194 │   │   self.config = config                                                                                                                                                                                                       │
[rank0]: │ ❱ 195 │   │   self.transformer = nn.Moduledict(                                                                                                                                                                                          │
[rank0]: │   196 │   │   │   dict(                                                                                                                                                                                                                  │
[rank0]: │   197 │   │   │   │   wte=VocabParallelEmbedding(config.embedding_size or config.vocab_size, config.d_model),                                                                                                                            │
[rank0]: │   198 │   │   │   │   emb_drop=nn.Dropout(config.embedding_dropout),        
[rank0]: AttributeError: module 'torch.nn' has no attribute 'Moduledict'

你们是不是自己都没跑过代码，这个Moduledict, ModuleDict大小写写错了跑不通，自己没测试过吗？

[drewjin]

@WuNein 感谢您的关注与反馈。

目前我们尚未正式支持 LLaDA 1.5。主要原因是，Diffulex 当前针对的是 block-wise dLLM 路线，暂不支持 full-attention dLLM。现阶段代码中仅保留了对 D2F-LLaDA-Instruct 的早期适配，但这一部分后续没有作为主测模型持续维护，因此目前不能保证其在最新代码上的可用性与稳定性。

就我们已经做过的完整测试而言，目前覆盖最充分的是 SDAR 系列：

• buffer size：1 ~ 4
• block size：4 / 8 / 16 / 32
• 对应 page size：4 / 8 / 16 / 32

我们优先完善 SDAR 支持，是因为该系列覆盖了多种块大小组合，能够帮助我们更系统地验证和优化整个运行时状态机，尤其是：

• 不同 page size 的兼容性
• page size ≠ block size 时的行为正确性

由于系统仍在快速开发和重构过程中，我们暂时无法保证每次优化后都对所有模型完成完整回归测试；相关 CI 和自动化测试流程也仍在持续完善中。这一点确实会带来部分旧适配路径失效或行为变化的问题，我们也在持续改进。

接下来我们会尽快重新测试并整理以下模型的支持状态：

• D2F-LLaDA
• D2F-Dream
• Fast-dLLM-v2

同时，我们也正在推进对以下模型的支持：

• LLaDA 2.0
• LLaDA 2.1
• LLaDA-DMax

感谢大家的理解与关注，我们后续会在文档和测试覆盖范围上补充得更清楚，尽量减少当前支持状态与实际可用性之间的信息落差。

[drewjin]

@WuNein BTW，若您需要跑 LLaDA 1.5，可以尝试用 D2F 的代码来做蒸馏训练，训出 Block Causal 能力，然后用 D2F 策略来跑。

[WuNein]

@drewjin 好的，不过麻烦readme里面 https://github.com/SJTU-DENG-Lab/Diffulex#:~:text=model_name%3D%22fast_dllm_v2%22%2C%20%20%23%20or%20%22dream%22%2C%20%22llada%22%2C%20etc. 写一下
普遍有Non-Block不支持的情况

[drewjin]

@WuNein 再次感谢反馈，我们目前正在加速完善 文档、CI、SKILL 等的支持工作。