Is Qwen3 pretraining architectural features fully supported now?

[tjoymeed]

Is Qwen3 pretraining architectural features fully supported now?

Hi Team,

Thanks a lot for your excellent work!

Is Qwen3 pretraining architectural features fully supported now?

Could you please provide an architectural feature list with support status?

Thanks again!

[cuichenx]

Hi, we support general pretraining (without reasoning or long context extension), as well as full and parameter-efficient finetuning.

[tjoymeed]

What about MoE pretraining?

[cuichenx]

All Qwen 3 variants are supported, including 6 dense models and 2 MoE models.

[tjoymeed]

The Qwen 3 pretraining actually has 3 stages:

1. pretraining for 4096 context length;
2. pretraining for enhancing reasoning;
3. pretraining for extending to long context.

So NeMo currently supports 1 (all architectural features) not 2 or 3?

Thanks!

[cuichenx]

Yes, what you described is correct

[tjoymeed]

Okay, when will stage 3 pretraining be fully supported? Thanks!

[tjoymeed]

All Qwen 3 variants are supported, including 6 dense models and 2 MoE models.

Could you please tell me where to find the recipe for Qwen3 MoE pretraining?

[cuichenx]

We're working on better long context training support right now, but I don't have any near term ETA to share with you at this time.

Qwen3 MoE recipes can be found here:
https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/recipes/qwen3_30b_a3b.py#L55
https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/recipes/qwen3_235b_a22b.py#L55

[tjoymeed]

Great! Thanks a lot!
Could you please enlighten me a bit on this:
how to modify the recipe for qwen3_30b_a3b into pretraining from scratch a qwen3_12b-a1b?
Thanks again!

[tjoymeed]

Okay, when will stage 3 pretraining be fully supported? Thanks!

Is it coming out soon? Anxiously awaiting it...

PAI-Megaton-Patch has the stage 3. Maybe there is a way to integrate the two together?

[cuichenx]

Great! Thanks a lot! Could you please enlighten me a bit on this: how to modify the recipe for qwen3_30b_a3b into pretraining from scratch a qwen3_12b-a1b? Thanks again!

There is no definitive answer to this, but you can check out the difference between 235b_a22b and 30b_a3b -- these parameters are downsized:
num_layers, hidden_size, num_attention_heads, moe_ffn_hidden_size.

Of course it's not guaranteed that downsizing these further to create 12b-a1b would create a model that still converges. Deep learning is an art :)

Okay, when will stage 3 pretraining be fully supported? Thanks!

Is it coming out soon? Anxiously awaiting it...

PAI-Megaton-Patch has the stage 3. Maybe there is a way to integrate the two together?

Sorry, I don't have any intel on when it will come out. I'll let you know once I know more.

[BoxiangW]

Hi, we are targeting support YaRN and other long context feature next release, which is NeMo 25.09. Currently it is not supported