[REQUEST] Qwen3 Architecture Support

[Thireus]

Problem

Qwen3 architecture is currently not supported.

Solution

Qwen3MoeForCausalLM and Qwen3ForCausalLM - https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f

Alternatives

No response

Explanation

Qwen3 architecture support is currently missing from exllamav2 (and exllamav3). Bringing architecture support would enable us to start converting the Qwen3 models to EXL2 (and EXL3).

Examples

No response

Additional context

No response

Acknowledgements

• I have looked for similar requests before submitting this one.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will make my requests politely.

[emzaedu]

The MOE Qwen3-30B-A3B variant was somewhat disappointing; overall, the model turned out to be even worse than the 2.5 14B. However, support for Qwen3ForCausalLM in exllama2 would be a real gift - before exllama3 eventually takes its place.

[pivtienduc]

The MOE Qwen3-30B-A3B variant was somewhat disappointing; overall, the model turned out to be even worse than the 2.5 14B. However, support for Qwen3ForCausalLM in exllama2 would be a real gift - before exllama3 eventually takes its place.

Do you try Qwen3-32B, I heard it better than Deepseek v3 and I though at last I can have a good model to deploy on my PC.

[emzaedu]

Do you try Qwen3-32B, I heard it better than Deepseek v3 and I though at last I can have a good model to deploy on my PC.

he dense 32B model is significantly smarter than the MOE 30B.

[hidoba]

The MOE Qwen3-30B-A3B variant was somewhat disappointing; overall, the model turned out to be even worse than the 2.5 14B. However, support for Qwen3ForCausalLM in exllama2 would be a real gift - before exllama3 eventually takes its place.

How does qwen3-14B compare to qwen2.5-14B?

[turboderp]

I added it to the dev branch now. I'm assuming Qwen3 (the non MoE versions at least) are just Qwen2 models without the attention bias but with Q/K norms instead, and it seems to work based on some limited testing.

[CyberTimon]

Thank you! I hope we get Qwen MOE support soon as I love how fast it is (and I bet it will be even faster and more optimised with ExLlamaV2 (or V3))

[emzaedu]

I keep running into issues when generating output with the 32B version of 4bpw — endless characters like ****, #####, HelloHelloHello, and similar. No such problems with the 14B version :(
tabbyAPI under windows 11

etc...

[Nrgte]

Same I just get gibberish as output from Qwen3.

[pivtienduc]

Yeah me too, Qwen3 of mine giving unrecognize text like this:

I also run it on TabbyAPI

Maybe something with the convertion

[turboderp]

This is a template issue. Working on it.

[emzaedu]

I also run it on TabbyAPI

Maybe something with the convertion

Judging by the message in your terminal, it indicates that the architecture is unrecognized. You should switch to the dev branch of the ExLlama2 - this will help eliminate the junk during response generation.
However, the issue with the template will still remain.

[Nrgte]

Yeah me too, Qwen3 of mine giving unrecognize text like this:

Getting exactly to the tea the same kind of output.

[pivtienduc]

I also run it on TabbyAPI

Maybe something with the convertion

Judging by the message in your terminal, it indicates that the architecture is unrecognized. You should switch to the dev branch of the ExLlama2 - this will help eliminate the junk during response generation. However, the issue with the template will still remain.

No, I already compile and build wheel on dev branch in order to load the model but so far it can't generate anything yet

[emzaedu]

No, I already compile and build wheel on dev branch in order to load the model but so far it can't generate anything yet

Then it shouldn't be showing up.

[pivtienduc]

No, I already compile and build wheel on dev branch in order to load the model but so far it can't generate anything yet

Then it shouldn't be showing up.

Sorry I post the wrong screenshot, this screenshot is captured when I haven't rebuild the wheel

Now after update the latest dev branch I got Qwen 32B working.