Stuck on missing prompt_template

[draker42]

When I try to talk to an exllamav2 driven model in Open WebUI, through TabbyAPI, I get:

2025-03-20 18:00:24.781 ERROR: if model.container.prompt_template is None:
2025-03-20 18:00:24.781 ERROR: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-20 18:00:24.781 ERROR: AttributeError: 'NoneType' object has no attribute 'prompt_template'

I've configured my IP address in TabbyAPI's config.yml and confirmed that works. I don't know what to put in for prompt_template:. I've tried 'chat_template' (which comes from the tokenizer_config.json in Qwen2.5-VL-7B-Instruct-exl2), and I've tried the names of some of the files that come with git clone https://github.com/theroyallab/llm-prompt-templates templates. I also tried Llama-3.2-1B-exl2. Clearly I'm stuck on step "4. Run TabbyAPI (and set your config to use a new template)"!!

Background:

• I have 2x nvidia Tesla M10 and 1x P1000. (old, I know. Maxwells are CUDA 5.0). Maybe it's not possible, but it feels so close!
• I have Debian 12 and I installed nvidia-tesla-* drivers. nvidia-smi says NVIDIA-SMI 470.256.02 Driver Version: 570.124.06 CUDA Version: 12.8 and it detects all (nine) GPUs.
• I have ollama running nicely with open-webui, and it shares VRAM between GPUS (slowly) but I want to try exllamav2 as recommended here: https://www.ahmadosman.com/blog/do-not-use-llama-cpp-or-ollama-on-multi-gpus-setups-use-vllm-or-exllamav2/
• I followed the guide for installing exllamav2 using pip into a virtual environment with python3.11.2.
• I installed pytorch using their web selector and typed pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118. This is CUDA 11.8, not 12.8. Does this matter?
• I had problems with half/__hfma type errors during the final pip install ., and I fixed that by using export TORCH_CUDA_ARCH_LIST="7.5" as found at How to solve the __hfma2 problem docker error oobabooga/text-generation-webui#2128 (Should I try 5.0?)
• I am able to connect open-webui to TabbyAPI with my API key, and it detects the available models. I am able to ask questions, but on submit I get 500: Internal Server error, and a stack trace from tabbyAPI's start.sh which ends with the quote above.
• (EDIT) I installed the models using ./start.sh download turboderp/Llama-3.2-1B-exl2-instructand ./start.sh download turboderp/Qwen2.5-VL-7B-Instruct-exl2 --revision 4.0bpw

I'm a relative newbie with LLMs and all of this, and I appreciate my setup is not conventional. Forgive me if I've missed the
FAQ, but I've not been able to find the solution to address this (what seems like a) tiny issue!

Thanks for any help,
-D.

[frenzybiscuit]

This is not tabbyapi's support page.

That being said, that field can be left blank (I dont use it).

Delete your config.yml, run ./start.sh again. It will regenerate.

I've tried the names of some of the files that come with

You have to actually download the files and place them in the correct folder...

[draker42]

Hello,
Thanks for the reply.
Sorry, it's a bit unclear to me the boundaries of exllama vs. tabbyAPI. However I've come to realize my problems are deeper than a prompt_template. The error actually says 'model.container' is a NoneType. Seems tabbyAPI is unable to get exllamav2 to load the model? But there is no error message until runtime...
I suppose to test exllamav2 without tabbyAPI I need to write a little python.
Meantime, ollama continues to work very well, if a bit slowly, and I haven't managed to get any other engine functional. I will test on some better hardware.
PS: yes of course I cloned the prompt templates into the templates/ directory!
Thank you,
-D.

[frenzybiscuit]

please post your config file (use a gist on github).

Which model are you trying to use?

[draker42]

Hi,
config.yml was regenerated as you suggested (awesome, btw) and all I do is change IP address to '0.0.0.0'.
Qwen2.5-VL-7B-Instruct-exl2 and Llama-3.2-1B-exl2 models, just for testing.
I don't have time to work on it right now -- tonight I'll work on it some more and report back.
Thanks,
-D.

[draker42]

Hi,
Tested on another system. Ran out of VRAM :( But learned some things:

2025-03-23 22:12:26.867 WARNING:  An unsupported GPU is found in this configuration. Switching to compatibility mode. 
2025-03-23 22:12:26.867 WARNING:  This **disables parallel batching** and features that rely on it (ex. CFG). 
2025-03-23 22:12:26.867 WARNING:  To disable compatability mode, **all GPUs must be ampere (30 series) or newer**. AMD GPUs are not supported.

Essentially, exllamav2 will not work better (or at all) than ollama on my old hardware. Understood.

Next:

2025-03-23 22:12:26.869 INFO:     Attempting to load a prompt template if present.
2025-03-23 22:12:26.869 WARNING:  TemplateLoadError: Model JSON path 
"/home/daniel/work/tabbyAPI/models/Llama-3.2-1B-exl2/chat_template.json" not found.
2025-03-23 22:12:26.885 WARNING:  TemplateLoadError: Could not find a value from chat_template key in the passed JSON. Check the
tokenizer config?
2025-03-23 22:12:26.887 WARNING:  TemplateLoadError: Could not find template from model name.
2025-03-23 22:12:26.888 WARNING:  Chat completions are disabled because a prompt template wasn't provided or auto-detected.
2025-03-23 22:12:27.915 INFO:     Loading model: /home/daniel/work/tabbyAPI/models/Llama-3.2-1B-exl2

It seems this little model is missing a prompt_template altogether. I did not get this error with the Qwen model. This helped also:

"/home/daniel/work/tabbyAPI/models/TinyLlama-1B-exl2/chat_template.json" not found.

I can just put the name of a template from the templates directory (e.g. 'chatml') in the 'prompt_template: ' line of config.yml

With respect to Open-WebUI and my original error -- I can type any garbage model name in and will get the error I quoted, so my error report is bogus. Thanks for looking at it though.

I finally got tabbyAPI/exllamav2 working together with Open WebUI using an RTX-2060 SUPER, and once a model is loaded it appeared automatically in Open WebUI as promised :)

I've also made it work on an old Quadro P1000. Looks like my Maxwell cards are out though.
Thanks,
-D.