main: failed to quantize model from 'gemma-3n-E2B-it.f16.gguf'

[0wwafa]

$ ./build/bin/llama-quantize --allow-requantize --output-tensor-type f16 --token-embedding-type f16 gemma-3n-E2B-it.f16.gguf gemma-3n-E2B-it.q5_k.gguf q5_k 2

main: build = 5760 (e8215dbb)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: quantizing 'gemma-3n-E2B-it.f16.gguf' to 'gemma-3n-E2B-it.q5_k.gguf' as Q5_K using 2 threads
llama_model_quantize: failed to quantize: tensor 'per_layer_token_embd.weight' data is not within the file bounds, model is corrupted or incomplete
main: failed to quantize model from 'gemma-3n-E2B-it.f16.gguf'

[aa956]

main: build = 5760 (e8215db)

Gemma 3n was not yet merged at commit e8215db

Update to at least 8846aac and try again?

[0wwafa]

that is the latest version:

latest=`curl -qsI "https://github.com/ggml-org/llama.cpp/releases/latest"|grep location|cut -d " " -f 2`
latest=`basename $latest|tr -d "\r"`
echo Downloading llama.cpp binaries version $latest
wget &>/dev/null "https://github.com/ggml-org/llama.cpp/releases/download/$latest/llama-$latest-bin-ubuntu-x64.zip" -O llama.zip

latest released version is b5760

[aa956]

Latest binary version, without gemma3n support, you can check the commit history.

Options are either wait for the next binary build release or compile yourself?

I've just tried to git pull the latest source (f667f1e), compile, install python tools requirements, convert to gguf and quantize.

No errors with E4B. I've no E2B downloaded unfortunately.

$ ./build/bin/llama-quantize --allow-requantize --output-tensor-type f16 --token-embedding-type f16 ~/tmp/gemma-3n-E4B-it-F16.gguf ~/tmp/gemma-3n-E4B-it-Q5_K.gguf q5_k 2
main: build = 5764 (f667f1e6)
main: built with cc (Debian 14.2.0-19) 14.2.0 for x86_64-linux-gnu
main: quantizing '~/tmp/gemma-3n-E4B-it-F16.gguf' to '~/tmp/gemma-3n-E4B-it-Q5_K.gguf' as Q5_K using 2 threads
llama_model_loader: loaded meta data with 44 key-value pairs and 847 tensors from ~/tmp/gemma-3n-E4B-it-F16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma3n
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = 1189b64daea8a7fef667a8d64ffa25253bc6861b
llama_model_loader: - kv   3:                           general.finetune str              = 1189b64daea8a7fef667a8d64ffa25253bc6861b
llama_model_loader: - kv   4:                         general.size_label str              = 6.9B
llama_model_loader: - kv   5:                            general.license str              = gemma
llama_model_loader: - kv   6:                   general.base_model.count u32              = 1
llama_model_loader: - kv   7:                  general.base_model.0.name str              = Gemma 3n E4B
llama_model_loader: - kv   8:          general.base_model.0.organization str              = Google
llama_model_loader: - kv   9:              general.base_model.0.repo_url str              = https://huggingface.co/google/gemma-3...
llama_model_loader: - kv  10:                               general.tags arr[str,5]       = ["automatic-speech-recognition", "aut...
llama_model_loader: - kv  11:                     gemma3n.context_length u32              = 32768
... skip ...
[ 845/ 847]          blk.34.post_ffw_norm.weight - [ 2048,     1,     1,     1], type =    f32, size =    0.008 MB
[ 846/ 847]              blk.34.post_norm.weight - [ 2048,     1,     1,     1], type =    f32, size =    0.008 MB
[ 847/ 847]                   blk.34.proj.weight - [  256,  2048,     1,     1], type =    f16, converting to q5_K .. size =     1.00 MiB ->     0.34 MiB
llama_model_quantize_impl: model size  = 13097.32 MB
llama_model_quantize_impl: quant size  =  8256.52 MB

main: quantize time = 93326.26 ms
main:    total time = 93326.26 ms
$ ls ~/tmp
-rw-rw-r-- 1  13G ~/tmp/gemma-3n-E4B-it-F16.gguf
-rw-rw-r-- 1 8,1G ~/tmp/gemma-3n-E4B-it-Q5_K.gguf
$

[0wwafa]

Yep. I know. I was just pointing out that the latest binary builds are behind.