LocalLLaMA

GGUF k-quants are really good at making sure the most important parts of the model are not x bit but q6_k if possible. GPTQ and AWQ models can fall apart and give total bullshit at 3 bits while the same model in q2_k / q3_ks with around 3 bits usually outputs sentences.

I have 2 gpus and AWQ never works for me on Oobabooga, no matter how I split the vRAM, oom in most of the cases.

Hell yeah! Two days in a row! We need more people doing format comparisons and benchmarks in general. Again, thank you for all of your hard work, and keep 'em coming!

How would you say EXL2 subjectively compares to GGUF? Have you had the chance to roleplay with both formats outside of Voxta+VaM (i.e., in SillyTavern)? I ask because I'm sure the increased generation speed is more important than anything when using Voxta+VaM so it might be easier to compare their output quality in SillyTavern.

On that note, would you say you now prefer using lzlv (70B, EXL2) over OpenChat 3.5 (7B, GGUF) with Voxta+VaM?

EXL2 5.0bpw was surprisingly doing much worse than GGUF Q2_K

Risitas.mov https://www.youtube.com/watch?v=QT13kk8HDDo

The major reason I use exl2 is speed, like on 2x4090 I get 15-20 t/s at 70b depending of the size, but GGUF I get like tops 4-5 t/s.

When using 3 gpus (2x4090+1x3090), it is 11-12 t/s at 6.55bpw vs GGUF Q6_K that runs at 2-3 t/s.

Though I agree with you, for model comparisons and such you need to have deterministic results and also the best quality.

If you can sometime, try 70b at 6bpw or more, IMO it is pretty consistent and doesn't have issues like 5bpw/bits.

The performance hit is too much on multigpu systems when using GGUF. I guess if in the future the speed gets to the same level, I would use it most of the time.

Great work as always! Regarding Exl2 its sensitive to calibration dataset - probably the one that was used is not related to your tests. I.e. you can get higher scores in HumanEval even in 3 bits that you would get in transformers 8bit. I hope that this standard will get more popular and finetuners will do their own measurement file/quants using their dataset. Never seen q2 gguf doing better than exl2 unless i mixed rope config.

Edit - for anything higher than 4.25bit i usually use 8bit head

Steve jobs would have been a better president than Donald Trump. But it's a silly comparison really, it's like comparing apples to oranges.

on 2.Xbpw, untick "add bos_token" avoiding the "cord string builder" looping

For real-time uses like Voxta+VaM, EXL2 4-bit is better

Wow, I didn't expect to see a Virt-a-Mate reference. You left no stone unturned and are doing God's work.

/u/WolframRavenwolf

Honestly, ever since I saw someone mention that with EXL2 I could run a 70b model on a single 4090/3090/24 VRAM I was instantly hooked. Especially since enabling the 8bit cache option meant you could run even higher context sizes albeit 2x more sometimes.

The main advantage as you mention is speed. As a RP'er myself, I care somewhat less about quality responses. Speed is king in my opinion since you can always swipe for more alternative responses. It's very hard to let go of 20-30 T/s vs <5 T/s on GGUF. 😭

Baseline of 70b is good enough to justify the tradeoff of quality. Besides, I don't have to buy ANOTHER 4090 to run 70b models.

Personally, I run waldie_lzlv-limarpv3-l2-70b-2.4bpw-h6-exl2 version of lzlv. It isn't broken for 1 and it seems to give somewhat better and creative responses.

Side note: Did you notice in Nous Cabybara 34b that spelling mistakes or weird sentences would form in longer contexts? Because sometimes I would get weird non-sensical sentences or stuff like I'll' or even a Chinese character

I wasn't aware that Exl2 had issues with quality. Your tests seem to suggest that equivalent bpw in Exl2 produce worse results than in GGUF. I wonder why that is.

I'm probably going to ask something extremely basic, but why GPTQ isn't an option? With OP's double GPU he can run 4bit 32g with 8k context, and I was under impression that the quality loss is barely noticeable. Though I noticed it absolutely messes up numbers (math, or historical dates).

The speeds don't really surprise me. They're going to take longer to load, but the math is about the same once they're stood up.

i run lzlv 2.4bpw without problems. make sure to disable bos token, then it should work way better.

It may be interesting to anyone running models across 2 3090s that in llama.cpp/koboldcpp there's a performance increase if your two GPUs support peering with one another (check with nvidia-smi topo -p2p r) - it wasn't working with my particular motherboard, so I installed an nvlink bridge and got a performance bump in token generation (an extra 10-20% with 70b, more with smaller models, except smaller models go much faster if you can fit them on one gpu).

I have no idea what the performance diff is between having a bridge and peering via pci-e if your system supports it. I also tested exl2 and there was no difference as I don't think it implements any sort of peering optimisations.

FYI, AWQ released 0.1.7 that fixes multi-GPU. Should alleviate OOM issues on multi-GPU, which became broken with newer versions of Huggingface libraries.

https://github.com/casper-hansen/AutoAWQ/releases/tag/v0.1.7

For the 2.4bpw and 2.6bpw exl2 models, you have to change a setting in ooba to get them to generate coherent text. Disable this setting:

Add the bos_token to the beginning of prompts

https://preview.redd.it/4v8m7ciu0y1c1.png?width=356&amp;format=png&amp;auto=webp&amp;s=785837b8466a3bcda3e49477424b7c377a8d542f

The very low bpw models need the above setting as well as being more strict with the prompt format. The higher bpw models are more flexible and can deal with prompt formats they were not specifically tuned for.

I would also set the VRAM for 2.4 to use only a single GPU. Spreading them out over two GPUs is not needed and will slow them down. That's the main reason I generate 2.4 (and 2.6bpw) versions is to allow people with only a single 3090 or 4090 to run 70B models at full speeds. Though obviously quality will be lower than the higher-bit models. For 2.6bpw to fit on a single 24 GB VRAM GPU, you will need to enable the cache_8bit option.

This is difficult to evaluate. It could be that exl2 just breaks the translation layer.