this post was submitted on 26 Nov 2023
1 points (100.0% liked)

LocalLLaMA

3 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago
MODERATORS
communick@poweruser.forum
1
Low memory bandwidth utilization on 3090? (alien.top)
submitted 11 months ago by Aaaaaaaaaeeeee@alien.top to c/localllama@poweruser.forum
6 comments fedilink hide all child comments
 

I get 20 t/s with a 70B 2.5bpw model, but this is only 47% of the theoretical maximum of 3090.

In comparison, the benchmarks on the exl2 github homepage show 35 t/s, which is 76% the theoretical maximum of 4090.

The bandwidth differences between the two GPUs aren't huge, 4090 is only 7-8% higher.

Why? Does anyone else have a similar 20 t/s ? I don't think my cpu performance is the issue.

The benchmarks also show ~85% utilization on 34B on 4bpw (normal models)

top 6 comments
sorted by: hot top controversial new old
[–] brobruh211@alien.top 1 points 11 months ago (1 children)

Hi! What are your settings for Ooba to get this to work? On Windows 11 on a single 3090, I keep getting CUDA out of memory error trying to load a 2.4bpw 70B model with just 4k context. It's annoying because this used to work but after a recent update it just won't load anymore.

[–] Aaaaaaaaaeeeee@alien.top 1 points 11 months ago (1 children)

8k with 2.4bpw and 20 t/s, the vram usage says 23.85/24.00 gb.

16k with 2.4bpw 20 t/s with fp8 cache

I have 0.5-0.6gb used for driving the monitor graphics on ubuntu.

Did you disable the nvidia system memory fallback that they pushed on Windows users? That's probably what you need.

[–] brobruh211@alien.top 1 points 11 months ago

Thanks for the detailed answer! Ubuntu does seem to be much more memory-efficient compared to Windows. However, the problem just fixed itself seemingly overnight. Now I'm not running into out of memory errors. 8-bit cache is a godsend for vram efficiency.

[–] mcmoose1900@alien.top 1 points 11 months ago (1 children)

Try exui instead of ooba.

[–] Aaaaaaaaaeeeee@alien.top 1 points 11 months ago

same story here.

[–] Sat0r1r1@alien.top 1 points 11 months ago

My results are the same as yours.

I use TabbyAPI, 70B 2.4bpw I get 20/T.