LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Any tricks to speed up 13B models on a 3090? (alien.top)

submitted 1 year ago by DustGrouchy1792@alien.top to c/localllama@poweruser.forum

5 comments fedilink hide all child comments

Are there any tricks to speed up 13B models on a 3090?

Currently using the regular huggingface model quantized to 8bit by a GPTQ capable fork of KoboldAI.

Especially when the context limit changes, it's pretty slow and far from even remotely real time.

you are viewing a single comment's thread
view the rest of the comments

[–] StaplerGiraffe@alien.top 1 points 1 year ago (2 children)

Perhaps you are using a wrong fork of KobolAI, I get much more tokens per second. Did you open the task manager and check that the GPU memory used indeed increases when loading and using the model?

Otherwise try out Koboldcpp. It needs gguf instead gptq, but needs no special fork. With cublas enabled you should get good token speeds for a 13B model.

[–] DustGrouchy1792@alien.top 1 points 1 year ago (1 children)

Can I get koboldcpp working with sillytavern without too much of a headache?

[–] StaplerGiraffe@alien.top 1 points 1 year ago

Sure, it provides the same API as KoboldAI.

[–] DustGrouchy1792@alien.top 1 points 1 year ago

I'm now using a 4bit GPTQ version of the same model. After generation completes the VRAM goes up to 16.2 GB (out of 24 GB) and I have nothing else using GPU as best I can tell (no browser windows with youtube, etc).

Still only getting a bit under 4.00 tokens per second. So I don't think stuff is getting offloaded to CPU.