LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Running large models below requirements? (alien.top)

submitted 11 months ago by Maelstrom100@alien.top to c/localllama@poweruser.forum

4 comments fedilink hide all child comments

Question about the possibility of running large models on a 3070ti 32gb ram, what's the best way to run them if possible, without quality loss?

Speed isn't an issue, just want to be able to run such models ambiently.

you are viewing a single comment's thread
view the rest of the comments

[–] reallmconnoisseur@alien.top 1 points 11 months ago

From my understanding, if you say you want to run the models without quality loss, then quantized models are not exactly what you are looking for, at least not below a certain threshold. With your setup you should be able to run 7B models in 8-bit.

For everything beyond that you'll need higher quantized models (e.g., 4-bit), which also introduce higher quality loss.

There was this post a while back which lined out the hardware requirements for 8-bit and 4-bit, for GPU and CPU setups. Of course you can go even higher with quantization and run even larger models, but it'll introduce more loss as well.