this post was submitted on 27 Nov 2023
1 points (100.0% liked)

LocalLLaMA

1 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago
MODERATORS
communick@poweruser.forum
1
Quantizing 70b models to 4-bit, how much does performance degrade? (alien.top)
submitted 9 months ago by ae_dataviz@alien.top to c/localllama@poweruser.forum
22 comments fedilink hide all child comments
 

The title, pretty much.

I'm wondering whether a 70b model quantized to 4bit would perform better than a 7b/13b/34b model at fp16. Would be great to get some insights from the community.

you are viewing a single comment's thread
view the rest of the comments
[–] NachosforDachos@alien.top 1 points 9 months ago

That was useful and interesting.

Speaking of hypothetical situations how much money do you think an individual would need to buy the computing power needed to provide themselves with a gpt 4 turbo like experience locally?