LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

load llama-2 in 8b quantization? (alien.top)

submitted 1 year ago by peterwu00@alien.top to c/localllama@poweruser.forum

3 comments fedilink hide all child comments

The question is probably too basic. But how do i load llama2 70B model using 8b quantization? I see TheBlokeLlama2_70B_chat_GPTQ but they only show 3b/4b quantization. I have 80G A100 and try to load llama2 70B model with 8b quantization. Thanks a lot!

you are viewing a single comment's thread
view the rest of the comments

[–] mcmoose1900@alien.top 1 points 1 year ago

Grab the original (fp16) models. They are quantized to 8-bit on the fly with bitsandbytes.

permalink
fedilink
source