LocalLLaMA

1 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago

MODERATORS

communick@poweruser.forum

load llama-2 in 8b quantization? (alien.top)

submitted 10 months ago by peterwu00@alien.top to c/localllama@poweruser.forum

3 comments fedilink hide all child comments

The question is probably too basic. But how do i load llama2 70B model using 8b quantization? I see TheBlokeLlama2_70B_chat_GPTQ but they only show 3b/4b quantization. I have 80G A100 and try to load llama2 70B model with 8b quantization. Thanks a lot!

you are viewing a single comment's thread
view the rest of the comments

[–] vec1nu@alien.top 1 points 10 months ago (1 children)

I haven't used gptq in a while, but i can say that gguf has 8 bit quantization, which you can use with llamacpp. Furthermore, if you use the original huggingface models, the ones which you load using the transformers loader, you have options in there to load in either 8 or 4bit.

[–] peterwu00@alien.top 1 points 10 months ago

thanks!