this post was submitted on 30 Nov 2023
1 points (100.0% liked)
LocalLLaMA
3 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Hi, thanks for your comment!
I saw e.g., the "inference.py" in the repo which I think I could utilize. It actually looks kind of simple. However, I am struggling with what to provide as the "model directory". Should I just download a Huggingface model (for example, I would like to work with TheBloke/Llama-2-70B-GPTQ), and then specify this as model directory? Or what kind of structure does ExLlama expect as model directory?
Yes, the model directory is just all the files from a HF model, in one folder. You can download them directly from the "files" tab of a HF model by clicking all the little download arrows, or there's
huggingface-cli
. Alsogit
can be used to clone models if you've gotgit-lfs
installed.It specifically needs the following files:
But it may utilize other files in the future such as tokenizer_config.json, so best just to download all the files and keep them in one folder.