this post was submitted on 30 Nov 2023
1 points (100.0% liked)
LocalLLaMA
3 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
There's a bunch of examples in the repo. Various Python scripts for doing inference and such, even a Colab notebook now.
As for the "usual" Python/HF setup, ExLlama is kind of an attempt to get away from Hugging Face. It reads HF models but doesn't rely on the framework. I've been meaning to write more documentation and maybe even a tutorial, but in the meantime there are those examples, the project itself, and a lot of other projects using it. TabbyAPI is coming along as a stand-alone OpenAI-compatible server to use with SillyTavern and in your own projects where you just want to generate completions from text-based requests, and ExUI is a standalone web UI for ExLlamaV2.
Hi, thanks for your comment!
I saw e.g., the "inference.py" in the repo which I think I could utilize. It actually looks kind of simple. However, I am struggling with what to provide as the "model directory". Should I just download a Huggingface model (for example, I would like to work with TheBloke/Llama-2-70B-GPTQ), and then specify this as model directory? Or what kind of structure does ExLlama expect as model directory?
Yes, the model directory is just all the files from a HF model, in one folder. You can download them directly from the "files" tab of a HF model by clicking all the little download arrows, or there's
huggingface-cli. Alsogitcan be used to clone models if you've gotgit-lfsinstalled.It specifically needs the following files:
But it may utilize other files in the future such as tokenizer_config.json, so best just to download all the files and keep them in one folder.