LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Exllama outside of text generation webui? (alien.top)

submitted 11 months ago by turamura@alien.top to c/localllama@poweruser.forum

5 comments fedilink hide all child comments

I want to use the ExLlama models because it enables me to use the Llama 70b version with my 2 RTX 4090. I managed to get it to work pretty easily via text generation webui and inference is really fast! So far so good...

However, I need the model in python to do some large scale analyses. I cannot seem to find any guide/tutorial in which it is explained how to use ExLlama in the usual python/huggingface setup.

Is this just not possible? If it is, can someone pinpoint me to some examplary code in which ExLlama is used in python.

Much appreciated!

you are viewing a single comment's thread
view the rest of the comments

[–] ReturningTarzan@alien.top 1 points 11 months ago (3 children)

There's a bunch of examples in the repo. Various Python scripts for doing inference and such, even a Colab notebook now.

As for the "usual" Python/HF setup, ExLlama is kind of an attempt to get away from Hugging Face. It reads HF models but doesn't rely on the framework. I've been meaning to write more documentation and maybe even a tutorial, but in the meantime there are those examples, the project itself, and a lot of other projects using it. TabbyAPI is coming along as a stand-alone OpenAI-compatible server to use with SillyTavern and in your own projects where you just want to generate completions from text-based requests, and ExUI is a standalone web UI for ExLlamaV2.

[–] turamura@alien.top 1 points 11 months ago

Got it to work! Thank you!!

load more comments (2 replies)