this post was submitted on 15 Nov 2023
1 points (100.0% liked)

LocalLLaMA

3 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago
MODERATORS
communick@poweruser.forum
1
Got Llama.cpp WebUI to work on Colab (alien.top)
submitted 1 year ago by chibop1@alien.top to c/localllama@poweruser.forum
6 comments fedilink hide all child comments
 

I got tired of slow cpu inference as well as Text-Generation-WebUI that's getting buggier and buggier.

Here's a working example that offloads all the layers of zephyr-7b-beta.Q6_K.gguf to T4, a free GPU on Colab.

It's pretty fast! I get 28t/s.

https://colab.research.google.com/gist/chigkim/385e8d398b40c7e61755e1a256aaae64/llama-cpp.ipynb

you are viewing a single comment's thread
view the rest of the comments
[–] nullnuller@alien.top 1 points 11 months ago

Great work. It would be nice to have some caching and automatically detecting and re-running the Colab session, as others have pointed out.