this post was submitted on 15 Nov 2023
1 points (100.0% liked)

LocalLLaMA

1 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago
MODERATORS
communick@poweruser.forum
1
Got Llama.cpp WebUI to work on Colab (alien.top)
submitted 10 months ago by chibop1@alien.top to c/localllama@poweruser.forum
6 comments fedilink hide all child comments
 

I got tired of slow cpu inference as well as Text-Generation-WebUI that's getting buggier and buggier.

Here's a working example that offloads all the layers of zephyr-7b-beta.Q6_K.gguf to T4, a free GPU on Colab.

It's pretty fast! I get 28t/s.

https://colab.research.google.com/gist/chigkim/385e8d398b40c7e61755e1a256aaae64/llama-cpp.ipynb

top 6 comments
sorted by: hot top controversial new old
[–] HadesThrowaway@alien.top 1 points 10 months ago

Koboldcpp also has an official colab, https://colab.research.google.com/github/LostRuins/koboldcpp/blob/concedo/colab.ipynb

[–] herozorro@alien.top 1 points 10 months ago

how long does it stay alive/online?

[–] nullnuller@alien.top 1 points 10 months ago

Great work. It would be nice to have some caching and automatically detecting and re-running the Colab session, as others have pointed out.

[–] Cute_Confection6105@alien.top 0 points 10 months ago (1 children)

I ran telegram bot with llama.cpp backend with this notebook https://colab.research.google.com/drive/1nTX1q7WRkXwSbLLCUs3clPL5eoJXShJq?usp=sharing

[–] herozorro@alien.top 0 points 10 months ago (1 children)

how do you reboot it after the coolab dies

[–] chibop1@alien.top 1 points 10 months ago

Just run it again.