chibop1

joined 1 year ago
sorted by: new top controversial old
1
Llama.cpp on colab with Bakllava-1.Q8_0 (Mistral+LLaVA 1.5) (alien.top)
 

I got Llama.cpp to work with BakLLaVA (Mistral+LLaVA 1.5) on colab.

Here's a working example that offloads all the layers of bakllava-1.Q8_0 to T4, a free GPU on Colab.

https://colab.research.google.com/gist/chigkim/a5be99a864c4196d5e379a1e6e280a9e/bakllava.ipynb

FYI, Colab has no persistent storage, and you cannot keep a Colab instance running for a long time. I guess it's on purpose for their business reason. You have to setup and download everything from scratch every time you run. Colab is more for demo/experimentation, not meant to run a server for production.

1
Video-LLaVA can describe both image and video input. (github.com)
Got Llama.cpp WebUI to work on Colab in c/localllama@poweruser.forum
[–] chibop1@alien.top 1 points 11 months ago

Just run it again.

1
Got Llama.cpp WebUI to work on Colab (alien.top)
 

I got tired of slow cpu inference as well as Text-Generation-WebUI that's getting buggier and buggier.

Here's a working example that offloads all the layers of zephyr-7b-beta.Q6_K.gguf to T4, a free GPU on Colab.

It's pretty fast! I get 28t/s.

https://colab.research.google.com/gist/chigkim/385e8d398b40c7e61755e1a256aaae64/llama-cpp.ipynb