this post was submitted on 12 Oct 2024
182 points (95.5% liked)

Selfhosted

40173 readers
649 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

Instructions here: https://github.com/ghobs91/Self-GPT

If you’ve ever wanted a ChatGPT-style assistant but fully self-hosted and open source, Self-GPT is a handy script that bundles Open WebUI (chat interface front end) with Ollama (LLM backend).

  • Privacy & Control: Unlike ChatGPT, everything runs locally, so your data stays with you—great for those concerned about data privacy.
  • Cost: Once set up, self-hosting avoids monthly subscription fees. You’ll need decent hardware (ideally a GPU), but there’s a range of model sizes to fit different setups.
  • Flexibility: Open WebUI and Ollama support multiple models and let you switch between them easily, so you’re not locked into one provider.
you are viewing a single comment's thread
view the rest of the comments
[–] TheHobbyist@lemmy.zip 22 points 1 month ago (25 children)

whats great is that with ollama and webui, you can as easily run it all on one computer locally using the open-webui pip package or in a remote server using the container version of open-webui.

Ive run both and the webui is really well done. It offers a number of advanced options, like the system prompt but also memory features, documents for RAG and even a built in python ide for when you want to execute python functions. You can even enable web browsing for your model.

I'm personally very pleased with open-webui and ollama and they both work wonders together. Hoghly recommend it! And the latest llama3.1 (in 8 and 70B variants) and llama3.2 (in 1 and 3B variants) work very well, even on CPU only, for the latter! Give it a shot, it is so easy to set up :)

[–] Tobberone@lemm.ee 5 points 1 month ago (4 children)

Do you know of any nifty resources on how to create RAGs using ollama/webui? (Or even fine-tuning?). I've tried to set it up, but the documents provided doesn't seem to be analysed properly.

I'm trying to get the LLM into reading/summarising a certain type of (wordy) files, and it seems the query prompt is limited to about 6k characters.

[–] TheHobbyist@lemmy.zip 3 points 1 month ago* (last edited 1 month ago) (1 children)

For RAG, there are some tools available in open-webui, which are documented here: https://docs.openwebui.com/tutorials/features/rag They have plans for how to expand and improve it, which they describe here: https://docs.openwebui.com/roadmap#information-retrieval-rag-

For fine-tuning, I think this is (at least for now) out of scope. They focus on inferencing. I think the direction is to eventually help you create/manage your own data which you get from using LLMs using Open-WebUI, but the task of actually fine-tuning is not possible (yet) using either ollama or open-webui.

I have not used the RAG function yet, but besides following the instructions on how to set it up, your experience with RAG may also be somewhat limited depending on which embedding model you use. You may have to go and look for a good model (which is probably both small and efficient to re-scan your documents yet powerful to generate meaningful embeddings). Also, in case you didn't know, the embeddings you generate are specific to an embedding model, so if you change that model you'll have to rescan your whole documents library.

Edit: RAG seems a bit limited by the supported file types. You can get it here: https://github.com/open-webui/open-webui/blob/2fa94956f4e500bf5c42263124c758d8613ee05e/backend/apps/rag/main.py#L328 It seems not to support word documents, or PDFs, so mostly incompatible with documents which have advanced formatting and are WYSIWYG.

[–] Tobberone@lemm.ee 2 points 1 month ago (1 children)

Thank you for your detailed answer:) it's 20 years and 2 kids since I last tried my hand at reading code, but I'm doing my best to catch up😊 Context window is a concept I picked up from your links which has provided me much help!

[–] TheHobbyist@lemmy.zip 1 points 1 month ago

Sure! It can be a bit of a steep learning curve at times but there are heaps of resources online, and LLMs can also be useful, even if it just in pointing you in the direction for further reading. Regardless, you can reach out to me or other great folks from the !localllama@sh.itjust.works or similar AI, ML or related communities!

Enjoy :)

[–] theterrasque@infosec.pub 2 points 1 month ago* (last edited 1 month ago) (1 children)

Increase context length, probably enable flash attention in ollama too. Llama3.1 support up to 128k context length, for example. That's in tokens and a token is on average a bit under 4 letters.

Note that higher context length requires more ram and it's slower, so you ideally want to find a sweet spot for your use and hardware. Flash attention makes this more efficient

Oh, and the model needs to have been trained at larger contexts, otherwise it tends to handle it poorly. So you should check what max length the model you want to use was trained to handle

[–] Tobberone@lemm.ee 1 points 1 month ago (1 children)

I need to look into flash attention! And if i understand you correctly a larger model of llama3.1 would be better prepared to handle a larger context window than a smaller llama3.1 model?

[–] theterrasque@infosec.pub 1 points 1 month ago

No, all sizes of llama 3.1 should be able to handle the same size context. The difference would be in the "smarts" of the model. Bigger models are better at reading between the lines and higher level understanding and reasoning.

[–] Zos_Kia@lemmynsfw.com 1 points 1 month ago (1 children)

There are not that many use cases where fine tuning a local model will yield significantly better task performance.

My advice would be to choose a model with a large context window and just throw in the prompt the whole text you want summarized (which is basically what a rag would do anyway).

[–] Tobberone@lemm.ee 1 points 1 month ago (1 children)

The problem I keep running into with that approach is that only the last page is actually summarised and some of the texts are... Longer.

[–] Zos_Kia@lemmynsfw.com 2 points 1 month ago (1 children)

Yeh, i did some looking up in the meantime and indeed you're gonna have a context size issue. That's why it's only summarizing the last few thousand characters of the text, that's the size of its attention.

There are some models fine-tuned to 8K tokens context window, some even to 16K like this Mistral brew. If you have a GPU with 8G of VRAM you should be able to run it, using one of the quantized versions (Q4 or Q5 should be fine). Summarizing should still be reasonably good.

If 16k isn't enough for you then that's probably not something you can perform locally. However you can still run a larger model privately in the cloud. Hugging face for example allows you to rent GPUs by the minute and run inference on them, it should just net you a few dollars. As far as i know this approach should still be compatible with Open WebUI.

[–] Tobberone@lemm.ee 1 points 1 month ago (1 children)

Thanks! I actually picked up the concept of context window, and from there how to create a modelfile, through one of the links provided earlier and it has made a huge difference. In your experience, would a small model like llama3.2 with a bigger context window be able to provide the same output as a big modem L, like qwen2.5:14b, with a more limited window? The bigger window obviously allow more data to be taken into account, but how does the model size compare?

[–] Zos_Kia@lemmynsfw.com 1 points 1 month ago

If I understand these things correctly, the context window only affects how much text the model can "keep in mind" at any one time. It should not affect task performance outside of this factor.

[–] TheHobbyist@lemmy.zip 1 points 4 weeks ago (1 children)

Someone recently referred me to this blog post about using RAG in open-webui. I have not tested if but the author seems to reach a good setup.

https://medium.com/@kelvincampelo/how-ive-optimized-document-interactions-with-open-webui-and-rag-a-comprehensive-guide-65d1221729eb

Perhaps this is of use to you?

[–] Tobberone@lemm.ee 2 points 4 weeks ago (1 children)

Thank you! Very useful. I am, again, surprised how a better way of asking questions affects the answers almost as much as using a better model.

[–] TheHobbyist@lemmy.zip 1 points 4 weeks ago (1 children)

Indeed, quite surprising. You got to "stroke their fur the right way" so to speak haha

Also, I'm increasingly more impressed with the rapid progress reaching open-weights models: initially I was playing with Llama3.1-8B which is already quite useful for simple querries. Then lately I've been trying out Mistral-Nemo (12B) and Mistrall-Small (22B) and they are quite much more capable. I have a 12GB GPU and so far those are the most powerful models I can run decently. I'm using them to help me in writing tasks for ansible, learning the inner workings of the Linux kernel and some bootloader stuff. I find them quite helpful!

[–] Tobberone@lemm.ee 2 points 4 weeks ago

I'm just in the beginning, but my plan is to use it to evaluate policy docs. There is so much context to keep up with, so any way to load more context into the analysis will be helpful. Learning how to add excel information in the analysis will also be a big step forward.

I will have to check out Mistral:) So far Qwen2.5 14B has been the best at providing analysis of my test scenario. But i guess an even higher parameter model will have its advantages.

load more comments (20 replies)