kobold.cpp should have its position. https://github.com/LostRuins/koboldcpp
this post was submitted on 26 Nov 2023
1 points (100.0% liked)
LocalLLaMA
4 readers
4 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 2 years ago
MODERATORS
Agreed, will add that !
What is your favorite project to interact with
I still don't have a favorite, tbh. I've tried a few of the UIs you shared, and I found them to be either too complicated or lacking in certain areas I need. Like many others, I ended up building my own.
Share your findings
Recently, I started collecting local UIs and this is what I've gathered so far: UI list.
I use kobold cpp for local llm deployment. It's clean, it's easy and allows for sliding context. Can interact with drop in replacement for OpenAI.
I mostly use a UI I made myself:
https://github.com/shinomakoi/AI-Messenger
Works with llama.cpp and Exllama V2, supports LLaVA, character cards and moar.
Right now, I'm using your earlier project [1]. It's proving to be incredibly helpful, thank you!.
Since it's a desktop application, it's more convenient for me than the WebUIs, because I tend to have a lot of tabs open in my browser, which makes it pretty chaotic. I have set up an AutoHotkey script to can easily launch it using a easy to remember hotkey.
Here is a new one I found the other day. Still seems to be WIP but overall I really like what is being done here - https://github.com/lobehub/lobe-chat
Wow looks very good indeed, how is the web extraction plugin? can you share some screenshots?
exui by turboderp (exllamav2 creator) is a nice ui for exl2 models. https://github.com/turboderp/exui
Can it serve on a CPU-only machine?
I wish we had a UI like this for GGUF (for Apple)
Nice, thanks for compiling this info.
https://github.com/serge-chat/serge
A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.
(...without websearch)
Serge is underrated, unknown, and development is slow because of it
I released a UI last week noco-ai/spellbook-docker (github.com). Has chat plugins with 50+ in v0.1.0 that handle things like simple math (multiplication, addition, ...), image generation, TTS, Bing news search, etc.
No exui?
https://github.com/turboderp/exui
Its blazing fast, vram efficient, supports minp and has a notebook mode... what else could I ask for.
I was using ooba before, but I have dumped it because its so much slower.
That looks very clean for sure.
Websearch is dope. Too bad for me because I am comfortable with pip, not npm. Setting this up will involve pulling some hair out, so I will not even attempt.
I have decent results with langchain and SERP API for google search with gpt 4 function calling. However, I would LOVE the implementation of ChatUI search functionality in python. I hope someone makes a wrapper (if thats even a thing - I am not a programmer by profession).
Im not great at troubleshooting errors but the install of chat-ui was pretty straightforward.
If you already have a llamacpp server it would be very easy to connect.
I enjoy the search functionality so much and I think its worth the hassle, if you need any help with it just comment here.
I have llamacpp server up and running. I will def give the install a shot!
If you need any help with the local.env. File, tell me and il help out
How chat-ui local? Last I tried they still require mongo.
I had some struggles with it, it works best for me in combination with llamacpp, and you need to run a docker command to start a mongo DB for you chats locally.
Even the search results can be queried on your device instead of API.
To keep track of this I put it all in a repo: https://github.com/JShollaj/Awesome-LLM-Web-UI
Thank you for all the recommendations and the list (Ive also been looking for some time :) )
Wow, thanks!
Cool! Can the list be added to the main repo ( GitHub - sindresorhus/awesome: 😎 Awesome lists about all kinds of interesting topics )
Or linked there under a small category?
People need to know about all of those great alternatives to ChatGPT :D
I have two kind of lists, one for OpenAI API-powered UIs (source, last updated this July), and another for one for local UIs (I'll update this with the list from XhoniShollaj). I feel like a better organization is needed, e.g. whether the UI is open source or not, model backend and architecture, differentiating feature, etc. Otherwise, the list is impossible to navigate (at least for me).
Thats a very neat layout u/itsuka_dev - love your project. I think we can keep both at the meantime (I want to add my mine to other Awesome lists for more exposure) - let me know what breakdown makes more sense from your end so I can improve my repo.
Thank you for the kind words! I stopped maintaining OAI UIs because things got a bit stagnant a few months ago (there were literally no new UIs for weeks, I think). But with the new features announced at DevDay, I'm expecting to see a surge in new UIs, especially those that leverages multi-modality. As a maintainer of list projects, this is such an exciting time.
I think we can keep both at the meantime
Absolutely. Someone needs to maintain an awesome list for local UIs, and I don't think my projects fall into that category. Besides, my list got a mix of native UIs in there too, which is important to me since I'm building both web and native UIs.
let me know what breakdown makes more sense from your end
For local UIs, sorting/grouping by model backend (e.g. llama.cpp, ollama, ExLlama) makes the most sense, IMO - and the rest of what I mentioned above is optional.
Thank you - I submitted a pull request to add it there. Hopefully gets approved. Let me know if there are other lists you would like me to add it to.
Actually will need 30 more days to get approved. Feel free to contribute additional projects to it at the meantime :)!
Any of these projects supply clustering multiple gpus/users, or even multiple machines.
Hi, is there a good UI to chat with ollama and local files (pdf, docx, whatever) and if possible multiple or even a lot of files ?
By the way, what is the difference between ollama and llamacpp, are the API incompatible ?
For PDF , docx and like 50 more formats, use h2oGPT, great for this kind of stuff.
Thank you
So, in the end which one would you recommend for someone just beginning to run LLMs locally? Windows machine (thus Sanctum is out of the question for now). I'm interested in 3 use cases, so maybe there would be a different answer for each of them:
- Python coding questions
- Linux shell questions
- RAG: in particular, I would like to be able to ask questions and have the model retrieve an answer online, supported by one or more working hyperlinks
You should look at LoLLMs webui, it has those options
I'll have a look into it and compare it to LM Studio.
I've got mixed experiences with Bavarder, native UI, fair choice of models to grab, but offen not working reliably. They seem to improve it slowly but steadily.