this post was submitted on 09 Nov 2023
3 points (100.0% liked)

Self-Hosted Main

504 readers
1 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

For Example

We welcome posts that include suggestions for good self-hosted alternatives to popular online services, how they are better, or how they give back control of your data. Also include hints and tips for less technical readers.

Useful Lists

founded 1 year ago
MODERATORS
communick@selfhosted.forum
3
Looking for a self-hosted chatgpt-like tool with an api (alien.top)
submitted 1 year ago by Ajnart@alien.top to c/main@selfhosted.forum
7 comments fedilink hide all child comments
 

Hello ! I am looking for a self-hosted or self-hostable nlp (like llama or chatgpt) but very very small one that could work with as low as ~300 mb of ram. It needs to have an API. I plan to integrate it into my dashboard project you might know, homarr. I'd like to make some kind of assistant to directly help within the app by using its integrations capabilities.

The tool needs to be self-hosted so that users won't leak the queries to anyone. A freemium service that you can either self-host or pay for would also work.

It does not need to have a huge knowledge base (doesn't need to know a good lobster recipe) , just to be able understand basic language inputs and in turn I will make it communicate with the key parts of the app

​

I apologize if this is not worded properly as I am fairly new to the world of LLMs.

top 7 comments
sorted by: hot top controversial new old
[–] MonsieurNoss@alien.top 1 points 1 year ago

Take a look at https://ollama.ai/ there is a docker image.

And there is a few good models (not as good as ChatGPT) you can run such as: openhermes2.5-mistral

I use it with chatbot-ollama.

[–] the9engines@alien.top 1 points 1 year ago (2 children)

ChatGPT is so far ahead and so advanced that:

- No model is even close to its quality

- Even if it it was released to the public, you would need so beefy machines to run it it makes no sense

We got to wait for some kind of breakthrough that would allow running high quality open source models locally.

Considering the cost of hosting anything, even if it was host on a PC at your place, the electricity bill alone would be higher than the ChatGPT API cost.

[–] bitspace@alien.top 1 points 1 year ago

This is the correct answer. Unfortunately it's not the answer many people want to hear, and a lot of people end up going with some grifter scheme that sells snake oil. The gen AI space is currently 99.5% grift and 0.5% legitimate business.

[–] grahamsz@alien.top 1 points 1 year ago

I can run VMWare's Open LLama 7B v2 Open Instruct on my laptop comfortably (though I have 64GB ram and 16GB VRAM) and my sense is that's it's probably somewhere between GPT2 and GPT3 in inference quality. It is, however, very slow. Even with my comparatively strong hardware, it's slow enough that I wouldn't want to use it in an interactive context (though it may be useful for background processing)

[–] tshawkins@alien.top 1 points 1 year ago

LocalAi

https://github.com/mudler/LocalAI

[–] persiusone@alien.top 1 points 1 year ago

I do a bunch of AI stuff, but you won't get chatgpt quality from anything else. It requires a massive amount of storage, memory and processing hardware- millions of dollars in hardware alone. Not sure what you're trying to do exactly, but that model is insane to attempt reproduction in any part

[–] AK1174@alien.top 1 points 1 year ago

you could use LocalAI or ollama. but neither is going to work with 300mb of ram, and it needs a bunch compute resources for response speed to be usable. these models are also not very capable, in comparison to openAI’s gpt’s, but that depends on what your goal is with the models.