Selfhosted

60366 readers

957 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

139

Do you host your own AI? (aussie.zone)

submitted 1 week ago by SuspiciousCarrot78@aussie.zone to c/selfhosted@lemmy.world

205 comments fedilink hide all child comments

Do you host your own ML / AI / LLM? What do you use, and what do you use it for?

you are viewing a single comment's thread
view the rest of the comments

[–] plasma8726@lemmy.today 10 points 1 week ago (1 children)

Thanks for this link. Because of this article, I had claude stand up a llama.cpp container next to my already running ollama container. It ran side by side tests with the same model and parameters, and the results blew ollama out of the water. I'm in the process of moving hermes and openwebgui over to the llama.cpp instance to see how it goes day to day.

[–] brucethemoose@lemmy.world 1 points 1 week ago* (last edited 1 week ago) (1 children)

If you’re using docker anyway, and “fast” pure GPU models, you might try a vllm container while you’re at it.

It should be much faster than even llama.cpp, albeit at the cost of context length, and it supports some exotic 4-bit quantization like SPQA.

Same with TabbyAPI. It’s quantization is SOTA, though it does not support CPU offloading, and it’s speed is somewhere between vllm and llama.cpp.

[–] plasma8726@lemmy.today 1 points 1 week ago (1 children)

Thanks! I'll look into this. I'm a bit limited at 12GB of VRAM right now.

[–] brucethemoose@lemmy.world 1 points 1 week ago* (last edited 1 week ago)

A 3060?

Exllama/TabbyAPI is still worth looking at if you are trying to run a model purely in GPU RAM. It’s easily the most VRAM efficient backend, it just doesn’t support CPU offloading (which is useful for MoEs if you have considerable spare CPU RAM) and more optimized for 4xxx and up Nvidia cards.

And TabbyAPI has a docker container you can use. Look for “exl3” models on huggingface.