this post was submitted on 12 Nov 2023
1 points (100.0% liked)

LocalLLaMA

1 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago
MODERATORS
communick@poweruser.forum
1
Spread LLM to multiple hosts (alien.top)
submitted 10 months ago by erick-fear@alien.top to c/localllama@poweruser.forum
3 comments fedilink hide all child comments
 

Hi all I'm wondering if is there a possibility to spread load of localLLM on multiple hosts instead of adding gpu's to speed up responses. My host do not have gpu's since I want to be power effective, but they have decent ammont of ram 128. Thx for all ideas.

you are viewing a single comment's thread
view the rest of the comments
[–] dodo13333@alien.top 1 points 10 months ago

If i didnt missunderstood your question, answer is Petals:

https://research.yandex.com/blog/petals-decentralized-inference-and-finetuning-of-large-language-models