LocalLLaMA

1 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago

MODERATORS

communick@poweruser.forum

What kind of specs to run local llm and serve to say up to 20-50 users (alien.top)

submitted 9 months ago by Appropriate-Tax-9585@alien.top to c/localllama@poweruser.forum

10 comments fedilink hide all child comments

Hi all,

Just curious if anybody knows the power required to make a llama server which can serve multiple users at once.

Any discussion is welcome:)

you are viewing a single comment's thread
view the rest of the comments

[–] seanpuppy@alien.top 1 points 9 months ago (1 children)

It depends a lot on the details tbh. Do they share one model? Do they each use a different lora? If its the latter theres some cool recent research on efficiently hosting many loras on one machine

[–] Appropriate-Tax-9585@alien.top 1 points 9 months ago

At the moment I’m just trying to grasp the basics, like for example what kind of GPUS I will need and how many. This is more for comparison to SaaS options, however in reality I need to setup a server for testing with just few users. I’m going to research into but I like this community and to hear others view on the case as many have tried to manage their own servers I imagine :)