LocalLLaMA

1 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago

MODERATORS

communick@poweruser.forum

LLM's in production hardware requirements (alien.top)

submitted 10 months ago by purton_i@alien.top to c/localllama@poweruser.forum

3 comments fedilink hide all child comments

We've put together an article using some guesstimates of what it would be like for an enterprise to deploy LLM's on prem.

https://bionic-gpt.com/blog/llm-hardware/

In short, I'm estimating $20,000 in hardware costs per 1000 users, minimum.

I'd be grateful if people could give me some feedback on the numbers and whether my assumptions look realistic.

Thanks

you are viewing a single comment's thread
view the rest of the comments

[–] AsliReddington@alien.top 1 points 10 months ago (1 children)

It's extremely overpriced. With INT4 llama.cpp does even crazier numbers. A system with 4090s can be made for $2500 in India & cheaper elsewhere for sure.

[–] pmelendezu@alien.top 1 points 10 months ago

Didn’t Nvidia ban the use of consumer grade cards for professional uses? You will need to use A100s and whatnot for a datacenter