this post was submitted on 25 Nov 2023
1 points (100.0% liked)

LocalLLaMA

1 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago
MODERATORS
communick@poweruser.forum
1
Looking for hardware and model recommendations -- data center install (alien.top)
submitted 10 months ago by GWBrooks@alien.top to c/localllama@poweruser.forum
3 comments fedilink hide all child comments
 

I'd like to run a local model on a server in a data center. Assumptions:

  • Lots of DDR4 RAM (assume 256GB with most available to the AI work) and storage, but no standalone GPU.
  • Looking for a web frontend and Auto GPT-ish ability (i.e., it can search for answers on the web) if such a thing is currently available with locally hosted models.
  • I'd like to train it on policy reports (assume 10-12 pages of text per report), but I can live without this if training is wholly off the table without a dedicated GPU.

I'm very comfortable with Linux, running servers, virtual environments, etc. But not spun up on the latest in locally hosted LLMs. Assume I'm an idiot about all of this and point me in the right direction? Thanks!

top 3 comments
sorted by: hot top controversial new old
[–] No_Baseball_7130@alien.top 1 points 10 months ago (1 children)

1: What is your budget?

2: Do you have access to the data center and are you able to put in a GPU?
3: Does the server have SXM sockets or just PCIE?

[–] GWBrooks@alien.top 1 points 10 months ago (1 children)

Well, the budget is $0 at the experimental stage -- we wanted to see what we could achieve by throwing a lot of RAM at the problem.

If we have to add a GPU, Yeah, we have access to the server. Current box would accommodate a Gen3 PCIe, FHHL slot. Frankly, if we have to invest in a GPU, we'll also upgrade the whole server to a later-gen/more-powerful CPU, NVME storage, etc.

[–] No_Baseball_7130@alien.top 1 points 9 months ago

you could get a gpu like a p100 16gb for simple ai work or a v100/A4 for slightly more heavy duty work

p100s only cost around 170$, so it's cheap to upgrade the gpu