LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Does Dual EPYC work for LLMs? (alien.top)

submitted 11 months ago by EvokerTCG@alien.top to c/localllama@poweruser.forum

9 comments fedilink hide all child comments

Continuing my quest to choose a rig with lots of memory, one possibility is dual socket MBs. Gen 1 to 3 EPYC chips have 8 channels of DDR4, so this gives 16 total memory channels, which is good bandwidth, if not beating GPUs, but can have way more memory (up to 1024GB). Builds with 64+ threads can be pretty cheap.

My questions are

Does the dual CPU setup cause trouble with running LLM software?
Is it reasonably possible to get windows and drivers etc working on 'server' architecture?
Is there anything else I should consider vs going for a single EPYC or Threadripper Pro?

you are viewing a single comment's thread
view the rest of the comments

[–] EvokerTCG@alien.top 1 points 11 months ago (1 children)

I want to keep my options open, and potentially have a large context, which can add up to 100GB to memory requirements.

I'm considering 1x genoa CPU with 12 channels. Something like the 9354 would be more than enough cores. I might start with a cheaper DDR4 machine first though.

How was it getting the Epyc machine set up? Are you using windows? What about a GPU?

[–] xinranli@alien.top 1 points 11 months ago (1 children)

Setting things up was straight forward, the process is no different to building a commercial or workstation platform. The 7773X machine runs Window 10 and I have another 7452 QS machine that runs Ubuntu. Both are mostly pain free. I have EPYC boards from both Supermicro and ASRock. I find the ASRock board to be more "modern" and has a better BIOS, but Supermicro has slightly better community and official support. In the very early Naples era AMD's BIOS had some GPU compatibility issues, but I think nowadays you can use any GPU you want.

You can get very cheap Genoa engineering samples or qualification samples off eBay so you can skip the older DDR4 platforms. Their sockets are very different, you wouldn't even be able to reuse the heatsink.

One thing to watch out for when buying EPYCs is to definitely avoid vendor locked CPUs. Any EPYC CPUs once installed in a DELL or Lenovo board will be physically altered forever to not be able to boot on any other board. I got one once and it was a debugging nightmare until I realized the CPU was intentionally bricked by DELL...

[–] EvokerTCG@alien.top 1 points 11 months ago

Thanks. I can't find 'qualification samples' on ebay in the UK, unless you just find them through a serial number or something.

The DDR5 ram is more expensive, but it should hold value fairly well. I'll look for a 12 channel board.