LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Threadripper pro - how much does core count matter? (alien.top)

submitted 11 months ago by EvokerTCG@alien.top to c/localllama@poweruser.forum

2 comments fedilink hide all child comments

So I'm looking into Threadripper pro systems, which can offer a pretty good memory bandwidth as they are 8 channel, and can have a huge amount of RAM. (I can put a 3090 or two in there too.)

I'm wondering how much the core count is going to affect performance. For example, the 5955WX has 16 cores while the 5995WX has 64 cores. They can both use the same memory though. There's little point spending extra if the limiting factor will be somewhere else.

you are viewing a single comment's thread
view the rest of the comments

[–] jeffwadsworth@alien.top 1 points 11 months ago

I use a Ryzen 12 core and can use llama.cpp with the 70b 8bit fine. Do not bother with hyper-threads, though.