LocalLLaMA

1 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago

MODERATORS

communick@poweruser.forum

AMD EPYC CPU or 1x RTX 4090? (alien.top)

submitted 10 months ago by M000lie@alien.top to c/localllama@poweruser.forum

13 comments fedilink hide all child comments

I plan to infer 33B models at full precision, 70B is second priority but a nice touch. Would I be better off getting an AMD EPYC server cpu like this or a RTX 4090? With the EPYC, i am able to get 384GB DDR4 RAM for ~400USD on ebay, the 4090 only has 24GB. Moreover, both the 4090 and EPYC setup + ram cost about the same. which would be a better buy?

you are viewing a single comment's thread
view the rest of the comments

[–] tvetus@alien.top 1 points 10 months ago

at full precision

Full precision is not as useful as you think. Even at 4bit, the losses are not that large.

70B

What is your motivation for such large models? You're sacrificing a lot of speed for the larger model.