I upgraded from gtx 1080 to rtx 3090 ti 2 weeks ago. I think going with rtx 3090 / 3090 ti / 4090 would be a good option for you, I don't know how big of a difference having stronger cpu would have, I think exllama v2 has some cpu bottlenecking going on, but I have no idea what is computed on cpu and why. There were moments during generation where it seemed like it was using only 1 thread and it was maxing it out, being bottleneck for gpu. I don't think ram matters a lot unless you train and merge loras and models.
this post was submitted on 25 Nov 2023
1 points (100.0% liked)
LocalLLaMA
1 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 10 months ago
MODERATORS
- The only correct answer. CPU doesn't matter.
Meh I’m running 13bs on a 13900k 64 gb ddr5 and a 6900xt with LM studio and it’s faster than my office workstations 12900ks 3090ti. Sometimes ram and processor with decent VRM is enough.
Your comparison proves his point! 13b will fit snuggly in your 6900 this is a head on comparison of the cards!
Welcome to the rabbit hole 😁. On a serious note, going for the newer generations pays dividends, in my opinion.
Curious, does anyone consider rtx A5000 / 6000? It’s twice the price of 4090/3090, but the performance improvement should be more than double
You want large vram on your video card. What ever is the largest ram video card that you can afford is the answer.
Is there a list where it has all GPUs and it can be ordered my VRAM?