Holy... 4x3090! No wonder it was hard to find my third one for reasonable price.
this post was submitted on 30 Nov 2023
1 points (100.0% liked)
LocalLLaMA
3 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 1 year ago
MODERATORS
I don’t even think lanes really matter when you’re not training.
Pretty much not at all. The main bottleneck is memory speed.
I barely see a difference between 4 and 12 cores on 5900X when running on CPU.
When running multi GPU, the lanes are the biggest bottleneck.
On single GPU, CPU does not matter.
8004 has six DDR5 channels afaik. That takes care of the memory bandwidth. The only issue would be an SP6 motherboard.
I would imagine that this new option you're talking about will be a good budget inference workhorse paired with multiple cards such as 3090s. 96 lanes of gen 5 will be a real enabler. That said, I think zen 2 epycs providing gen 4 lanes are cheaper still so there are good options available.
3090 doesn't support PCIE 5.0, only 4.0
The 4090 does, and it makes some sense to use them in x8 5.0 configuration, but only if you have a pallet of these GPUs.
So that really depends. You're talking about running a multi gpu setup. If all of your model is in the gpu, then your processor will not be a bottleneck at all. The clock speed of the PCIe bus is independent of the cpu cores, unless you're messing with overclocking. That's why they advertise PCIe 3.0, 4.0, 5.0, etc. The PCIe version dictates the bandwidth per lane.
That being said, multi gpu setups do introduce some overhead. If a model is split between GPUs, the PCIe interface becomes a modest bottleneck as they pass data back and forth. The greater the number of GPU's the model is split across, the greater the bottleneck.