I really wish there was a site where you could plug in your hardware and see what t/s speed you could expect from it, so if anyone has a link like that, I'd be interested. I haven't been able to find one, and feel like I'm pretty much a noob when it comes to understanding what parts of hardware are important for local fine tuning and inference and running models, so please bear with me as I ask a bunch of probably dumb questions.
Broadly and in order, I think single GPU VRAM matters (more gb the better), then local RAM matters (same, but speed matters too I think?), then PCIE bus bandwidth speeds in gb/s matters, then additional GPU's matter (for 60% and 30% and decreasing speedups from there), and finally CPU and/or NVME space might matter a little. Does that sound broadly correct?
So the situation is I've got a ton of 30 series NVIDIA GPU's from a mining operation I wrapped up.
I could never sell them on r/hardwareswap or anywhere else, bc nobody would buy in bulk, and I'm sure the hell not wasting my time selling and shipping 75+ individual GPU's to whoever. I do have racks and mobos and power supplies and whatever too, but I don't think that matters. I also have a decent amount of 6800 and 6700xt and 5700xt AMD cards, but I don't think that matters either - please correct me if I'm wrong.
I'd like to use as many GPU's as possible for local fine tuning and inference, and am trying to figure out the best path for that. After reading about PCIE bandwidth and the speedups from 2 and 3 additional GPU's, I'm afraid the real answer is "sell some GPU's and buy an M2 Ultra Mac pro" or something like that, but if we couldn't do that route, what is the best path forward?
An EPYC server build with as many 3090's and 3080's as I can fit and either 96gb (2 sticks, full DDR5 speed) or 192gb (4 sticks, only DDR4 speed) of ram? Which ram config is better? I think the DDR5 vs DDR4 speed actually makes a difference, but am not sure how much of a difference.
Researching EPYC mobos, I think I can fit maybe 6 or 7 GPU's into an EPYC build, does that sound about right? Anyone know of any PCIE-rich mobo's or architectures that I could fit notably more GPU's than that into? I do have a bunch of mining mobo's, but don't think they're usable?
I'm pretty sure there's nothing possible like a beowulf cluster of mining boards + GPU's that you can use for model fine tuning / running, is that correct?
I also have a Threadripper linux box I could upgrade that can currently fit 4-6 GPU's, and could upgrade to an AM5 mobo and a 79503xd CPU pretty easily. I don't know how this stacks up against an EPYC build, does anyone have any ideas on that?
I looked up my current linux box mobo and the PCIE lanes only have 32gb/s bandwidth, so think a mobo upgrade to AM5 with 128gb/s would be necessary to get decent speeds, does that sound right?
Sorry for all the questions and my general lack of knowledge, any guidance or suggestions on maximizing a bunch of GPU's are very welcome.
Yeah, I'm on this side of the issue. How strong is your relationship, really? It's not a pleasant thing to contemplate or hear, but starting and growing a new business is usually hell on relationships, and if your relationship is marginal or weak already, it may be better to decide and act on that now.
Divorce or unhappiness in a marriage is the dominant case (>50% for divorce, probably more like >75% when you factor in how many marriages are net unhappy), so real talk, with a base rate that high, most people are not currently in a relationship that is net positive and net happy and that will last. The rate of divorce when one partner is an entrepreneur is higher already, and highest still when undergoing the stress to start and grow a new business.
Given that context, seriously evaluate if she's the right person to be your partner as you go forward, because it's going to get 10x harder and more stressful for both of you if you actually put in the work and successfully grow your business, and if she's fighting you already, it's going to be five times harder to put in the work to succeed along the way.
And if your current relationship is weak in any way, the odds are that the cumulative years of greater stress will break you up anyways.
If you believe that is likely to be the case, much better to do it now before more success and more money, and then you'll also be able to put in longer hours to grow the biz without being harassed and nagged and stressed about it.