I realize this is totally unhelpful but, the DGX - 8x H100 costs just slightly more than the median price of a new house in the US . . .
I'm not saying this is a poor decision but . . . man that is one hell of a decision.
this post was submitted on 31 Oct 2023
1 points (100.0% liked)
LocalLLaMA
3 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 1 year ago
MODERATORS
if it's a company that could be a drop in the bucket
Yes! Put more money in it, the company!!!
OP isn't buying them for his personal setup, though that would be a baller move.
Yeah, it is not my money but still stressful
Wait, whos money is it? Can't you just rent as well?
Can be hard to rent if all the capacity is bought out. But if it's just 1 DGX then they might be better off renting.
I tried to rent from LambdaLabs yesterday but there was no availability for any gpu
OP isn't buying them for his personal setup
Tbh I don't really see how this explains anything. Sure, OP doesn't go bankrupt buying it for the company but I'm 99% certain that it's still a bad financial decision.
Definitely thought this was for his homelab
My friendβs company has a bunch of DGX idling for months. Ainβt that something.
H100 in the DGX is not the H100 PCI-e, but about 30% faster. When in doubt, just go DGX
I will talk to my boss for more money π
Out of curiosity, what kind of projects are you working on that require purchasing such GPUs rather than renting on the cloud?
Nooooo! DGX you pay for the name and "service" by Nvidia. PCIe is lacking fast interconnect with nvswitch. There is a layer in between: HGX.,it's basically DGX without the branding.
You can get such systems from Supermicro and ASUS
where you buying from, eBay? there are no reputable sellers, atm
Not living in the US atm but no reputable sellers neither here π
Supermicro makes SXM using servers for H100 I think. So you don't have to buy PCIE H100s or be forced to use the DGX.
-
You will be lucky to find a supplier who does not have a long waiting list. The demand in the enterprise sector is real and I'm calling BS on any supplier having stock before Q2 2024.
-
You have done absolutely no research or even begun to look into the architecture and capability of the hardware you are discussing. If you have seriously been given the task to choose a hardware platform for your company then I worry for your companies future. There is a reason system architects in large organisations get paid a lot.
If you are fine tuning you MAY get away with a NVLinkd pair of H100s if running smaller models, however you will be massively nerfed for any 'proper' work, and certainly have no chance of training your own model.
NVLink gets a bad name. It shouldn't. Think of it as PCIE on steroids, connecting all devices so they don't have to touch the PCIE bandwidth. Or even more valuable, not requiring CPU cycles and instead being able to directly communicate with each other. Saving a massive amount of latency as well as general optimisation.
The SXM options are the best bet for serious work due to their interconnectivity capabilities. The PCIE devices are essentially sxm modules on a PCB with a massive power limit applied to minimise overheating or cooling issues. PCIE - 250w / SXM - 450W .
And that's not even touching on the use of infiband or other compatible fabrics for direct compute access from connected devices ( again skipping CPU cycles and communication ). RDMA ftw.
So again, I'm calling BS. Usually I just smile and move on when reading another fantasists bs story that never turns out to result in anything. However they are becoming more and more common, especially on this sub.
If I am wrong, I apologise profusely. As stated previously, If you are honestly the member of staff that has been put in charge of a procurement decision like this then I truly feel sorry for whoever you work for.
Hello,
Can you give an estimation of what the prices are ?
If you need help with other vendors check out https://gpumonger.com
Gathers all pricing from all cloud gpu vendors.
Do tell us how you intend to train models. Specifically, which open source projects you're using, frameworks etc. Some of us are borrowing these cards with blood and sweat :)