this post was submitted on 25 Nov 2023
1 points (100.0% liked)
LocalLLaMA
1 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 10 months ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
It is not just the 4090's getting vacuumed up. A large amount of the post crypto crash stock has been snapped up for AI use. The used market only a year ago was flooded with things like mi25's and above that were being liquidated. Even the p40's have started to become more expensive. When I purchased mine a year ago it was just over $150 USD, now you will struggle to find one for under $250. For some reason the AMD compute capable cards coming out of China seem to be even more scarce than the Nvidia ones. I highly suspect there is some secret AMD specific pipeline in use that has not become public knowledge right now.
People are starting to really come to grips that 'this time is different' when it comes to the AI boom and it is starting to really impact GPU pricing and availability. The only upside compared to the Crypto boom I guess is that with AI based use cases is that PCIe bus speeds matter and this is stopping people buying anything and everything then slapping 8 GPU's in an AI mining rig.
Things are only going to get worse from here though, Nvidia and AMD both are too caught up in the server space right now to bother with consumer offerings that might compete. The average gamer is not going to demand more than the existing 24GB on their GPU as games simply do not need more at current resolutions. That leaves the limited workstation market and those have always come with a premium. The Pascal based Quadro cards are still selling for twice as much as a p40 and show no sign of coming down. They are not going to rush out and drop a "RTX AI Card" like they did with crypto because the server market would snap them up to drive lower speed training and inference farms.
China has a lot of used crypto GPU farms where you had racks of GPU's chugging away at crypto crunching. How hard would it be to convert them for A.I, use?
That 'depends'. Most of the crypto farms run on low cost motherboard/cpu combos with 8+ GPUs essentially connected via a single PCIe lane. If you wanted to do training or even inference on that, you would need to relocate those GPU's to a more capable system and then limit the number of cards to a maximum of 4 cards per system or less. At which point if you are talking about cards with 8GB or less VRAM you have an expensive to run and set up system with 32GB VRAM and fairly low performance. That is why the higher 16GB+ cards are all disappearing.