tvetus

joined 1 year ago
sorted by: new top controversial old
AMD EPYC CPU or 1x RTX 4090? in c/localllama@poweruser.forum
[โ€“] tvetus@alien.top 1 points 11 months ago

at full precision

Full precision is not as useful as you think. Even at 4bit, the losses are not that large.

70B

What is your motivation for such large models? You're sacrificing a lot of speed for the larger model.

I am going to buy H100s. There are too many options. in c/localllama@poweruser.forum
[โ€“] tvetus@alien.top 1 points 1 year ago

Can be hard to rent if all the capacity is bought out. But if it's just 1 DGX then they might be better off renting.