AnOnlineHandle

joined 1 year ago

Quantizing 70b models to 4-bit, how much does performance degrade? in c/localllama@poweruser.forum

[–] AnOnlineHandle@alien.top 1 points 11 months ago (10 children)

What sort of vram is needed to run a 4bit 70B model?

Why are you running local models? What are you doing with them? in c/localllama@poweruser.forum

[–] AnOnlineHandle@alien.top 1 points 1 year ago

I've tried training LoRas on my own writing with mixed success.