this post was submitted on 18 Nov 2023
1 points (100.0% liked)

LocalLLaMA

3 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago
MODERATORS
communick@poweruser.forum
1
Is it possible to fine tune a 33B model with 48GB vRAM? (alien.top)
submitted 11 months ago by tgredditfc@alien.top to c/localllama@poweruser.forum
10 comments fedilink hide all child comments
 

Currently I have 12+24GB VRAM and I get Out Of Memory all the time when try to fine tune 33B models. 13B is fine, but the outcome is not very good so I would like to try 33B. I wonder if it’s worthy to replace my 12GB GPU with a 24GB one. Thanks!

you are viewing a single comment's thread
view the rest of the comments
[–] Aaaaaaaaaeeeee@alien.top 1 points 11 months ago

start with Lora rank=1, 4bit, flash-attention-2, context 256, batchsize=1 until your reach your maximum. Qlora 33b definitely works on just 24gb, it worked back a few months ago.