LocalLLaMA

1 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago

MODERATORS

communick@poweruser.forum

Is it possible to fine tune a 33B model with 48GB vRAM? (alien.top)

submitted 10 months ago by tgredditfc@alien.top to c/localllama@poweruser.forum

10 comments fedilink hide all child comments

Currently I have 12+24GB VRAM and I get Out Of Memory all the time when try to fine tune 33B models. 13B is fine, but the outcome is not very good so I would like to try 33B. I wonder if it’s worthy to replace my 12GB GPU with a 24GB one. Thanks!

you are viewing a single comment's thread
view the rest of the comments

[–] a_beautiful_rhind@alien.top 1 points 10 months ago (1 children)

Should work on single 24g gpu as well as either qlora or alpaca_lora_4bit. You won't get big batches or big context but it's good enough.

[–] tgredditfc@alien.top 1 points 10 months ago

Thanks! I have some problems to load GPTQ models with transformer loader.