LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Is it possible to fine tune a 33B model with 48GB vRAM? (alien.top)

submitted 11 months ago by tgredditfc@alien.top to c/localllama@poweruser.forum

10 comments fedilink hide all child comments

Currently I have 12+24GB VRAM and I get Out Of Memory all the time when try to fine tune 33B models. 13B is fine, but the outcome is not very good so I would like to try 33B. I wonder if it’s worthy to replace my 12GB GPU with a 24GB one. Thanks!

you are viewing a single comment's thread
view the rest of the comments

[–] Sabin_Stargem@alien.top 1 points 11 months ago

It isn't practical for most people to make their own models, that requires industrial hardware. The "b" is billion, which indicates the size and potential intelligence of the model. Right now, the Yi-34b models are the best for that size.

I recommend using a Mistral 7b as your introduction to LLM. They are small but fairly smart for their size. Get your model from Huggingface. For your model, something like Mistral Dolphin should do fine.

I recommend KoboldCPP for running a model, as it is very simple to use. It uses GGUF format, allowing you to use your GPU, RAM, and CPU. Other formats are exclusively GPU, offering greater speed but less flexibility.