Could put a swap file on LTFS. Or just load more RAM via your local MicroCenter's IPoAC.
this post was submitted on 28 Nov 2023
1 points (100.0% liked)
Machine Learning
1 readers
1 users here now
Community Rules:
- Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
- Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
- Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
- Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.
founded 11 months ago
MODERATORS
The model is LLM,
Do You Have the Slightest Idea How Little That Narrows It Down?
That is exactly the reason, I'm literally looking at my options and from the comments, it definitely isn't looking good 😭
Let's say I want to work on Noon (https://huggingface.co/Naseej/noon-7b), how much would I actually need?
Paste your model name in this HF space, https://huggingface.co/spaces/Vokturz/can-it-run-llm
https://imgur.com/a/Ednemii (the result)
It seems you need less than 32 GiBs vram
And thanks to you, it now works. I knew exactly what to do and what I needed to get it to work. Thanks, man!
Not trying to be rude, but you’re also like saying you want to participate in a car race and you don’t have a fast car kind of a prerequisite unfortunately.
You could look at things that offload your model to disk but they’re going to be slow as hell.
You can pick up an old Xeon based server preconfigured with 512gb-1tb RAM for $350-1000. RAM will be slower 1033-2400 speed. AVX should be there by default, AVX2/AVX-512 even better. AVX2 on E5-2600 v3 series. The setup won’t rival an eight way SXM4 A100, but you can load some big models with slow responses.
/r/learnmachinelearning
Sell your old ram and use the $ to upgrade. If you have an extra slot, search for donated ram and drop it in (it needs to be the same make/model/density). Or, use a flash drive as readyboost, a server to do the heavy lifting, using VRAM, etc.