Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 11 months ago

MODERATORS

communick@academy.garden

[D] Massive amount of RAM needed to run a model. (alien.top)

submitted 9 months ago by theonewhoask11@alien.top to c/machinelearning@academy.garden

11 comments fedilink hide all child comments

I have a good PC, but the problem is with my RAM. The model is LLM, which makes the memory requirement understandable, but I can't afford to get that much RAM right now so I'm looking for a roundabout. Any help?

you are viewing a single comment's thread
view the rest of the comments

[–] theonewhoask11@alien.top 1 points 9 months ago (2 children)

That is exactly the reason, I'm literally looking at my options and from the comments, it definitely isn't looking good 😭

Let's say I want to work on Noon (https://huggingface.co/Naseej/noon-7b), how much would I actually need?

[–] The-Protomolecule@alien.top 1 points 9 months ago

Not trying to be rude, but you’re also like saying you want to participate in a car race and you don’t have a fast car kind of a prerequisite unfortunately.

You could look at things that offload your model to disk but they’re going to be slow as hell.

[–] KingsmanVince@alien.top 1 points 9 months ago (1 children)

Paste your model name in this HF space, https://huggingface.co/spaces/Vokturz/can-it-run-llm

https://imgur.com/a/Ednemii (the result)

It seems you need less than 32 GiBs vram

[–] theonewhoask11@alien.top 1 points 9 months ago

And thanks to you, it now works. I knew exactly what to do and what I needed to get it to work. Thanks, man!