LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Is it possible to run Llama on a 4gb ram? (alien.top)

submitted 11 months ago by TheHumanFixer@alien.top to c/localllama@poweruser.forum

7 comments fedilink hide all child comments

Yes. This has to be the worst ram you guys have ever seen but hear me out. Is it possible? I want to run the full 70gb model but that’s far out of question and I’m not even going to bother. Can I atleast run the 13gb or at least the 7gb?

you are viewing a single comment's thread
view the rest of the comments

[–] DarthInfinix@alien.top 1 points 11 months ago

Hmm, theoretically if you switch to a super light Linux distro, and get the q2 quantization 7b, using llama cpp where mmap is on by default, you should be able to run a 7b model, provided i can run a 7b on a shitty 150$ Android which has like 3 GB Ram free using llama cpp