Sure, it's just going to generate 5 tokens per week
this post was submitted on 10 Nov 2023
1 points (100.0% liked)
LocalLLaMA
3 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 1 year ago
MODERATORS
It will never be this bad, at most, it would be 2min / t
What if models could run directly from storage instead of RAM?
You can already do that. That's what mmap does. It uses a file on storage as if it were RAM. It's not speedy. Since even the fastest SSD is slow compared to RAM.
Thanks didn't know that
RAM is storage, just faster to access and write to.
Do you want it to bake cookies, too?
Spoiler alert MRAM is just RAM but more expensive. You would be better off just buying more RAM.
Ram is just faster storage.
None of that is relevant to LLMS.
Did they confirm the LLM is on device memory? That wouldn’t make much sense to me at all. Siri already takes an input and sends it to the cloud to return a response. Why wouldn’t they use the same concept and just connect LLM to the cloud to process the response then send to the phonev
I ran a 13b Q_4 on a Raspberry Pi4 8Gb with Llama.cpp with no special settings, it just automatically cashed from disk... Was mega slow and got worse with more tokens, but did it. Don't know if it was Llama.cpp or Raspberry Pi OS that automatically cached.
You can cmake Llama.cpp on many platforms.