this post was submitted on 14 Nov 2023
1 points (100.0% liked)

LocalLLaMA

14 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 2 years ago
MODERATORS
communick@poweruser.forum
1
Good and fast model around ~1B to run on web? (alien.top)
submitted 2 years ago by palpapeen@alien.top to c/localllama@poweruser.forum
9 comments fedilink hide all child comments
 

I've playing with a lot of models around 7B but I'm now prototyping something that would be fine with a 1B model I think, but there's just Phi-1.5 that I've seen of this size, and I haven't seen a way to run it efficiently so far. llama.cpp has still not implemented it for instance.

Anyone has an idea of what to use?

you are viewing a single comment's thread
view the rest of the comments
[–] vatsadev@alien.top 1 points 2 years ago

RWKV 1.5B, its Sota for its size, outperforms tinyLlama, and uses no extra vram for fitting its whole ctx len in browser.