LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Anyone have a 1B or 3B model that is mostly coherent? (alien.top)

submitted 11 months ago by multiverse_fan@alien.top to c/localllama@poweruser.forum

16 comments fedilink hide all child comments

I've tried a few of these models but it was some months ago. Have y'all seen any that can hold a conversation yet?

you are viewing a single comment's thread
view the rest of the comments

[–] paryska99@alien.top 1 points 11 months ago (1 children)

Thanks for the input.

What inference engine did you use? It's possibly a bug as these things tend to happen with the new models.
I for one can't wait for the lookahead decoding in llamacpp and others, combine that with some smaller models and we'll have blazing fast speeds on pennies worth of hardware from what i recon.

[–] CardAnarchist@alien.top 1 points 11 months ago

I use koboldccp.

You are probably right about it being a bug as at first I couldn't get the model to work at all (it crashed koboldccp when loading up) but it was just because I had a week old version of koboldccp. I needed to download the version that came out like 4 days ago (at that time) ha! Then it loaded up fine but with that already mentioned quirk. I guess it will get fixed in short time.

Yeah the future of local LLM's lies in the smaller models for sure!