LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Best model for situational awareness and a 4090? (alien.top)

submitted 1 year ago by Ok-Scar011@alien.top to c/localllama@poweruser.forum

1 comments fedilink hide all child comments

I'm trying to use a LLM to help me flesh out some filler for my stories. I find that a lot of people that do this put a lot of emphasis and importance in the quality of the writing it produces, where as I'm looking more for something that is capable of advanced reasoning and understanding. I plan on going through and rewriting everything to fit my personal prose, but I like to use ChatGPT to kind of get the ball rolling. The problem is that it's censorship is a bit much. I don't usually write NSFW stuff, but even things like violence and bloodshed get censored pretty heavily.

Is there a model that excels at understanding more than others that can be used on a 4090? I don't care about speed, just decent results.

you are viewing a single comment's thread
view the rest of the comments

[–] Ravenpest@alien.top 1 points 1 year ago

Speed+quality = Nous-Capybara 34b. Offload 13 layers to system and get a Q5_K_M. If you have enough system RAM and a decent CPU you wont even feel it. Just quality, Euryale 1.3 70b. It will be slow - up to 200 seconds for a single message at Q5_K_M - but it will deliver.