this post was submitted on 27 Nov 2023
1 points (100.0% liked)

LocalLLaMA

3 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago
MODERATORS
communick@poweruser.forum
1
most powerful model for an A6000? (alien.top)
submitted 11 months ago by crackinthekraken@alien.top to c/localllama@poweruser.forum
11 comments fedilink hide all child comments
 

so I got this shiny new GPU and I want to push it to the limit. What’s the most powerful, smartest model out there? Ideally something with as much long-term memory as possible. I’m coming off of ChatGPT 4 and want something local and uncensored

you are viewing a single comment's thread
view the rest of the comments
[–] Sea_Particular_4014@alien.top 1 points 11 months ago

If you're on Windows, I'd download KoboldCPP and TheBloke's GGUF models from HuggingFace.

Then you just launch KoboldCPP, select the .gguf file, select your GPU, enter the number of layers to offload, set the context size (4096 for those), etc and launch it.

Then you're good to start messing around. Can use the Kobold interface that'll pop up or use it through the API with something like SillyTavern.