GoofAckYoorsElf

joined 11 months ago
sorted by: new top controversial old
How to run 70B on 24GB VRAM ? in c/localllama@poweruser.forum
[–] GoofAckYoorsElf@alien.top 1 points 10 months ago (1 children)

that 64gb of RAM is cutting it pretty close

Holy crap...

What is considered the best uncensored LLM right now? in c/localllama@poweruser.forum
[–] GoofAckYoorsElf@alien.top 1 points 10 months ago

Mistral-Hermes-2.5-7b-8bit

I've tried that one. It is... strange.

What is considered the best uncensored LLM right now? in c/localllama@poweruser.forum
[–] GoofAckYoorsElf@alien.top 1 points 10 months ago (1 children)

nous-capybara-34b

I haven't been able to use that with my 3090Ti yet. I tried TheBloke's GPTQ and GGUF (4bit) versions. The first runs into memory issues, the second, loaded with llama.cpp (which it seems to be configured on) loads, but is excruciatingly slow (like 0.07t/sec).

I must admit that I am a complete noob regarding all the different variants and model loaders.

What is considered the best uncensored LLM right now? in c/localllama@poweruser.forum
[–] GoofAckYoorsElf@alien.top 1 points 10 months ago (3 children)

I've been going with WizardLM-33B-V1.0-Uncensored-GPTQ for a while and it's okay. Is Nous-Capybara-34b better?

Why are you running local models? What are you doing with them? in c/localllama@poweruser.forum
[–] GoofAckYoorsElf@alien.top 1 points 10 months ago

Quite some the stuff that commercial/corporate models won't let me do and which I wouldn't do even if they let me. Private stuff. Yes, NSFW can of course be a part of it.

Furthermore, things where I think the commercial/corporate models are too expensive (no, I have not checked my power bill yet...).