46G goliath-120b.Q2_K
So the smallest one I found (I didn't quantize this one myself, found it on HF somewhere)
And it was very slow. about 13t/s prompt_eval and then 2.5t/s generating text, so only really useful for me when I need to run it on my laptop (I get like 15t/s with 120b model on my 2x3090 rig at 3bpw exl2)
As for the models it's self, I like it a lot and use it frequently.
TBH, this ram thing is more helpful for me because it lets me run Q5 70b models instead of just Q4 now.
I wish we had a UI like this for GGUF (for Apple)
Hey mate, no need to do this now. There's a terminal command you can run instead. I did this on my M1 and it works fine.
> sudo sysctl iogpu.wired_limit_mb=57344
I did that for my 64GB, you'd want to change the 57344 to whatever you want for your 96GB
64GB M1 Max here. Before running the command, if I tried to load up goliath-120b: (47536.00 / 49152.00) - fails
And after sudo sysctl iogpu.wired_limit_mb=57344 : (47536.00 / 57344.00)
So I guess the default is: 49152
That totally worked. I can run goliath 120b on my m1 max laptop now. Thanks a lot.
The reason? Political topics are often dominated by those who have most resources, including more ideological zealots. I have no intention to explain things over and over again. Most people deserve to be handled by a bot if they want to to engage in political discussions, because they're just a biobot themselves (NPC meme).
This won't work, because you can convince bots like ChatGPT and get them to agree with your political views. This doesn't work with real humans though.
Thanks for this, I'm about 3GB short of running Goliath-120b on my 64gb mbp.
What hardware are you running these on now?
I can run the 3.0bpw exl2 of Goliath on my 2x3090. But for Venus, I could only load it when I dropped the context down to 2048.
Are the spelling issues with the 120b's because we're running them at 3bpw vs 4+ for the 70b and smaller?
Yeah okay, I'll give them a try again. I only ever tried one, and it was completely insane, always ended up with something sexual and after a while started randomly spamming words like 'sex toy'.
Looks like it was taken down / experimental: https://old.reddit.com/r/LocalLLaMA/comments/16qrdpa/plotbot_13b_finetuned_llama_2_model_for_writing/
haha damn, I should have taken the NSFW warning seriously before clicking the huggingface link in front of people lol.
Is this model any good for SFW stuff?
I wish Mozilla would just stick to Firefox, and invest the rest of the money into some dividend paying fund, so they aren't so reliant on Google for funding for their software engineers.