Worldly-Mistake-8147

joined 1 year ago
sorted by: new top controversial old
1
Goliath120B.exl2 way too emotional (alien.top)
 

I'm using 4.5bpw quant courtesy u/Panchovix for RP.

Every time we speak about loneliness, lost love or something like this, it start t-to... s-speak... *sob* speak like... this.

And I can't find a way to recover the conversation from this state. The narrations are fine, only character's speech is affected. It especially prominent when using Mirostat due to lack of repetition control.

So far I tried, with little to no success:

  • instruct it to speak as usual,
  • rewriting it's reply,
  • temporary switching to sampler strategy with repetition penalty,

Anyone else experiences this, and how else to deal with it?

How bottlenecked are LLMs by CPU clock? (Budget options to host multiple GPUs) in c/localllama@poweruser.forum
[–] Worldly-Mistake-8147@alien.top 1 points 11 months ago

Holy... 4x3090! No wonder it was hard to find my third one for reasonable price.

πŸΊπŸ¦β€β¬› **Big** LLM Comparison/Test: 3x 120B, 12x 70B, 2x 34B, GPT-4/3.5 in c/localllama@poweruser.forum
[–] Worldly-Mistake-8147@alien.top 1 points 11 months ago

Have you tried kobold horde?

πŸΊπŸ¦β€β¬› LLM Format Comparison/Benchmark: 70B GGUF vs. EXL2 (and AWQ) in c/localllama@poweruser.forum
[–] Worldly-Mistake-8147@alien.top 1 points 11 months ago

I'm probably going to ask something extremely basic, but why GPTQ isn't an option? With OP's double GPU he can run 4bit 32g with 8k context, and I was under impression that the quality loss is barely noticeable. Though I noticed it absolutely messes up numbers (math, or historical dates).

Where and how to run Goliath 120b GGUF with good performance? in c/localllama@poweruser.forum
[–] Worldly-Mistake-8147@alien.top 1 points 1 year ago (2 children)

I'm sorry for a little side-track, but how much context you able to squeeze into your 3 GPUs with Goliath's 4bit quant?
I'm considering to add another 3090 to my own doble-GPU setup just to run this model.