I found the Yi 34B models almost unusable due to repetition issues until I tried settings recommended in this discussion:
I have the same issue with LoneStriker_Nous-Capybara-34B-5.0bpw-h6-exl2. Whole previous messages will often get shoved into the response. I basically gave up and went back to Mistral-OpenHermes.
WizardLM is really old by now. Have you tried any of the Mistral finetunes? Don't discount it just because of the low parameter count. I was also running WizardLM-33b-4bit for the longest time, but Mistral-Hermes-2.5-7b-8bit is just so much more capable for what I need.
I’m not sure who told who that Mistral models are only 8k
The official Mistral product information.
Our very first foundational model: 7B parameters, fast-deployed and easily customisable. Small, yet powerful for a variety of use cases. Supports English and code, and a 8k context length. link
Does Mistral themselves actually mention 32k anywhere?
Capybara doesn't use Alpaca format, so that wouldn't do anything. Regardless, it's not that type of repetition. It's not speaking for the user, it's literally just copy/pasting part of the conversation into the answer.