LocalLLaMA

Using exl2 I have not had too many issues with the models. I did not d/l the GGUF because I read all the issues concerning BOS tokens.

They are similar to 70b but with poorer reasoning. With proper template they follow most instructions.

I've just been plodding along with them using dynamic temperature and min_P

Try setting repetition penalty to 1.0, it helped a lot on base model before finetune. Are you limited to trying gguf only? There was an issue with llama.cpp that made it to So that BOS token was always inserted, but Yi works best without BOS token. Make sure that llama cpp version you have in oobabooga has this fixed or try running newest llama.cpp exe yourself. I get good results with my 2 private yi-34b qlora fine-tunes and with LoneStriker's spicyboros 3.1-2 (all exl2). I think it's better than llama 70b 2.4bpw... I didn't check dolphin or nous Capybara. To be honest I am not sure I was filling in the context to 4096 in any case, I think I kept it around 300-2000.

(i know this is not a roleplay question but anyway ^_^) Settings i use for Silly tavern and the Nous Capybara model. Works perfect so far, but you also need the character CFG globally enabled to 1.5 to make it stop looping. { "temp": 0.1, "temperature_last": true, "top_p": 1, "top_k": 25, "top_a": 0, "tfs": 1, "epsilon_cutoff": 0, "eta_cutoff": 0, "typical_p": 1, "min_p": 0.05, "rep_pen": 1, "rep_pen_range": 0, "no_repeat_ngram_size": 15, "penalty_alpha": 0, "num_beams": 1, "length_penalty": 1, "min_length": 0, "encoder_rep_pen": 1, "freq_pen": 0, "presence_pen": 0, "do_sample": true, "early_stopping": false, "add_bos_token": true, "truncation_length": 2048, "ban_eos_token": false, "skip_special_tokens": true, "streaming": true, "mirostat_mode": 2, "mirostat_tau": 2.55, "mirostat_eta": 0.1, "guidance_scale": 1, "negative_prompt": "", "grammar_string": "", "banned_tokens": "", "type": "ooba", "legacy_api": false, "rep_pen_size": 0, "genamt": 1024, "max_length": 16128 }

I use nous-capybara for nlp processing with json output for work.

Nous capybara yi-34 q6 gguf works very well for complex summarization (which requires some instruction following) of large texts. Temp=0, repeat penalty =[0.9-1.1]. Prompt: SYSTEM: USER: ASSISTANT:

neural chat 7b_q8:

Paragraph One: Life is full of ups and downs, twists and turns. It's important to find a balance between embracing change and holding onto what truly matters in our lives. Throughout the years, I have learned that the key to happiness lies within fostering strong connections with family, friends, and people who bring positivity into our world. In times of adversity, it is these relationships that provide unwavering support and love.

Paragraph Two: As we progress through life, challenges will undoubtedly present themselves. The ability to adapt and grow from those difficulties is crucial for personal growth and self-improvement. Embracing change can often be frightening, but it also offers opportunities to learn new skills, strengthen our abilities, and push the boundaries of what we once thought was possible. Ultimately, it's through embracing these life lessons that we evolve into our best selves, fostering resilience and personal fulfillment.