UPDATE: I forgot to mention that I used q8 of all models.
So I've had a lot of interest in the non-code 34b Finetunes, whether it's CodeLlama base or Yi base. From the old Samantha-34b and Synthia-34b to the new Dolphin-Yi and Nous-Capybara 34b models, I've been excited for each one because it fills a gap that needs filling.
My problem is that I can't seem to wrangle these fine-tunes into working right for me. I use Oobabooga (text-gen-ui), and always try to choose the correct instruction template either specified on the card or on TheBloke's page, but the models never seem to be happy with the result, and either get confused very easily or output odd gibberish from time to time.
For both Yi models, I am using the newest ggufs that TheBloke put out... yesterday? Give or take. But I've tried the past 2-3 different ggufs for the same model he's updated with when they came out.
The best luck I've had with the new Yi models was doing just plain chat mode with my AI Assistant's character prompt as the only thing being sent in, but even then both Yi fine-tunes that I tried eventually broke down after a few thousand context.
For example, after a bit of chattering with the models I tried a very simple little test on both: "Please write me two paragraphs. The content of the paragraphs is irrelevant, just please write two separate paragraphs about anything at all." I did that because previous versions of these two struggled to make a new line, so I just wanted to see what would happen. This absolutely confused the models, and the results were wild.
Has anyone had luck getting them to work? They appear to have so much potential, especially Nous Capybara which went toe to toe with GPT-4 in this benchmark, but I'm failing miserably at unlocking its full potential lol. If you have gotten it to work, could you please specify what settings/instructions you're using?
neural chat 7b_q8:
Paragraph One: Life is full of ups and downs, twists and turns. It's important to find a balance between embracing change and holding onto what truly matters in our lives. Throughout the years, I have learned that the key to happiness lies within fostering strong connections with family, friends, and people who bring positivity into our world. In times of adversity, it is these relationships that provide unwavering support and love.Paragraph Two: As we progress through life, challenges will undoubtedly present themselves. The ability to adapt and grow from those difficulties is crucial for personal growth and self-improvement. Embracing change can often be frightening, but it also offers opportunities to learn new skills, strengthen our abilities, and push the boundaries of what we once thought was possible. Ultimately, it's through embracing these life lessons that we evolve into our best selves, fostering resilience and personal fulfillment.repeated a few times, it never fails, different topics, paragraph, newline, paragraph