I really like OpenHermes-7B for chat/RP purposes because it just seems to me to say much more creative and entertaining things than Llama-based models I’ve tried. It also seems to have pretty good accuracy on accuracy and explanation quality for single prompts, sometimes even including coding (the most I ever do is simple R scripts). I run it through OobaBooga.
But on the flip side, it seems to have very poor context both of things I’ve said previously in the conversation and the perceived relationship of things in its surroundings based on the initial character prompt. And it basically never advances the story. Xwin-70B feels much less interesting in the way it speaks to me, but it can drive the story and mostly seems to understand what is going on.
What actual variables affect memory, as well as the LLMs desire to actually drive the story/conversation forward? Explain it to me like you would explain to a scientist in a non-machine learning field. Also, are there any Mistral based models out or on the horizon that do a better job in these areas where OpenHermes struggles?
It’s still a pre-print, you can release anything as a preprint. I’d only be concerned if this type of stuff started actually getting peer reviewed and accepted into respected journals.