Hi, I have searched for a long time on this subreddit, in Ooba's documentation, Mistral's documentation and everything, but I just can't find what I am looking for.
I see everyone claiming Mistral can handle up to 32k context size, however while it technically won't refuse to generate anything above like 8k, the output is just not good. I have it loaded in Oobabooga's text-generation-webui and am using the API through SillyTavern. I loaded the normal Mistral 7B just to check, but with my current 12k story, all it can generate is gibberish if I give it the full context. However, I also checked using other fine-tunes of Mistral.
What am I doing wrong? I am using the GPTQ version on my RX 7900 XTX. Is it just advertising that it won't crash until 32k or something, or am I doing something wrong for not getting coherent output above 8k? I did mess with the alpha values, and while doing so does eliminate the gibberish, I do get the idea that the quality does suffer somehow.
I can't speak to running on AMD cards, but Mistral uses what's called "Sliding Window Attention." That means, Mistral only looks at the last 4k tokens of context, but each of those tokens looked at the 4k before it. That is, it doesn't have to recompute the entire attention KV cache until 32k tokens have passed.
E.g., imagine the sliding window was only 6 words/punctuations. If you wrote "Let's eat grandma! She sounds like she" then the model can remember here "grandma" is a food item and will have put that information into the word "she". Meanwhile, for "Let's eat, grandma! She sounds like she" then the added comma makes clear the speaker is speaking to "grandma" so "She" is probably a different person, and it may assume the sound is the end of the cooking for the preparation of eating.
A model that didn't have sliding window attention and had a limit of 6 words/punctuations would only see "grandma! She sounds like she" and would have to make up the context -- it wouldn't remember anything about eating.
I don't believe messing with alpha values is a good idea, but I've never done it on any model. My Mistral 7B instance in chat mode had no trouble with a conversation extending past 9k tokens, though for obvious reasons it couldn't remember the beginning of the conversation and it was expectedly dumb with esoteric information, being only a 7B model.
Very interesting. Well, in hindsight I should've noticed but performance did decrease after 8k tokens, but became completely unusable after 10k. I actually am pretty disappointed to still know nothing. No one actually documents what does and doesn't work and when or how. I can barely find anything about SWA (I know what it is in essence) but no one documents how it works, where and if you can set the window size and whether or not it's in Ooba's app.
And then there is the problem that I don't know if it's supported on AMD cards like you said. Try to look it up, sliding window attention on Google by itself just gives endless pages of "tutorials" and "guides" that don't tell anything. And combining it with Rocm just gives random results that don't lead anywhere useful.