LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Is there a way to prevent coherency degradation when using high levels of RoPE scaling? (alien.top)

submitted 11 months ago by tenmileswide@alien.top to c/localllama@poweruser.forum

5 comments fedilink hide all child comments

For the sake of argument, let's say VRAM is no object.

If I set alpha_value to around 2.5 to 3 when loading a normally 4k base context model, I can get up to about 10k context before things start noticeably falling apart. Extending context higher than this, even if I increase alpha_value higher to go along with it, the model gets progressively less coherent.

I've found that I can attenuate this a little bit by messing around with different alpha values at different context loads, but it never really gets usable. It gets closer to where it needs to be, but still nothing I'd actually want to run.

Is this just the nature of the beast when it comes to extending context?

you are viewing a single comment's thread
view the rest of the comments

[–] FieldProgrammable@alien.top 1 points 11 months ago (1 children)

Yes. See this post and the graphs in it for an illustration of what happens to model performance with different context (this post is for 2k native context Llama 1 models so just scale the X axis accordingly for Llama 2).

As you increase the RoPE scaling, the positional embeddings of the prompt are deviating further and further from what the model was trained on. The different compression methods simply attempt to trade off usable quality at longer contexts in exchange for reduced performance at lower contexts. If the model is fine tuned on the compressed scaling, then this alleviates some of the losses, this is what is done with models like SuperHOT and Llongma, which fine tune the model on linear RoPE scaled data.

[–] qrios@alien.top 1 points 11 months ago

I don't think it's so simple as "the nature of the beast."

From my own experiments, you can maintain coherence by having stuff scale more the further back it is, but at some cost to accuracy. So stuff further back is more confused, but still accessible, and stuff more recent is still grounding the generation.

I haven't tested super thoroughly though.