https://arxiv.org/abs/2309.17453
This paper might be useful. They use window attention with attention sinks to deal with longer texts.
https://arxiv.org/abs/2309.17453
This paper might be useful. They use window attention with attention sinks to deal with longer texts.