What's the question?
this post was submitted on 14 Nov 2023
1 points (100.0% liked)
LocalLLaMA
3 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 1 year ago
MODERATORS
See, you're attending to the initial token across all layers and heads. :P
IYH kindly post the paper link
If you are wondering if it could be implemented, there was a modified transformers library. The author practically made changes, renamed the library to attention_sinks and presented it as a drop-in solution to use it:
https://github.com/tomaarsen/attention_sinks/
But it was impossible to maintain, so devs of transformers suggested him to make a patch for transformers and to maintain it, so it could be properly incorporated into the library and to be future-proof.
The author of this code has been working on this patch since beginning of the October:
it's already implemented in llama.cpp