Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 2 years ago

MODERATORS

communick@academy.garden

[R] Simplifying Transformer Blocks (alien.top)

submitted 2 years ago by APaperADay@alien.top to c/machinelearning@academy.garden

1 comments fedilink hide all child comments

Paper: https://arxiv.org/abs/2311.01906

GitHub: https://github.com/bobby-he/simplified_transformers

Abstract:

A simple design recipe for deep Transformers is to compose identical building blocks. But standard transformer blocks are far from simple, interweaving attention and MLP sub-blocks with skip connections & normalisation layers in precise arrangements. This complexity leads to brittle architectures, where seemingly minor changes can significantly reduce training speed, or render models untrainable.
In this work, we ask to what extent the standard transformer block can be simplified? Combining signal propagation theory and empirical observations, we motivate modifications that allow many block components to be removed with no loss of training speed, including skip connections, projection or value parameters, sequential sub-blocks and normalisation layers. In experiments on both autoregressive decoder-only and BERT encoder-only models, our simplified transformers emulate the per-update training speed and performance of standard transformers, while enjoying 15% faster training throughput, and using 15% fewer parameters.

https://preview.redd.it/6pz53ro6260c1.png?width=1129&format=png&auto=webp&s=ce9dad8de6cac575970e38d93a35786fe8880506

top 1 comments

sorted by: hot top controversial new old

[–] TheMeddlingMonk@alien.top 1 points 2 years ago

I’m a bit confused by the motivation of removing the skip connections. I’ve read things from people in the mechanistic interpretability community refer to the “residual stream” as not just a training hack but as an important part of the reason why transformer models work well. That the fact that information can propagate through the layers past the nonlinearities is seen as a feature not a bug.