Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 1 year ago

MODERATORS

communick@academy.garden

[R] Hierarchically Gated Recurrent Neural Network for Sequence Modeling (alien.top)

submitted 11 months ago by APaperADay@alien.top to c/machinelearning@academy.garden

1 comments fedilink hide all child comments

Paper: https://arxiv.org/abs/2311.04823

Code: https://github.com/OpenNLPLab/HGRN

Models: https://huggingface.co/OpenNLPLab

Abstract:

Transformers have surpassed RNNs in popularity due to their superior abilities in parallel training and long-term dependency modeling. Recently, there has been a renewed interest in using linear RNNs for efficient sequence modeling. These linear RNNs often employ gating mechanisms in the output of the linear recurrence layer while ignoring the significance of using forget gates within the recurrence. In this paper, we propose a gated linear RNN model dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes forget gates that are lower bounded by a learnable value. The lower bound increases monotonically when moving up layers. This allows the upper layers to model long-term dependencies and the lower layers to model more local, short-term dependencies. Experiments on language modeling, image classification, and long-range arena benchmarks showcase the efficiency and effectiveness of our proposed model. The source code is available at this https URL.

https://preview.redd.it/thph9bpmjh3c1.png?width=965&format=png&auto=webp&s=8e4871cd280ef7e5b771b463435d47da11dca52d

you are viewing a single comment's thread
view the rest of the comments

[–] CatalyzeX_code_bot@alien.top 1 points 11 months ago

Found 1 relevant code implementation for "Hierarchically Gated Recurrent Neural Network for Sequence Modeling".

Ask the author(s) a question about the paper or code.

If you have code to share with the community, please add it here 😊🙏

To opt out from receiving code links, DM me.