Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 1 year ago

MODERATORS

communick@academy.garden

[D] What academic papers/textbooks do you recommend to learn about the ReLU activation function? (alien.top)

submitted 11 months ago by fouried96@alien.top to c/machinelearning@academy.garden

4 comments fedilink hide all child comments

CHAT GPT recommends the paper "Rectified Linear Units Improve Restricted Boltzmann Machines" by Vinod Nair and Geoffrey E. Hinton as it is one of the foundational papers introducing and exploring the benefits of ReLUs in neural networks. It also says it is a good starting point to learn about ReLUs and their advantages in machine learning models.

But, from your experience do you have any other papers or textbooks or even videos that you would recommend to someone learning about it? I don't mind if they're math heavy, as I do have a Bsc Honours in App Math.

Thanks!

you are viewing a single comment's thread
view the rest of the comments

[–] d84-n1nj4@alien.top 1 points 11 months ago

I believe it was first created in “Cognitron: A self organizing multilayered neural network”, but was not referred to as ReLU. It was popularized by “Deep Sparse Rectifier Neural Networks” and “Rectified Linear Units Improve Restricted Boltzmann Machines”.

In regard to deep learning and GPU use: It’s efficient compared to other activation functions because it consists of comparison and thresholding operations, and the derivative is just 1 when positive and 0 if not (for backpropagation). It’s effective because it adds non-linearity to layers of linear operations like the convolution.