Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 1 year ago

MODERATORS

communick@academy.garden

[D] Simplest mathematical example of a function that can only be solved by gradient descent (alien.top)

submitted 11 months ago by EatMeMonster@alien.top to c/machinelearning@academy.garden

25 comments fedilink hide all child comments

I'm trying to teach a lesson on gradient descent from a more statistical and theoretical perspective, and need a good example to show its usefulness.

What is the simplest possible algebraic function that would be impossible or rather difficult to optimize for, by setting its 1st derivative to 0, but easily doable with gradient descent? I preferably want to demonstrate this in context linear regression or some extremely simple machine learning model.

you are viewing a single comment's thread
view the rest of the comments

[–] tarsiospettro@alien.top 1 points 11 months ago (1 children)

I do not understand most comments here.

Gradient descent is just the same as the tangent method. It is ubiquitously used, e.g. find minimum of whatever polynomial of degree >= 4.

Calculating derivative and finding 0 of the derivative is still the same problem. You look for a numerical solution using gradient descents. Bisection is slower and not so effective for multivariate functions.

I would say the opposite: there are more optimisation problems where gradient descent is used than not (excluding everything which can be solved by linear systems)

[–] DoctorFuu@alien.top 1 points 11 months ago

This reminds me that I still didn't read the paper on forward-forward algorithm and thus I'm not even sure if it's still gradient descent.