Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 10 months ago

MODERATORS

communick@academy.garden

[D] Simplest mathematical example of a function that can only be solved by gradient descent (alien.top)

submitted 10 months ago by EatMeMonster@alien.top to c/machinelearning@academy.garden

25 comments fedilink hide all child comments

I'm trying to teach a lesson on gradient descent from a more statistical and theoretical perspective, and need a good example to show its usefulness.

What is the simplest possible algebraic function that would be impossible or rather difficult to optimize for, by setting its 1st derivative to 0, but easily doable with gradient descent? I preferably want to demonstrate this in context linear regression or some extremely simple machine learning model.

you are viewing a single comment's thread
view the rest of the comments

[–] wojcech@alien.top 1 points 10 months ago

Go back to your history: Cauchy is the earliest person I'm aware of to have used gradient descent, and he motivated it as

one ordinarily starts by reducing them to a single one by successive eliminations, to eventually solve for good the resulting equation, if possible. But it is important to observe that 1◦ in many cases, the elimination cannot be performed in any way; 2◦ the resulting equation is usually very complicated, even though the given equations are rather simple

That is, the usefulness of gradient descent is motivated when you have rough idea of when you are close to the minimum, but you don't want to go through the hassle of algebra. (realistically, if you can solve it with gradient descent, you could probably solve it algebraicly, we just don't have the same stupidly easy to implement computational routines for it)

https://www.math.uni-bielefeld.de/documenta/vol-ismp/40_lemarechal-claude.pdf