LocalLLaMA

> Multiple passes at lower learning rates isn't supposed to produce different results.

Oh, I was wrong on this, then, my bad.

So would my interpretation be correct that this is essentially causing the overfitting to still happen, just significantly slower, and that a higher LR would work? The problem is at first the average loss tanked in the span of like a single epoch to near zero which overfit, but this LR didn't have the same effect.

gpt4 is claiming this comment's claim is wrong, but I can't trust it blindly ofc, i'll look into my initial claim to verify

Multiple passes at lower learning rates isn't supposed to produce different results.

Overfitting is not a technical challenge, its a mathematical property which undeniably exists when ever the training data is smaller than the full problem domain and simultaneously the learning rate (importantly - multiplied by the number of epochs!) would result in a higher specialization ration on the learned to unobserved data than would be expected based on the ration of the learned to unobserved size.

Basically if you learn 1 digit addition but half your training sets involve the left number being 1 and none of your training sets involve your left number being 5 then likely your model will treat 5 and 1 the same (since it's so over trained on examples with 1s)

GPT-4:

The statement contains several inaccuracies:

1. Multiple passes at lower learning rates: It's not entirely true that multiple passes with lower learning rates will produce identical results. Different learning rates can lead to different convergence properties, and multiple passes with lower learning rates can help in fine-tuning the model and potentially avoid overfitting by making smaller, more precise updates to the weights.
2. Overfitting as a mathematical property: Overfitting is indeed more of an empirical observation than a strict mathematical property. It is a phenomenon where a model learns the training data too well, including its noise and outliers, which harms its performance on unseen data. It's not strictly due to the size of the training data but rather the model's capacity to learn from it relative to its complexity.
3. Learning rate multiplied by the number of epochs: The learning rate and the number of epochs are both factors in a model's training process, but their product is not a direct measure of specialization. Instead, it's the learning rate's influence on weight updates over time (across epochs) that can affect specialization. Moreover, a model's capacity and the regularization techniques applied also significantly influence overfitting.
4. Example of learning 1 digit addition: The example given is somewhat simplistic and does not fully capture the complexities of overfitting. Overfitting would mean the model performs well on the training data (numbers with 1) but poorly on unseen data (numbers with 5). However, the example also suggests a sampling bias in the training data, which is a separate issue from overfitting. Sampling bias can lead to a model that doesn't generalize well because it hasn't been exposed to a representative range of the problem domain.

Overall, while the intention of the statement is to describe overfitting and the effects of learning rates, it conflates different concepts and could benefit from clearer differentiation between them.

Multiple passes at lower learning rates isn't supposed to produce different results.

Oh yes it is. The whole point of gradient descent is to slowly explore the dimensions of the gradient. With smaller steps you have a totally different trajectory than with bigger steps. And every pass makes you move.

If you choose a too small learning rate you often will indeed just move slower on the same path but a too big learning rate makes you skip entire paths.

OP seems to have been in that case with their first attempt.

I think that's not the whole story. The smaller increments can lead to "course changes" that would not have happened otherwise. Might let things slip into other local minima and all that. It's not just several small steps instead of one big one. The straight line that is the big step will become a curve, capable of bringing you into an entirely different place. The whole dataset can have its impact before some giant leaps jump into a single direction. As a laymen, maybe I've got this wrong, but I really don't see how you can categorically dismiss the possiblity of creating a much more robust and effective architecture instead of essentially jumping to conclusions and then somewhat fixing it up.

I think the idea has merits. How do you propose for others to contribute to the training dataset?