this post was submitted on 24 Nov 2023
1 points (100.0% liked)

Machine Learning

1 readers
1 users here now

Community Rules:

founded 1 year ago
MODERATORS
 

As said in the title I’m curious if grokking has been proven to happen with llm, could it be the case with gpt-4?

you are viewing a single comment's thread
view the rest of the comments
[–] yannbouteiller@alien.top 1 points 1 year ago

Am I correct to say that "grokking" is apparently an effect of regularization, as in reaching good generalization performance from pushing the weights to be as small as possible until the model reaches a capacity that is smaller than the dataset?