this post was submitted on 24 Nov 2023
1 points (100.0% liked)

Machine Learning

1 readers
1 users here now

Community Rules:

founded 1 year ago
MODERATORS
 

As said in the title I’m curious if grokking has been proven to happen with llm, could it be the case with gpt-4?

top 3 comments
sorted by: hot top controversial new old
[–] sciencesebi3@alien.top 1 points 11 months ago

How... would that happen?

[–] yannbouteiller@alien.top 1 points 11 months ago

Am I correct to say that "grokking" is apparently an effect of regularization, as in reaching good generalization performance from pushing the weights to be as small as possible until the model reaches a capacity that is smaller than the dataset?

[–] Alt-Depixelator-777@alien.top 1 points 11 months ago

"...grass groks being walked on..." llms do not grok, nor do they grok grokking, but mainly, they do not grok "not grokking"