Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 11 months ago

MODERATORS

communick@academy.garden

Bill Gates told a German newspaper that GPT5 wouldn't be much better than GPT4: "there are reasons to believe that we have reached a plateau" [N] (www.handelsblatt.com)

submitted 10 months ago by we_are_mammals@alien.top to c/machinelearning@academy.garden

130 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] interesting-_o_-@alien.top 1 points 10 months ago (1 children)

Could you please share a citation for the mentioned research papers?

Last I looked into this, the hypothesis was that increasing parameter account results in a predictable increase in capability as long as training is correctly adapted.

https://arxiv.org/pdf/2206.07682.pdf

Very interested to see how these larger models that have plateaued are being trained!

[–] COAGULOPATH@alien.top 1 points 10 months ago

Could you please share a citation for the mentioned research papers?

I'm interested in seeing this as well.

He probably means that, although scaling might still deliver better loss reduction, this won't necessarily cash out to better performance "on the ground".

Subjectively, GPT4 does feel like a smaller step than GPT3 and GPT2 were. Those had crazy novel abilities that the previous one lacked, like GPT3's in-context learning. GPT4 displays no new abilities.* Yes, it's smarter, but everything it does was possible, to some limited degree, with GPT3. Maybe this just reflects test saturation. GPT4 performs so well that there's nowhere trivial left to go. But returns do seem to be diminishing.

(*You might think of multimodality, but they had to hack that into GPT4. It didn't naturally emerge with scale, like, say, math ability.)