this post was submitted on 25 Nov 2023
1 points (100.0% liked)
Machine Learning
1 readers
1 users here now
Community Rules:
- Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
- Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
- Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
- Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.
founded 11 months ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Research papers have also observed diminishing returns issues as models grow.
Hell maybe even GPT-4 was hit by this and that's why GPT-4 is not a single giant language model but running a mixture of experts design of eight 220B models trained for subtasks.
But I think even this architecture will run into issues and that it's more like a crutch. I mean, you'll eventually grow each of these subtask models too large and might need to split them as well, but this might mean you run into too small/niche fields per respective model and that sounds like the end of that road to me.
I mean, that is literally how any form of AGI will work. No one in the field has ever thought one model will be capable of reaching AGI. All these models are highly specialized for the task in which they are trained. Any move towards an AGI will be getting many of these highly specialized AI's to work in conjunction with one another. Much like how our own brains work.
You should double check what mixture of experts actually means 🙂
Could you please share a citation for the mentioned research papers?
Last I looked into this, the hypothesis was that increasing parameter account results in a predictable increase in capability as long as training is correctly adapted.
https://arxiv.org/pdf/2206.07682.pdf
Very interested to see how these larger models that have plateaued are being trained!
I'm interested in seeing this as well.
He probably means that, although scaling might still deliver better loss reduction, this won't necessarily cash out to better performance "on the ground".
Subjectively, GPT4 does feel like a smaller step than GPT3 and GPT2 were. Those had crazy novel abilities that the previous one lacked, like GPT3's in-context learning. GPT4 displays no new abilities.* Yes, it's smarter, but everything it does was possible, to some limited degree, with GPT3. Maybe this just reflects test saturation. GPT4 performs so well that there's nowhere trivial left to go. But returns do seem to be diminishing.
(*You might think of multimodality, but they had to hack that into GPT4. It didn't naturally emerge with scale, like, say, math ability.)