Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 11 months ago

MODERATORS

communick@academy.garden

Bill Gates told a German newspaper that GPT5 wouldn't be much better than GPT4: "there are reasons to believe that we have reached a plateau" [N] (www.handelsblatt.com)

submitted 10 months ago by we_are_mammals@alien.top to c/machinelearning@academy.garden

130 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] jugalator@alien.top 1 points 10 months ago (4 children)

Research papers have also observed diminishing returns issues as models grow.

Hell maybe even GPT-4 was hit by this and that's why GPT-4 is not a single giant language model but running a mixture of experts design of eight 220B models trained for subtasks.

But I think even this architecture will run into issues and that it's more like a crutch. I mean, you'll eventually grow each of these subtask models too large and might need to split them as well, but this might mean you run into too small/niche fields per respective model and that sounds like the end of that road to me.

[–] therealnvp@alien.top 1 points 10 months ago

You should double check what mixture of experts actually means 🙂

load more comments (3 replies)