I know AI/LLM hate is strong here, so this is going to get some blow back. But there's a lot of Linux folk on here, so let me frame it this way....
My understand of the Linux/unix design philosophy is building small, efficient programs that do a limited set of tasks very well and that can be strung together with other programs that do other tasks very well. This is in opposition to the " be everything" program concept of Windows and Microsoft Office Suite. At least this is how would describe the difference to non technical friends: Nothing you think of as your OS in windows is actually what Linux is replacing. You're getting the Linux kernel packaged up in a distro that combines a bunch of smaller pieces (file explorer, window manager, etc) that you can still customize from there.
When I look at the approach to AI, I see the same thing. I've dabbled enough in ML/LLMs to know that LLMs are effectively very fancy next word predictors or for the case of image/video GenAI, next pixel predictors. As others have said countless times, there's no consciousness or understanding of the context, but you can ask it things in natural language and it will try to produce whatever you asked for in the same app regardless of context.
From a science project standpoint, this is cool, but it doesn't seem scalable or consistently reproducable and the energy use and easily found blunders seem to support that thought.
So, my question is why is no one building AI with a Linux philosophy? Small purpose built ML models with a language processing/triage model on top? Oh this person has a question about history, send them to the history module. This person wants to edit a photo, send them to the photo editing module. Then let those modules dig deeper from there. That's how we do customer service with real people after all. With this way we could refine each specialization individually instead of having a giant model that consumes tons of resources and is error prone.
"Attention is all you need" is the place to start that question, as this is where the transformer gets introduced, and it's authored by the team at google brain. Notably, not OpenAi, who later authored a paper introducing gpt-1. Transformers were introduced as a way to shrink the model and support parallelization, not to make it larger.
Convolutional networks, lstms, kernel based vision models, unet all of that had already existed before, and yes, people had just thrown more complexity at the matter myself included, but it never resulted in the kind of pay off that transformers seemed to have been able to achieve.
So it's not like the community hadn't tried just throwing more compute or scale at the issues, it's just that it didn't result in this kind of emergent complexity that we've seen with transformers. And that's true if networks in general. There is no guarantee that throwing "just more complexity" at the system is going to result in different properties. But there is also no guarantee it won't. Practitioners of complex systems analysis understand this, that there are no guarantees regarding boundary conditions in complex systems.
And this is the second leg of support for unexpected, because if it was to be expected, why not just go there? And we can see that with the relatively asymptomatic performance we see in frontier llms. They are still improving. But marginally compared to the massive jumps we saw, for example between gpt1 and gpt2, or gpt 2 and gpt 3. Even at gpt 3 and gpt 4 we saw that asymptote beginning to form. And since four, improvements have been very meh, inspite of just throwing more parameters at the problem. Things have been improving but it's largely around the engineering around the models, not the models themselves, inspite of throwing more and more complexity at the problem.
And maybe at one trillion parameters, there is some new boundary conditions which result in new emergent properties. But we don't know that. So if we go there and find that out, it would be unexpected, at least in that we've been throwing exponential complexity at a problem to get sublinear performance improvements.
Attention is all you need: https://arxiv.org/pdf/1706.03762
Gpt-1: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
Complex systems: https://onlinelibrary.wiley.com/doi/10.1155/2020/6105872