I know AI/LLM hate is strong here, so this is going to get some blow back. But there's a lot of Linux folk on here, so let me frame it this way....
My understand of the Linux/unix design philosophy is building small, efficient programs that do a limited set of tasks very well and that can be strung together with other programs that do other tasks very well. This is in opposition to the " be everything" program concept of Windows and Microsoft Office Suite. At least this is how would describe the difference to non technical friends: Nothing you think of as your OS in windows is actually what Linux is replacing. You're getting the Linux kernel packaged up in a distro that combines a bunch of smaller pieces (file explorer, window manager, etc) that you can still customize from there.
When I look at the approach to AI, I see the same thing. I've dabbled enough in ML/LLMs to know that LLMs are effectively very fancy next word predictors or for the case of image/video GenAI, next pixel predictors. As others have said countless times, there's no consciousness or understanding of the context, but you can ask it things in natural language and it will try to produce whatever you asked for in the same app regardless of context.
From a science project standpoint, this is cool, but it doesn't seem scalable or consistently reproducable and the energy use and easily found blunders seem to support that thought.
So, my question is why is no one building AI with a Linux philosophy? Small purpose built ML models with a language processing/triage model on top? Oh this person has a question about history, send them to the history module. This person wants to edit a photo, send them to the photo editing module. Then let those modules dig deeper from there. That's how we do customer service with real people after all. With this way we could refine each specialization individually instead of having a giant model that consumes tons of resources and is error prone.
I mean people are. All the time. They just don't get the attention things like lmk have.
For example, SAM 3 exactly what I think you are asking, but for images.
But there is another point in here about how"actually, just bigger model better" and that's the thing with transformers. Them basically becoming chat bots through clever training and massive size and training datasets wasnt expected. You don't get that behavior from much smaller transformers. And so there was an apparently emergent phenomena in this case. A small network isn't going to do what you think it is going to do precisely because it's over constrained.
Thanks, I think this is what I'm getting at. Is there an inherent advantage to all in one over modular? And it sounds like they're is. I know over constraining is an issue with training and there is no scenario with ML or LLM where you get to 100% accuracy. It's just not the point of the technology. But I could focus on getting an image editing tool 95-99% of the way there and test that vs. having that functionality bundled up with everything else and potentially have that function suffer as we improve another area. If a bigger transformer is benefiting from the other areas of expertise, that is interesting. I still believe you have to hit a point of diminishing returns where more bigger no longer equals more better
So I have a book on my shelf on complex systems analysis and that might be a place to start, but this concept of emergent properties in complex systems isn't a new one, and it's well established in complex systems theory, and especially true in network and graph theory.
Basically, complex systems, and especially networked systems, develop different emergent properties as they scale.
Do you have a source for 'unexpected'?
"Attention is all you need" is the place to start that question, as this is where the transformer gets introduced, and it's authored by the team at google brain. Notably, not OpenAi, who later authored a paper introducing gpt-1. Transformers were introduced as a way to shrink the model and support parallelization, not to make it larger.
Convolutional networks, lstms, kernel based vision models, unet all of that had already existed before, and yes, people had just thrown more complexity at the matter myself included, but it never resulted in the kind of pay off that transformers seemed to have been able to achieve.
So it's not like the community hadn't tried just throwing more compute or scale at the issues, it's just that it didn't result in this kind of emergent complexity that we've seen with transformers. And that's true if networks in general. There is no guarantee that throwing "just more complexity" at the system is going to result in different properties. But there is also no guarantee it won't. Practitioners of complex systems analysis understand this, that there are no guarantees regarding boundary conditions in complex systems.
And this is the second leg of support for unexpected, because if it was to be expected, why not just go there? And we can see that with the relatively asymptomatic performance we see in frontier llms. They are still improving. But marginally compared to the massive jumps we saw, for example between gpt1 and gpt2, or gpt 2 and gpt 3. Even at gpt 3 and gpt 4 we saw that asymptote beginning to form. And since four, improvements have been very meh, inspite of just throwing more parameters at the problem. Things have been improving but it's largely around the engineering around the models, not the models themselves, inspite of throwing more and more complexity at the problem.
And maybe at one trillion parameters, there is some new boundary conditions which result in new emergent properties. But we don't know that. So if we go there and find that out, it would be unexpected, at least in that we've been throwing exponential complexity at a problem to get sublinear performance improvements.
Attention is all you need: https://arxiv.org/pdf/1706.03762
Gpt-1: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
Complex systems: https://onlinelibrary.wiley.com/doi/10.1155/2020/6105872