Naively averaging weights of models trained on disjoint datasets won’t work for LLMs or 1+ hidden layer DNNs
Why would simply aggregating the weights like this categorically fail to produce a reasonable model? Assuming of course that the datasets are all “the same” in some meaningful sense (e.g., equally representative of the same underlying X→Y mappings).
I’m in the same boat as you OP. Got in the back door with a basic:woefully inadequate skill set in 2019, all self taught, somehow into a research role lol. Gunning now for my second role and I think no have a chance but boy is it tough out there even with 4 YOE.
Have you to jump ship from you initial role, and when?
print("I understand. And how did that make you feel?")TBH I didn’t completely understand the question, but it is clear that you didn’t either.
Interesting. I love a good thought experiment :)
But what about the idea of bagging? As in aggregating multiple models together that have all been trained on different examples, and thus learned different things. Why is that not subject to similar criticism?