I've only seen merging of same-upstream-pretrained-model-at-same-size.
At very least, you should be able to merge any 2 models with the same tokenizer via element-wise addition of the log probs just before sampling. This would also unlock creative new samplers. IE instead of adding logprobs, maybe one model's logprobs constrains the other's in interesting ways.
But, 2 models with same architecture and same dataset will be heavily biased in the same direction, even if you take 2 different finetunes, so this approach seems like it will have a low ceiling of potential.
Also, if you're just doing a linear interpolation of same-dimensioned weights, why not just collapse them all into a normal-sized model? IE 70B + 70B should still == 70B.
That said, you would get much more interesting models if you allowed mergers of different architectures, trained from different initializations, and with different datasets. I would think that the research on "token healing" would allow you to merge any 2 models, even if they have different tokenizers.
This seems like a cool way forward.
I agree, which is why I'm bearish on model merges, unless you're mixing model families (IE mistral + Llama).
These franken-merges are just interweaving finetunes of the same base model in a way that, it'd make more sense to me if they just collapsed all params into a same-sized model via element-wise interpolation. So, merging weights makes sense, but running params in parallel like these X-120B, there's no payout I can see in doing that beyond collapsing the weights.
If I prompt a frankenmerge with the usual instruct dreck I use, they fail to answer numerous questions in a useful manner. However, it's a different story using them in chat mode or probably anything creative - the outputs can be coherent but feel way less AI-like.