I've only seen merging of same-upstream-pretrained-model-at-same-size.
Not anymore.
Here's a merge of llama 2 13B and llama 1 33B https://huggingface.co/chargoddard/llama2-22b
this post was submitted on 28 Nov 2023
1 points (100.0% liked)
LocalLLaMA
3 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 1 year ago
MODERATORS
How does this work? Like I'm really confused at a conceptual level on how you merge models with different numbers of different sized layers.
reading the readme, it sounds like they're running some attention heads that were either already same-dimensioned across both models, or, they may have included a linear projection layer to accomplish it. Then, they say they trained on 10M tokens to "settle in the transplant", which doesn't sound like enough to me, and they concur this model isn't useful until further training.
https://huggingface.co/chargoddard/llama2-22b/blob/main/frankenllama_22b.py shows how the tensors are padded up to fit
Wonder how L1 65b would do with L2 70b.
Not for the kind of merging I've seen. But I remember a paper back in the day that suggested you could find high-dimensional axes within different models, and if you rotated the weights to align, you could merge different models to your advantage, and maintain knowledge from both seed models. This included models that were trained from different initializations.
I think that the only reason this franken-merging works is because people are mostly just merging finetunes of the same base, so these high-d vectors are already aligned enough that the mergers work.
At very least, you should be able to merge any 2 models with the same tokenizer via element-wise addition of the log probs just before sampling. This would also unlock creative new samplers. IE instead of adding logprobs, maybe one model's logprobs constrains the other's in interesting ways.
What, run two models at once? This doesn't seem cost-effective for what you'd get.
Most merges that are popular are weight mixes, where portions of different models are averaged in increasingly complex ways. Goliath is a layer splice, sections of Xwin and Euryale were chopped up and interweaved together. This is the kind of merge I'm interested in but getting useful models out of the process is way more art than science.
Call it a voodoo, not an art.
I mean, voodoo and all forms of magic with a k are basically art in my opinion.
This doesn't seem cost-effective for what you'd get.
I agree, which is why I'm bearish on model merges, unless you're mixing model families (IE mistral + Llama).
These franken-merges are just interweaving finetunes of the same base model in a way that, it'd make more sense to me if they just collapsed all params into a same-sized model via element-wise interpolation. So, merging weights makes sense, but running params in parallel like these X-120B, there's no payout I can see in doing that beyond collapsing the weights.
If I prompt a frankenmerge with the usual instruct dreck I use, they fail to answer numerous questions in a useful manner. However, it's a different story using them in chat mode or probably anything creative - the outputs can be coherent but feel way less AI-like.
Git rebasin claims to do this.
But its untested on large models. There is a branch for it in mergekit, as well as a stable diffusion implementation (which works fantastically as a regular merger).
rebasin! I was trying to recall this, thank you. Can it mix model families, do you know? I thought it was just for identical architectures.