Hi everyone, I'd like to share something that I've been working on for the past few days: https://huggingface.co/nsfwthrowitaway69/Venus-120b-v1.0
This model is the result of interleaving layers from three different models: Euryale-1.3-L2-70B, Nous-Hermes-Llama2-70b, and SynthIA-70B-v1.5, resulting in a model that it larger than any of the three used for the merge. I have branches on the repo for exl2 quants at 3.0 and 4.85 bpw, which will allow the model to run in 48GB or 80GB of vram, respectively.
I love using LLMs for RPs and ERPs and so my goal was to create something similar to Goliath, which is honestly the best roleplay model I've ever used. I've done some initial testing with it and so far the results seem encouraging. I'd love to get some feedback on this from the community! Going forward, my plan is to do more experiments with merging models together, possibly even going even larger than 120b parameters to see where the gains stop.
I could not get any of the quants loaded, looks like the config is looking for XX of 25 safetensors
with exl2-3.0bpw having only XX of 06 safetensors
๐ค How are you trying to load it? I tested both quants in text-generation-webui and they worked fine for me. I used exllama2_hf to load it
Models on ooba without "exl" on the folder name will redirect to transformers by default, so that may be the reason he got that by default.
Defaulted to transformers, loaded right away in ExLlamav2_HF, thank you I didn't know what I don't know.