mlabonne

joined 11 months ago
sorted by: new top controversial old
NeuralHermes-2.5: Boosting SFT models' performance with DPO in c/localllama@poweruser.forum
[–] mlabonne@alien.top 1 points 11 months ago (1 children)

Yes, I'd say it'd probably work better than the current approach. If you look at the reward plots on wandb, it feels like the problem is too easy for the model, hence slight improvement.

https://preview.redd.it/xhuyiquojg3c1.png?width=2398&format=png&auto=webp&s=67725747e6cd9254e38728149fb6cea3ba85d71e

NeuralHermes-2.5: Boosting SFT models' performance with DPO in c/localllama@poweruser.forum
[–] mlabonne@alien.top 1 points 11 months ago

It's a good question, I can give it a try. Ideally, you'd want a 16k version of the preference dataset to make sure that DPO doesn't ruin it. But considering the low number of training samples, it probably works fine.

1
NeuralHermes-2.5: Boosting SFT models' performance with DPO (alien.top)
 

​

https://preview.redd.it/oa22kq7vod3c1.jpg?width=1792&format=pjpg&auto=webp&s=4ce176c4f1a4b988b2202a359c67505d759dfd9e

I just released the NeuralHermes-2.5-Mistral-7B model, which is a DPO fine-tuned version of OpenHermes-2.5-Mistral-7B. Teknium, the creator of the SFT model, confirmed on Twitter that this version improves benchmark scores in AGIEval, GPT4All, and TruthfulQA.

Take is a simple proof of concept: I used Intel's orca_dpo_pairs (from neural-chat-7b-v3-1) in a ChatML format, and only trained it for one hour on an A100 (using Goole Colab). But it shows the potential of DPO to boost the performance of SFT models, basically for free. I released all the code so that everyone can easily experiment with it and find better parameters (it shouldn't be difficult). You can also access the W&B project.

Note that the preference dataset is also entirely synthetic, with preferred answers coming from GPT-4/3.5 and rejected responses coming from Llama 2 13b chat. It's a very cheap and efficient way to convert an instruction dataset (OpenOrca in this case) into a preference dataset. I wasn't very successful in my previous experiments with DPO using other datasets, so I think there's something very interesting with this one. We can easily reproduce this dataset and improve it with other sources.

I just wanted to share these thoughts and experiments with the community. I'm writing an article about DPO and this model is actually a lucky by-product of it. I'll share it when it's ready.

If you want to try the model, TheBloke already provided GGUF and AWQ versions of it.

ExLlamaV2: The Fastest Library to Run LLMs in c/localllama@poweruser.forum
[–] mlabonne@alien.top 1 points 11 months ago

Thanks for your excellent library! It makes sense because I started writing this article about two months ago (chatcode.py is still mentioned in the README.md by the way). I had a very low throughput using ExLlamaV2 without flash-attn-2. Do you know if it's still the case? I updated these two points, thanks for your feedback.

ExLlamaV2: The Fastest Library to Run LLMs in c/localllama@poweruser.forum
[–] mlabonne@alien.top 1 points 11 months ago (2 children)

I'm the author of this article, thank you for posting it! If you don't want to use Medium, here's the link to the article on my blog: https://mlabonne.github.io/blog/posts/ExLlamaV2_The_Fastest_Library_to_Run%C2%A0LLMs.html