Dorialexandre

joined 1 year ago
sorted by: new top controversial old
Why is Mistral-7b so capable? Any ideas re: dataset? in c/localllama@poweruser.forum
[–] Dorialexandre@alien.top 1 points 1 year ago (1 children)

My current hunch is that they use a lot of non easily accessible online ressources (including a specific archive owned by someone named Anna).

MonadGPT, an early modern chatbot trained on Mistral-Hermes and 17th century books. in c/localllama@poweruser.forum
[–] Dorialexandre@alien.top 1 points 1 year ago

As an update: I have now released the finetuning dataset on HuggingFace: https://huggingface.co/datasets/Pclanglais/MonadGPT

Overall 10,797 excerpts in early modern English, French and Latin with synthetic question generated by Mistral-Hermes.

MonadGPT, an early modern chatbot trained on Mistral-Hermes and 17th century books. in c/localllama@poweruser.forum
[–] Dorialexandre@alien.top 1 points 1 year ago

Well that was actually my original motivation for finetuning. Even GPT-4 is not so good with a proper prompt: the text feels fake and/or struggle to maintain cultural consistency. I think finetuning works better for this task, as there are too many directives to give and it helps to relieve the model from anachronistic RLHF.

As for the applications, I mostly think about education, especially if the model is properly connected to a RAG database. Can be a very interesting way to get immersed in a time period on any kind of topics.

MonadGPT, an early modern chatbot trained on Mistral-Hermes and 17th century books. in c/localllama@poweruser.forum
[–] Dorialexandre@alien.top 1 points 1 year ago

Link to the ongoing demo for MonadGPT, with generous GPU support from HuggingFace : https://huggingface.co/spaces/Pclanglais/MonadGPT

The model has been published as well (and soon the dataset): https://huggingface.co/Pclanglais/MonadGPT?text=Hi.

1
MonadGPT, an early modern chatbot trained on Mistral-Hermes and 17th century books. (alien.top)