Trained on a larger # of tokens. All the llama models are under trained it appears, especially the 70b
this post was submitted on 16 Nov 2023
1 points (100.0% liked)
LocalLLaMA
4 readers
4 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 2 years ago
MODERATORS
They didn't lobotomize it for safety.
I second this. Mistral-7B gave me good results. After fine-tuning it's result is even better.
Mistral-7B gave me good results
Can you expand upon that? Do you mean in terms of its ability to write at a college level without major grammatical errors?
Are there notable finetunes to your knowledge? I've started using LLMs today, starting with openorca mistral 7B and it seems pretty good.
On HuggingFace you can find many fine-tuned/quantized models. Look for models from TheBloke on HuggingFace.
I assume the progress is based on well structured, high quality training data, combined with an incremental "learning schedule". At least that's where some reports of massive progress seem to be coming from and it's also very intuitive that this would help a lot.
Lack of censorship is a key factor as it maximises the predictive abilities of the model.
It doesn’t seem too capable. Has anyone else tried running this locally or on runpod?
The results are okay, but I'm hard-pressed to call it "very capable". My perspective on it is that other bigger models are making mistakes they shouldn't be making because they were "trained wrong".
I'm guessing GQA helped. Llama2 70b and 34b used Grouped Query Attention but it wasn't used for Llama2 7/13b.
Knowledge is a strange goal for any model when we have the internet. IMO. Just connect your model to a web search.
My current hunch is that they use a lot of non easily accessible online ressources (including a specific archive owned by someone named Anna).
oh, anna !
Is there any version of mistral or llama2 with RHLF applied to make tasks of text summarisation without having the censorship. Sometimes the output is totally different from what one could expect with the input sentences. Even if I state in the prompt to avoid applying censorship and focus on the input.
Do people find that it holds up in use? Or are we mostly going on benchmarks? I’m skeptical of benchmarks, and a highly performant 7B model would be of great use.
French qualité. Yes, this is a thing now. Get used to it. HuggingFace is french too.
They matched parameters and tokens when training.
Podcast on Spotify "No Priors" has the CEO of Mistral on who discusses this.
I don’t know what this means but will listen to the podcast to find out
Why can we get a 20 - 34b version of this very capable Mistral?
Having used it a lot, I can say for sure that without much prompting it readily produces junk web text, urls etc, so it is not a fully filtered or fully synthetic dataset.
My guess would be that it's just 'a bit better filtered than llama-2', and maybe slightly more trained on that set. Slightly better quality set, slightly more trained on that set.
My intuition based on this, is that per parameter size EVERYTHING open source could be optimized considerably more.
As far as I know (I might be wrong) it's partly the team that made llama1 (and maybe made the first steps for llama2?). So they already knew what they were doing. How llama could be improved* and so on.
*The dataset
It's simply the time bonus - coming after all the big models.
- better filtering - kill outright junk
- you use already big models (OpenAI and LLama) that you can use for data tuning and filtering
- use available synthetic data
We’re only in the first inning too. Buckle up