PS. This is Text from Bing AI.
one week to train it
six months to finetune it until it's safely and ethically useless
this post was submitted on 09 Nov 2023
1 points (100.0% liked)
LocalLLaMA
4 readers
4 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 2 years ago
MODERATORS
Maybe. If it won't be killed because safety. I also think that it's implausible it'll be much better than what we have, or even catch up to 3.5 fully.
Well, two areas where Llama can improve are:
- Multilingual capabilities
- Mixture of Experts architecture
Already Mistral 7B fine tunes reaching parity with gpt-3.5 in most benchmarks.
I’d be very surprised if Llama-3 70B fine tunes don’t significantly outperform GPT-3.5 in nearly every metric.
All they need to do is make it 180B and most people will have no way to abuse it. (Or use it)
Llama 3 180B model on every quantization & compression steroid available might bang. Also it will show how far or close oss llms are from gpt4
Assuming it will be open.
Awesome. I hope there's a 30b~ model
That's the prevailing idea based on all the info we have so far:
- Llama 1 training was from around July 2022 to January 2023, Llama 2 from January 2023 to July 2023, so Llama 3 could plausibly be from July 2023 to January 2024.
- In August, a credible rumor from an OpenAI researcher claimed that Meta talked about having the compute to train Llama 3 and Llama 4, with Llama 3 being as good as GPT-4.
- In an interview with Lex Fridman published Sept. 28, Mark Zuckerberg has said they're always training another model and already working on the next generation when talking about Llama.
- At Meta Connect on Sept. 27 - 28, they said more news about Llama will be put out next year.
WSJ published an exclusive on Sept. 10 that said Meta's next LLM won't start training until early 2024, meaning a release wouldn't happen until much later, but they may have been mistaken since this seems to contradict Mark's recent words. Meta could have also accelerated their plans to stay relevant in the LLM race, especially since leaks about their LLM development have shown they've put more emphasis on productizing Llama and incorporating it within their apps.
Dell's LLM-for-businesses plan is a joke, btw. They seem to not know that quantized models even exist, or perhaps they are pretending to so that their customers have to buy more Dell hardware.
Half of the regulars in this sub could set up a business with a better in-house LLM inference system than what Dell's offering.
Personally I'm waiting for the Llama Box 2 - Zuckerberg Signature Edition.
Heard the manufacturing plant burned down
it should have a plug in system making fine tuning a breeze
See what the source is
I tried coaxing an answer out of it, and as far as I got was
- One random redditors comment saying "next year" was announced at Meta Connect
- "It makes sense" due to the spacing between 1 and 2
My speculation is that the "safety" of major LLM providers like Facebook won't be substantial. They are probably designed so that "popping the lock" isn't difficult. A fig leaf and shrug to placate outsiders, while allowing their actual audience to carry on.
I'm not sure that it's plausible to add meaningful safety to open source LLMs.
Incidentally, that does worry me some in the long term.
Ooooh, I really want to see the marvelous "mechanism" to prevent open source model from misuse.
*Provided you use the non fine tuned model straight from Meta's download page so that they're safe.
They can ingrain those refusals pretty deep to where it will be irritating to use and hard to fine tune out. Vicuna has a bit of this.
The point is Meta till now always also released base models.
That would require them to release only fine-tuned model like llama2-chat was. Then they can bork it which would be irreversible. But if they give out base model without some stuff - it can be easily added.
I'm more interested in the next Mistral release, none of that corporate "safety" bs. It would also be good to have a truly open-source model (that releases both weight and the training data).
The funny things the next mistral rumored not open-source
What are these imaginary 'abuses' this random picture of some text is talking about?
Bing AI as a source is shit. Many times the quoted primary sources don't support whatever Bing AI claims.
Always check Bing’s sources. It hallucinates out the wazoo and won’t change its mind if corrected or confronted.
Putting "safety" mechanism in foundational models is dumb imo. They are not just text generators, they are statistical models about human languages, and it shouldn't have made up arbitrary biases about what language should look like.
It's not hard to fine tune base models for any bias you want. "Zero bias" isn't possible. There's always some bias in the training data.
Sure, but that's not a reason to purposefully add more biases into it.
"Meta had partnered with Dell" Oh so if you hit a key to hard while inputting a prompt the entire model will break?
YES! This is the good news I needed this morning!
Ys