LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Why didn't gpt4 work at first and how did they "fix it"? (alien.top)

submitted 11 months ago by Amgadoz@alien.top to c/localllama@poweruser.forum

11 comments fedilink hide all child comments

According to this tweet,

when gpt4 first finished training it didn’t actually work very well and the whole team thought it’s over, scaling is dead…until greg went into a cave for weeks and somehow magically made it work

So gpt-4 was kind of broken at first. Then greg spent a few weeks trying to fix it and then it somehow worked.

So why did it not work at first and how did they fix it?
I think this is an important question to the OSS community,

you are viewing a single comment's thread
view the rest of the comments

[–] maxinator80@alien.top 1 points 11 months ago (1 children)

Sam Altman mentioned that GPT4 is actually super difficult to work with. So I guess it simply isn't as straight forward as pushing in a prompt at the front and getting tokens out the back. Anything further would be speculation, but there must be something.

[–] CosmosisQ@alien.top 1 points 11 months ago

He's just alluding to the fact that most enterprise customers are too stupid to use base models as they expect to be interacting with a human-like dialogue-driven agent or chatbot rather than a supercharged text completion engine. It's a shame given that, used properly, the GPT-4 base model is far superior to the lobotomized version made generally available through the API.