Sorry if this is too off topic, but I was browsing the twitter verse and stumbled upon some very interesting slides going over the basics of LLMs by an early engineer at OAI (He's first name on a few of their early papers?). Unfortunately, I closed the tab and lost the slides.
Does anyone happen to know off the top of their head which slides these are? They started by going over the basics of language representations (bag of words, vec, etc) and embeddings ... I didn't get past there before the unfortunate closing.
ermm, idk what you mean by any of those words.
Auto-regressive just means it's a time series that depends on its previous predictions.
So, when you predict a token at time t -- you condition on the previous tokens you already predicted.
Consider, "the cat in the hat". A transformer that predicted it would have predicated it in the following manner (assuming that each of the words are a token bc I'm lazy):
-P("the"|prompt) is highest
-P("cat"|"the",prompt) is highest
-P("in"|"the","cat",prompt) is highest
So you can see there is a dependency between each of its predictions and the next prediction. This is what is meant by auto-regressive.