this post was submitted on 09 Nov 2023
1 points (100.0% liked)

LocalLLaMA

1 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago
MODERATORS
communick@poweruser.forum
1
GPT-4's 128K context window tested (alien.top)
submitted 10 months ago by Ok_Relationship_9879@alien.top to c/localllama@poweruser.forum
6 comments fedilink hide all child comments
 

This fella tested the new 128K context window and had some interesting findings.

* GPT-4’s recall performance started to degrade above 73K tokens

* Low recall performance was correlated when the fact to be recalled was placed between at 7%-50% document depth

* If the fact was at the beginning of the document, it was recalled regardless of context length

Any thoughts on what OpenAI is doing to its context window behind the scenes? Which process or processes they're using to expand context window, for example.

He also says in the comments that at 64K and lower, retrieval was 100%. That's pretty impressive.

https://x.com/GregKamradt/status/1722386725635580292?s=20

top 6 comments
sorted by: hot top controversial new old
[–] blaselbee@alien.top 1 points 10 months ago

Is the web version of ChatGPT 128k, or just via the api?

[–] Familiar_Yak3962@alien.top 1 points 10 months ago

Meta's lm-infinite has similar attention properties.

[–] Tiny_Arugula_5648@alien.top 1 points 10 months ago

Their needle in a haystack test isn't very compelling. Sure no test is flawless but a random out of context fact placed at different points in the context window there is a lot of reasons why the model would fail to retrieve that.

[–] doppelkeks90@alien.top 1 points 10 months ago

So what are the implications in real day useage?

  • It's able to retrieve every information from at least 65k if it's small enough.

  • What are the results with bigger chunks to be retrieved?

  • Is it able to process all of the 64k tokens in order to generate an answer that takes all the 64k into account.

For sure it's interesting but many more test are needed to be done to have a full picture of the real capabilities.

[–] Distinct-Target7503@alien.top 1 points 10 months ago

Someone compared that with Claude 2 100K?

Also, gpt4 32K have same 100% accuracy in all its context? Is that 64 on 180 "absolute" or relative?

[–] ArtifartX@alien.top 1 points 10 months ago
  • If the fact was at the beginning of the document, it was recalled regardless of context length

Lol at OpenAI adding a cheap trick like this, since they know the first thing people will test at high context lengths is recall from the beginning.