this post was submitted on 27 Nov 2023
1 points (100.0% liked)
LocalLLaMA
1 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 10 months ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I've seen the "... beats GPT-4" enough times that now whenever I see a title that suggests a tiny model can compete with GPT-4 I see it as a negative signal; that the authors are bullshitting through some benchmarks or some other shenanigans.
It's annoying because the models might be legitimately good models for being open and within their weight class but now you've put my brain in BS detecting mode and I can't trust you've done good faith measurement anymore.
Yeah I dont think authors are intentionally bullshitting or intentionally doing "benchmark cosmetics", but maybe it's more lack of knowledge on whats going on in terms of (most of) benchmarks and their the image that has become ruined in the meantime.
Sure, but name-dropping the biggest name in the game and comparing yourself favourably to it is a big swing. It's either a naive at best marketing claim or it's untrue.
There are SO many models "bullshitting through some benchmarks or some other shenanigans" that I'm cooking my own benchmark system LOL.
Yeah I just roll my eyes and continue onwards