It's inevitable people will game the system when it's so easy, and the payoff can be huge. Not so long ago people could still get huge VC checks for showing off GitHub stars or benchmark numbers.
this post was submitted on 14 Nov 2023
1 points (100.0% liked)
LocalLLaMA
1 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 10 months ago
MODERATORS
To be fair, it's pretty clear that openai update their models with every kind of test people throw at them as well.
if you're interested in running your own models for any reason, you really should build your own evaluation dataset for the scenarios you care about.
at this point, all the public benchmarks are such a mess. Do you really care if the model you select has the highest MMLU? Or, do you care only that it's the best-performing model for the scenarios you actually need?
This seems to me at least like the most logical conclusion. I’m currently working on developing some level of moral/ethical dilemma scenarios to interpret different perspectives and response strategies, for my personal use cases of discussion and breaking down topics into manageable levels and then exploring the nuances, it is very effective. Seems to be far too broad of a “use case” to define one set of benchmarks unless it’s incredibly comprehensive and refined over and over as trends develop
With the abundance of models, most developers and users have to select a small subset of available models for own evaluation, and that has to be based on some already available data about models' performance. At that stage, selecting models with, for example, highest MMLU score is one way to go about it.
Huh...I figured this has already been happening for a while on closed dataset LLMs. The leaderboard has not directly indicated a models ability to do real-world work from my experience. Some of the lower ranking models seem to do better with what I put them through than the top ranking models. Just my personal opinion and observation.
When a measure becomes a target, it ceases to be a good measure
yeah people praising 7b and 13 b models here and there, but....they just hallucinate! Then 120b goliath, no matter how terrible its initial idea was, is just really good in normal conversations. Im trying to love giga praised open hermes 2.5 and other mistral finetunes, but they are just better next-token-predictors, unlike larger models which are actually able to reason.
Benchmark test questions can't be made public. It's too easy to cheat.