this post was submitted on 23 Nov 2023
1 points (100.0% liked)
LocalLLaMA
1 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 10 months ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
It's important that we not disclose all our test questions, or models will continue to overfit and underlearn. Now, to answer your question:
When evaluating a code model, I look for questions with easy answers, then tweak them slightly to see if the model gives the easy answer or figures out that I need something else. I'll give one example out of tens*:
Most of the models I've tested will give a correct answer to the wrong question: seek(1024) and truncate(). That removes everything after the first 1 KiB of the file.
(*I'm being deliberately vague about how many questions I have for the same reason I don't share them. Also it's a moving target.)