Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 2 years ago

MODERATORS

communick@academy.garden

[D] The Status of Open Source Code LLMs (alien.top)

submitted 2 years ago by RealAGIFan@alien.top to c/machinelearning@academy.garden

3 comments fedilink hide all child comments

I've been pondering something recently. Did you notice that achieving over 70% on the well-known HumanEval pass@1 hasn't been making major headlines? Models like WizardCoderV2, Phind, Deepseek, and XwinCoder have all surpassed the 67% reported in GPT-4’s report. Some of them are even closely tailing the 82% of GPT-4 API’s. So, are these models really performing that well?
Here's something intriguing: I found this image in the latest release of XwinCoder’s repo: Xwin-LM/Xwin-Coder at main · Xwin-LM/Xwin-LM (github.com)

Results in XwinCoder repo

It shows that GPT-4 achieves a 60% pass@1 on APPS-introductory, which is higher than CodeLLaMA-34B’s pass@100 (56.3) and XwinCoder-34B’s pass@5 (43.0). Interesting, isn't it?
This suggests that judging a model based on a single benchmark might not provide the full picture. This leads me to a couple of questions:

What exactly is the gap here? How can we definitively say one model outperforms another?
How are other recent models performing on benchmarks like APPS and DS1000?

I'm interested in hearing your thoughts on this. Has anyone experimented with these new models? What was your experience like?

top 3 comments

sorted by: hot top controversial new old

[–] Disastrous_Elk_6375@alien.top 1 points 2 years ago

This suggests that judging a model based on a single benchmark might not provide the full picture.

Duh... This has been a recurring problem with all these "benchmark leaderboards". It turns out that "training on the testing set is all you need"...

[–] Caffeine_Monster@alien.top 1 points 2 years ago

I haven't got round to trying the xwin coder models, but the precursor 70b chat model was extremely impressive when compared against both chat GPT 3.5 and 4.

[–] koolaidman123@alien.top 1 points 2 years ago

If you look at something like evolinstruct data its so similar to humane al itd be a surprise if models trained on that data (or other synthetic data) dont perform well

As a rule of thumb i only generally trust base models (even then its iffy) on benchmarks and for finetuned models only by using it