Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 1 year ago

MODERATORS

communick@academy.garden

[D] The Status of Open Source Code LLMs (alien.top)

submitted 11 months ago by RealAGIFan@alien.top to c/machinelearning@academy.garden

3 comments fedilink hide all child comments

I've been pondering something recently. Did you notice that achieving over 70% on the well-known HumanEval pass@1 hasn't been making major headlines? Models like WizardCoderV2, Phind, Deepseek, and XwinCoder have all surpassed the 67% reported in GPT-4’s report. Some of them are even closely tailing the 82% of GPT-4 API’s. So, are these models really performing that well?
Here's something intriguing: I found this image in the latest release of XwinCoder’s repo: Xwin-LM/Xwin-Coder at main · Xwin-LM/Xwin-LM (github.com)

Results in XwinCoder repo

It shows that GPT-4 achieves a 60% pass@1 on APPS-introductory, which is higher than CodeLLaMA-34B’s pass@100 (56.3) and XwinCoder-34B’s pass@5 (43.0). Interesting, isn't it?
This suggests that judging a model based on a single benchmark might not provide the full picture. This leads me to a couple of questions:

What exactly is the gap here? How can we definitively say one model outperforms another?
How are other recent models performing on benchmarks like APPS and DS1000?

I'm interested in hearing your thoughts on this. Has anyone experimented with these new models? What was your experience like?

you are viewing a single comment's thread
view the rest of the comments

[–] Caffeine_Monster@alien.top 1 points 11 months ago

I haven't got round to trying the xwin coder models, but the precursor 70b chat model was extremely impressive when compared against both chat GPT 3.5 and 4.