mattapperson

joined 11 months ago
[–] mattapperson@alien.top 1 points 11 months ago

This can kinda be done, but it’s not as simple as just that. You would need to also infer in many cases the prompt templates. Also many/most benchmarks are designed with untuned models in mind, meaning you typically need to add a system prompt/instructions… doing that also adds complexity because the best prompt for one model is likely different from the next. Also chat vs instruct vs base models in the same eval would be… meh. That said I think there is value in this and working on it as part of my cli tool with some warnings that the results might be less then quantitative

[–] mattapperson@alien.top 1 points 11 months ago

It’s just a side project for now in my free time. Started building it for my own sanity. But it’s not really in any shape that someone could just jump right in and help. So unless you’re a VC willing to throw money at me to make it my full time job lol… probably a couple weeks?

My goal is to make it not just a tool to run evals, but to create a holistic build, test, use toolkit to do everything from:

  • Cleaning datasets
  • Generating synthetic training data from existing data and files
  • Creating LoRAs and full fine tunes
  • Prompt evaluation and automated iterations
  • Running evaluations/benchmarks.

Trying to do all that in a way that is appreciable and easy to use and understand for your average software engineer, not just ai scientists. This stuff should require the setup of 20 libraries, writing all the glue code, or require knowing Python.

[–] mattapperson@alien.top 1 points 11 months ago

Yes custom evals are supported.

[–] mattapperson@alien.top 1 points 11 months ago (6 children)

I am working on just such a tool... but it's not ready yet. I am building a CLI tool that lets you just run `$ ai evals run humanevalsplus openhermes-2.5` and your good to go. Uses Llama.cpp

[–] mattapperson@alien.top 1 points 11 months ago (1 children)

Here is a new one I found the other day. Still seems to be WIP but overall I really like what is being done here - https://github.com/lobehub/lobe-chat