LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

Any Easy and Local Way to Run Benchmarks? (alien.top)

submitted 11 months ago by xadiant@alien.top to c/localllama@poweruser.forum

7 comments fedilink hide all child comments

I want to see if some presets and custom modifications work well in benchmarks, but running HellaSwag or MMLU looks too complicated for me, and it takes 10+ hours to upload 20GBs of data.

I assume there isn't a convenient webui for chumps to run benchmarks with (apart from ooba perplexity, which I assume isn't the same thing?). Any advise?

you are viewing a single comment's thread
view the rest of the comments

[–] mattapperson@alien.top 1 points 11 months ago (6 children)

I am working on just such a tool... but it's not ready yet. I am building a CLI tool that lets you just run `$ ai evals run humanevalsplus openhermes-2.5` and your good to go. Uses Llama.cpp

[–] ResearchTLDR@alien.top 1 points 11 months ago (1 children)

I would also be interested in this! Especially if we could create new custom evals and load them in.

[–] mattapperson@alien.top 1 points 11 months ago

Yes custom evals are supported.

load more comments (4 replies)