Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 1 year ago

MODERATORS

communick@academy.garden

[D] Why are ML model outputs not tested regarding statistical significance? (alien.top)

submitted 1 year ago by Tigmib@alien.top to c/machinelearning@academy.garden

57 comments fedilink hide all child comments

Often when I read ML papers the authors compare their results against a benchmark (e.g. using RMSE, accuracy, ...) and say "our results improved with our new method by X%". Nobody makes a significance test if the new method Y outperforms benchmark Z. Is there a reason why? Especially when you break your results down e.g. to the anaylsis of certain classes in object classification this seems important for me. Or do I overlook something?

you are viewing a single comment's thread
view the rest of the comments

[–] SMFet@alien.top 1 points 1 year ago (2 children)

Editor at AI journal and reviewer for a couple of the big conferences. I always ask for statistical significance as the "Rube Goldberg papers" as I call them are SO so common. Papers that complicate things to oblivion without any real gain.

At the very least, a bootstrap of the test results would give you some idea of the potential confidence interval of your test performance.

[–] senderosbifurcan@alien.top 1 points 1 year ago

Lol, I hate reviewing those papers. "Interpretable quantized meta pyramidal cnn-vit-lstm with GAN based data augmentation and yyyy-aware loss for XXXX" no GitHub though...

load more comments (1 replies)