Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 11 months ago

MODERATORS

communick@academy.garden

[D] Why are ML model outputs not tested regarding statistical significance? (alien.top)

submitted 10 months ago by Tigmib@alien.top to c/machinelearning@academy.garden

57 comments fedilink hide all child comments

Often when I read ML papers the authors compare their results against a benchmark (e.g. using RMSE, accuracy, ...) and say "our results improved with our new method by X%". Nobody makes a significance test if the new method Y outperforms benchmark Z. Is there a reason why? Especially when you break your results down e.g. to the anaylsis of certain classes in object classification this seems important for me. Or do I overlook something?

you are viewing a single comment's thread
view the rest of the comments

[–] GullibleEngineer4@alien.top 1 points 10 months ago (1 children)

Isn't cross validation (for prediction tasks) an alternative to and I daresay even better than statistical significance tests?

I am referring to the seminal paper of Statistical Modeling: The Two Cultures by Leo Breiman if someone wants to know where am I coming from.

Paper: https://www.jstor.org/stable/2676681

[–] Brudaks@alien.top 1 points 10 months ago

Cross-validation is a reasonable alternative, however, it does increase your compute cost 5-10 times, or, more likely, means that you generate 5-10 times smaller model(s) which are worse than you could have made if you'd just made a single one.