this post was submitted on 16 Nov 2023
1 points (100.0% liked)

Machine Learning

1 readers
1 users here now

Community Rules:

founded 1 year ago
MODERATORS
 

Often when I read ML papers the authors compare their results against a benchmark (e.g. using RMSE, accuracy, ...) and say "our results improved with our new method by X%". Nobody makes a significance test if the new method Y outperforms benchmark Z. Is there a reason why? Especially when you break your results down e.g. to the anaylsis of certain classes in object classification this seems important for me. Or do I overlook something?

you are viewing a single comment's thread
view the rest of the comments
[–] Current_Ferret_4981@alien.top 1 points 1 year ago

Are you going to make assumptions on the statistical distributions in order to make such tests accurate? Part of the reason nobody does this is because it's arbitrary and irrelevant in many cases due to incorrect application of standardized methods. Combine that with the fact that it's expensive to perform and has no real value for researchers it doesn't really make sense.