Statistical Tests for Comparing Machine Learning Model Performance from Multiple Runs
Hi,
Suppose I have a neural network classifier C, based on, e.g., a CNN or Transformer.
And suppose further that I have a modification, called M, of C that I hypothesize that the accuracy of C should be better.
I can afford to run experiments for N runs (e.g., N=5, which differs by initialization) for C and C+M.
What test statistic should I use to demonstrate that the modification shows 'significant' improvement?
Moreover, for each configuration (C or C+M), should I report standard deviation (stddev) of accuracy or standard error (stddev/sqrt(5)) ?
From the context, I have often seen ML papers report stddev but some also report stderr.
Also, I have typically seen those papers that perform multiple runs do not perform any statistical tests to quantify the improvement of the methods they propose. I find this trend discerning.
Thank you very much in advance for your answer!
Crossposting: https://www.reddit.com/r/AskStatistics/comments/1tkv9xs/statistical_tests_for_comparing_machine_learning/