What do you do with your sealed test set if you find a bug after validating your model against that test set
I have a model that I've been tuning on various train-validate splits, picked the best one and then evaluated against a sealed hold out set. While checking the results I noticed something off, and after reviewing my code I noticed a small bug in my code.
But now I'm not sure if I'm allowed to use the same sealed test to report my numbers. Because I used the results of the last run to improve my model, which introduces a small bias. If I had a different test set I probably would not have noticed the bug.
The bug in question was an off-by-one indexing error in a data ingestion stage.
I plan to write one sentence in my paper for transparency, but not sure if the editors/reviewers will flag this as a concern.