Rounding vs Improperly reporting accuracy
Say I am validating a caliper with a 0.05” resolution by comparing it with a gage block that’s actual value is 6.001”. With the caliper, I measured 6.1”.
When doing an error calculation Abs(Block-Measured), would I report the error and pass/fail based on an error of 0.1” or 0.099”? On one hand, the .099” ignores the accuracy of the gage block, but on the other, the 0.099” implies accuracy to the caliper that does not exist.
Any advice?
(Note: this isn’t an actual procedure I’m just trying to find out how to correctly report error with different resolutions)