Abstract

The estimate of a quantity and its standard uncertainty are commonly associated with the mean and the standard deviation of a set of data obtained from measuring repeatedly the measurand (this procedure is known as 'type A' analysis). Both can be affected by the resolution of the measuring instrument. The resolution is the smallest difference between indications that can be distinguished and it is obviously limited in the case of digital instruments. In this paper, we characterize the effect of the limited resolution on a set of measurements. We realized that, if a certain minimum limit for the standard deviation of the measured data is not achieved, biased estimates and unrealistic uncertainties can be obtained. Beyond this limit, the influence of the resolution on the mean is insignificant and standard deviation includes the effect of resolution such that, in order to carry out the uncertainty evaluation, additional corrections appear to be unnecessary. © 2006 BIPM and IOP Publishing Ltd.