Using measurements with large round-off errors for interval estimation of normal process variance

Large round-off errors may affect efforts to estimate the distribution parameters. The ratio between the standard deviation σ and the scale step h, δ = σ/h, of the measurement instrument, for which rounding off is large when δ <; 0.5, determines the significance of the round off. In this study the authors present a new variance interval estimator based on the method of moments (MoM) approach using the bootstrap technique. The authors compare the MoM interval estimator with two a-parametric estimators, the naïve estimator and Sheppard´s correction, using simulation. They find that the MoM interval estimator performs better than the a-parametric estimators in terms of coverage probability and interval length, especially for medium and large samples. The MoM interval estimator should be used to compensate for the large round off errors that can occur when using measurement instruments whose scale step is too large.