Towards fuzzy method for estimating prediction accuracy for discrete inputs, with application to predicting at-risk students

In many practical situations, we need, given the values of the observed quantities x₁, ⋯, x_n, to predict the value of a desired quantity y. To estimate the accuracy of a prediction algorithm f(x₁, ⋯, x_n), we need to compare the results of this algorithm´s prediction with the actually observed values. The value y usually depends not only on the values x₁, ⋯, x_n, but also on values of other quantities which we do not measure. As a result, even when we have the exact same values of the quantities x₁, ⋯, x_n, we may get somewhat different values of y. It is often reasonable to assume that for each combinations of x_i values, possible values of y are normally distributed, with some mean E and standard deviation σ. Ideally, we should predict both E and σ, but in many practical situations, we only predict a single value ỹ. How can we gauge the accuracy of this prediction based on the observations? A seemingly reasonable idea is to use crisp evaluation of prediction accuracy: a method is accurate if ỹ belongs to a k₀-sigma interval [E-k₀·σ, E+k₀·σ], for some pre-selected value k₀ (e.g., 2, 3, or 6). However, in this method, the value ỹ = E+k₀·σ is considered accurate, but a value E+(k₀+ε)·σ (which, for small ε>0, is practically indistinguishable from ỹ) is not accurate. To achieve a more adequate description of accuracy, we propose to define a degree to which the given estimate is accurate. As a case study, we consider predicting at-risk students.