Blind versus unblind performance evaluation of binarization methods

Since 2009, DIBCO is becoming the main benchmarking for evaluating objectively the performance of binarization methods by means of various quantitative measures. The usual evaluation protocol is performed on blind datasets, which contain degradations with unknown occurrence and variety. This leads to generate every year random ranking depending on the used dataset. This paper aims to propose a comparative analysis for evaluating binarization methods on unblind dataset based on a new degradation typology and takes into account the written types. Unblind dataset is produced by categorizing DIBCO images according to the dominant degradation and written (handwritten or printed) types. Experiments conducted on blind and unblind DIBCO images show that a binarization method can deal better than others when it is performed on a specific degradation or written type.