Preference evaluation of a large set of vocoded speech signals

Starting from an IEEE Recommended Practice for Speech Quality Measurements and from previous work of the authors, a large set of vocoded speech signals has been evaluated in terms of preference. The set of speech samples has been taken from the vocoder survey of the 1967 Conference on Speech Communication and Processing, Boston, Mass. The test samples are evaluated by several methods: direct comparisons, the isopreferenee method, the relative preference method, the category judgment method, and the absolute preference judgment method. Due to the size of the test material, not all the test samples could be evaluated by all these methods. The test results are discussed and it is shown that, in certain respects, reliable system evaluations pose formidable problems. An effort to rank order the systems, which are described by small sets of test samples of frequently very different quality, for good reasons shows only limited success. The majority of the systems are of about equal preference with only insignificant differences. There are only a few systems that are outside this group and are either significantly better or worse than the rest.