SOME CONSEQUENCES OF THE COMPLEXITY OF INTELLIGENT PREDICTION

What is the relationship between the complexity of a learnerand the randomness of his mistakes ? This question was posed in [4] whoshowed that the more complex the learner the higher the possibility that hismistakes deviate from a true random sequence. In the current paper we reporton an empirical investigation of this problem. We investigate two character-istics of randomness, the stochastic and algorithmic complexity of the binarysequence of mistakes. A learner with a Markov model of order kis trainedon a finite binary sequence produced by a Markov source of orderk¤and istested on a different random sequence. As a measure of learner’s complexitywe define a quantity called thesysRatio, denoted by ½, which is the ratio be-tween the compressed and uncompressed lengths of the binary string whoseithbit represents the maximuma posteriori decision made at statei of thelearner’s model. The quantity ½is a measure of information density. Themain result of the paper shows that this ratio is crucial in answering the aboveposed question. The result indicates that there is a critical threshold½¤suchthat when ½·½¤the sequence of mistakes possesses the following features:(1) low divergence¢from a random sequence, (2) low variance in algorithmiccomplexity. When½ > ½¤, the characteristics of the mistake sequence changessharply towards a high ¢and high variance in algorithmic complexity. It isalso shown that the quantity½is inversely proportional to kand the value of½¤corresponds to the valuek¤. This is the point where the learner’s modelbecomes too simple and is unable to approximate the Bayes optimal decision.Here the characteristics of the mistake sequence change sharply