Towards a calibrated corpus for compression testing

Summary form only given. A mini-corpus of twelve `calibrated´ binary-data files have been produced for systematic evaluation of compression algorithms. These are generated within the framework of a deterministic theory of string complexity. Here the T-complexity of a string x (measured in taugs) is defined as C_T(x_i)=Σ_ilog₂(k_i +1), where the positive integers k_i are the T-expansion parameters for the corresponding string production process. C_T(x) is observed to be the logarithmic integral of the total information content I_x of x (measured in nats), i.e., C_T (x)=li(I_x). The average entropy is H¯_x=I_x/|x|, i.e., the total information content divided by the length of x. Thus C_T(x)=li(H¯_x×|x|). Alternatively, the information rate along a string may be described by an entropy function H_x(n),0⩽n⩽|x| for the string. Assuming that H_x (n) is continuously integrable along the length of the x, then I _x=∫₀^|x|H_x(n)δn. Thus C_T(x)=li(∫₀^|x|H_x (n)δn). Solving for H_x(n): that is differentiating both sides and rearranging, we get: H_x(n)=(δC_T(x|n)/δn)×log_e (li^-1(C_T(x|_n))). With x being in fact discrete, and the T-complexity function being computed in terms of the discrete T-augmentation steps, we may accordingly re-express the equation in terms of the T-prefix increments: δn≈Δ_i |x|=k_i|p_i|; and from the definition of C_T(x): δC_T(x) is replaced by Δ_i C_T(x)=log₂(k_i+1). The average slope over the i-th T-prefix p_i increment is then simply (Δ_iC_T(x))/(Δ_i|x|)=(log ₂(k_i+1))/(k_i|p_i|). The entropy function is now replaced by a discrete approximation