Database management with sequence trees and tokens

An approach to organizing storage in database systems is presented that, under a wide range of conditions, saves both storage space and processing time. Text values in a database are replaced by short, fixed-length, rank-preserving numeric tokens. The actual values are stored in separate, non-redundant storage. Database operations that depend only on the relative magnitude of data values can be performed directly on the tokens. Tokenization is shown to improve database performance most in situations where there are a lot of ad hoc queries and a low volume of database insertions relative to other operations