Efficient Storage Allocation of Large-Scale Extendible Multi-dimensional Scientific Datasets

Data analyses in scientific domains involve storage, retrieval, processing and visualization of large scale multidimensional datasets. The datasets incrementally grow by appending new data to the dataset without reorganizing the already allocated data storage. The datasets, typically modeled as k-dimensional arrays, are maintained in files where the array elements are allocated in a sequence of consecutive storage locations according to some ordering of an array mapping function. Such mapping functions limit the degree of extendibility of the arrays to one dimension only. To allowing new data elements to be appended to the dataset effectively implies allowing for the arbitrary extendibility of the array. We present a mapping function Fscr_*(), that uses additional O(klogEscr) storage, where Escr is the number of array segments appended. The function is realized with approximately the approximately the same order of complexity as a conventional array function. The algorithm presented for addressing elements of datasets in extendible multi-dimensional array files apply equally well to addressing memory resident extendible arrays