HOCT: A Highly Scalable Algorithm for Training Linear CRF on Modern Hardware

Conditional Random Fields (CRFs) are widely used in machine learning and natural language processing fields. A number of methods have been developed for CRF training. However, even with state-of-the-art algorithms, the training of CRF is still very time and space consuming. This make it infeasible to use CRFs in large-scale data analysis tasks. This paper proposes an efficient algorithm, HOCT, for CRF training on modern computer architectures. First, software prefetching techniques are utilized to hide cache miss latency. Second, we exploit SIMD to process data in parallel. Third, when dealing with large data sets, we let HOCT instead of operating system to manage swapping operations. Our experiments on various real data sets show that HOCT yields a fourfold speedup when the data can fit in memory, and over a 30-fold speedup when the memory requirement exceeds the physical memory.