Parallel lexical analysis on multi-core machines using divide and conquer

The first step in a multi-phase compiler is lexical analysis that collects characters of a source program and transforms them into tokens for other phases. Tools have been developed to generate such tokens automatically but sequentially. In modern era, multi-core machines are very common and hence it gives opportunity to parallelize tokenization to improve lexical analysis phase. This paper discusses three approaches to parallelize lexical analysis and compares their performances. It is observed that speedup achieved by these approaches is substantial.