ClustDB: A High-Performance Tool for Large Scale Sequence Matching

High throughput sampling of expressed sequence tags (ESTs) has generated huge collections of transcripts that are difficult to compare with each other using existing tools for sequence matching. The major problem is lack of computer memory. We therefore present a new exact and memory efficient algorithm for the simultaneous identification of matching substrings in large sets of sequences. Its application to more than six million human ESTs in Genbank of date 2005-04-06, counting more than 3.3 billion base pairs, takes less than four hours to find all more than seven million clusters of multiple substrings of at least 50 nucleotides in length, say, by using a standard PC with 2 GB of RAM, 2.8 GHz processor speed. The corresponding program ClustDB is able to handle at least eight times more data than VMATCH, the most memory efficient exact software known today. Our program is freely available for academic use