Adopting the MapReduce framework to pre-train 1-D and 2-D protein structure predictors with large protein datasets

Sequence based machine learning approaches for 1-D and 2-D protein structure prediction tasks have long been limited by relatively small datasets, namely proteins with experimentally determined structure. Recent advances in machine learning provide a means of using unlabeled data and, as a result, this opens up access to a much larger sequence space in the context of protein structure prediction. Here we present a 3-stage pipeline to construct a representative protein sequence dataset, generate training data and pre-train deep network models for 1-D and 2-D protein structure prediction tasks. To handle the complexities of managing the large dataset, we implemented our pipeline using the MapReduce framework. This allowed us to leverage existing tools such as Hadoop. The result is the ability to apply large amounts of novel, protein sequence data to 1-D and 2-D protein structure prediction. We also used our pipeline to curate a non-redundant protein sequence dataset that we have made available with accompanying data.