Practical Optimizations for Perceptron Algorithms in Large Malware Dataset

Due to the increasing number of malware samples in the past 4 years, machine learning algorithms emerged as an important tool in automated malware detection. This approach to create the detection model requires, however, a lot of time with a continually growing data-set. Often changes in malware families and the increasing training time makes the model less efficient and increases the probability of false alarms. This paper approaches this matter by reducing the time needed to create a detection model on very large databases and suggests three different optimization techniques. First, the perceptron algorithm was adjusted to use the map-reduce paradigm in order to make it run in a distribute manner. Second, hardware specific optimizations were applied for faster mathematical computations. Finally, a cache system was used to reduce the quantity of data processed by the algorithm. Even if these methods were designed and tested for malware databases they can easily be adjusted for other databases as well.