Intelli-log : A real-time log analyzer

In this paper, we present a technique to analyze and correlate the different types of computer log files. Log files are generated from servers and network devices to record operations that occur in the computers and networks. As log files are too enormous to manualize, we develop a tool to maximize accuracy as well as efficiency while high speed processing is the goal. Firstly, we must improve the accuracy by using learning algorithms to classify the normal operations from the abnormal ones such algorithms include tf-idf, association rules, k-means clustering, and decision tree. Secondly, we may adapt for less accuracy in order to gain speed for both with and/or without parallel processing techniques. We also construct an adaptive learning algorithm to update the model. Then we flush out out-of-date model while the logs are being captured and processed. The result can achieve the goal as they can reach about 30-40% in real-time processing with nearly zero false positive results.