Dynamic Detection of Unknown Malicious Executables Base on API Interception

In this paper, we propose a new approach for the dynamic detection of malicious executables on the platform of Windows. Our approach extracts signatures of malicious executable´s behaviors by using API (Application Program Interface) interception technique which makes possible the detection of unknown malicious executables. The dynamic detection of unknown malicious executables is achieved in three major steps: getting the sequence of API function calls of the executable, processing the API sequence to generate a vector, calculating the similarity between the vector and the feature library constructed by security policies to verify if the executable is malicious. The experiment confirms that this approach is effective in detection of unknown malicious executables.