Heuristic Path Pruning Algorithm Based on Error Handling Pattern Recognition in Detecting Vulnerability

Using symbolic execution in path sensitive detection of software vulnerabilities develop quickly recently. It has a lot of application and research progress. However, symbolic execution still suffers from the scalability problem in practice, especially when applied to large scale or very complex programs. In this paper, we propose a new heuristic path pruning algorithm based on non-fatal error handling(NFEH) pattern recognition. Firstly, it use three patterns to recognize NFEH module in binary code and then stop analyzing NFEH branches when using symbolic execution. To demonstrate the effectiveness of this new approach, we have implemented a prototype tool PrunEbt based on a binary static integer vulnerability detecting tool (Statictaint). Experimental results are quite encouraging. PrunEbt can effectively recognize NFEH modules in binary program, significantly reduce the search space and have no omissions of true vulnerabilities. At last, it has detected one zero-day integer vulnerability in widely used image recognition library Libpng v1.5.13.