Mixed Obfuscation of Overlapping Instruction and Self-Modify Code Based on Hyper-Chaotic Opaque Predicates

Static disassembly is used to analyze program control flow that is the key process of reverse analysis. Aiming at the problem that attackers are always using static disassembly to analyze control transfer instructions and control flow graph, a mixed obfuscation of overlapping instruction and self-modify code based on hyper-chaotic opaque predicates is proposed, jump offsets in overlapping instructions and data offsets in self-modify code are constructed with opaque predicates. Control transfer instructions are modified into control transfer unrelated ones with the combination of characteristics of overlapping instruction and self-modify code. Experiments and analysis show that control flow graph can be obfuscated by mixed obfuscation due to the difficulty of hyper-chaotic opaque predicates for attackers to analyze.