Complexity-based packed executable classification with high accuracy

Abstract

Signature-based anti-virus scanner that utilizes specific bytes string is one of the popular malware detection methods. This method shows high efficiency and low false rate in detecting malware. However, it can``t successfully detect the malware when packer method is applied to the malware. Packer compresses or encrypted the target le and pads compressed or encrypted le to additional section. Therefore, it changes the bit structure of the malware. When packed executable le is executed, compressed or ncrypted le is decompressed or decrypted on memory. Then, the program that is loaded on the memory is executed. This means that function of the packed executable program is same as an original program. Signature-based anti-virus scanner employs two methods to over-come the difficulty of detecting packed malware: packed executable classification and generic unpacking. Packed executable classification decides whether the target program is packed or not. Then, generic unpacking is performed depends on the decision of a packed executable classification. Various generic unpacking methods have been researched [13, 14, 15, 22], but these generic unpacking methods consume long time and computing resources. Thus, accuracy of a packed executable classification is the critical issue in efficient malware detection. Because a false positive case happens, unnecessary time and resource is wasted performing generic unpacking on a non-packed executable file. On the other hand, a false negative case happens, malware detection would fail. Previous packed executable classification methods show high false rate and can be easily evaded. PEiD[11] can be easily evaded by false signature. Bintropy[20] shows about 10% false rate. This method also can be evaded by monotonous padding bytes. Finally, Roberto $\It{et al}$. show high classification accuracy rate about 97%, but this method can be evaded by PE headers modification. In this thesis, we propose highly accurate classification met...