From Exploit to a Shell-Code
Software vulnerability is basically an incorrect or invalid handling of input parameters passed to a vulnerable program or simply software bug. A specially crafted input exploiting such vulnerability is called software vulnerability exploit or simply exploit.
If the software vulnerability is unknown to the others or undisclosed to the software manufacturer then the actual code that uses it often called a zero-day exploit or a zero day attack.
A common lifecycle of the zero day exploit is as follows:
1.  The software manufacturer releases product containing the vulnerability, usually an unknown one.
2.  The attacker finds the vulnerability before software developer does or before he was informed by the users.
3.  The attacker creates and distributes an exploit.
4.  The manufacturer finds the vulnerability and starting writing the fix.
Since the attackers wont announce, for an understandable reason, that they have found a vulnerability it might take several months to find out about the existence of such an exploit. In some cases it might take even years like in case with Microsoft IE when it confirmed the existence of vulnerability in IE 7 which affected previous versions as well and due to this fact the zero day exploits are considered the most dangerous and undetectable.
Every exploit contains a part called a shell-code. An initial purpose of the shell-code is to start an operating shell program that provides a communication interface between user and operating system and to establish a connection with remote control server (for example attacker's computer) for further instructions. Once the shell-code is injected and the control of the compromised machine is gained, the malware or botnet agent will be downloaded and installed, the new station will be attacked or any other scenario may occur.
The following image shows the structure of exploit and shell-code.
|FILE BODY||NOP SLED||SHELL-CODE BODY||FILE BODY|
Picture 1 - exploit structure
The NOP SLED is used to catch instruction pointer register
of CPU and transfer it to body of the shell-code. NOP sled is comprised of a
meaningless sequence of single or multi-byte instructions which have only one
purpose, to start execution of a shell-code from a correct position.
Plain exploits may be easily detected by anti-virus engines simply by using recognition patterns of NOP-sleds, exploit (shell-code) content or body. To avoid this, malware writers develop a polymorphic shell-code that mutates from generation to generation (from execution to execution) while keeping original execution algorithm. As in case of virus mutation, where different packers are used to hide its real code, shell-code mutation is based on different encryption engines which modify exploit's body keeping the original shell-code algorithm.
The following image shows structure of exploit and polymorphic shell-code.
|FILE BODY||NOP SLED||DECRIPTION PROCEDURE||SHELL-CODE BODY||FILE BODY|
Picture 2 - encrypted exploit structure
As it could be seen from Picture 2, the encrypted exploit as well as plain exploit starts with NOP sled which transfer instruction pointer register to next exploit's section. Unlike the regular exploit the NOP sled of the encrypted exploit is followed by decryption procedure that is used to decode the encoded shell-code.