Wednesday, April 9, 2014

What can a performance test tool do to help with Heartbleed?

   Heartbleed is a recently discovered vulnerability that affects openSSL based web servers.  Hackers can potentially read up to 64KB of memory on the server due to an out-of-bound bug on some versions of the openSSL library and can potentially glean vital information such as a server's private key, user credentials, etc...
     Thanks to the instructions from this gist post I patched my Ubuntu server and used this online tool to check it before and after patching.

    When searching for information on patching my server, I encountered a static file hosted on the site.  This file is only 788 bytes in length and yet, downloading it took over 30 seconds in the first attempt.  I tried two more times and here are the times for each of the downloads.

  • 32.2sec
  • 13.5sec
  • 30.4sec

     The last attempt,  setting up TCP connection alone, took more than 7 seconds, while the server response took 23 seconds.  This shows that the openssl site is seriously overwhelmed.  This site could have benefited from some performance testing and planning for such a big event.

     Yes, that would have been nice, but it is what it is. What about the impact on the massive amount of administrators faced with patching this? What can be done for them?

    In my case,  there is only one server to patch.  What if an organization has thousands of servers? How can we find out which servers need to be patched?   Checking them one by one is too time-consuming.  It turns out a simple script on NetGend can help out on this.

    The following script checks vulnerability on two hosts. It can be easily adapted to read a list of server host names or IP addresses from, say, a csv file and start checking the servers in parallel.  In less than a minute, you will be able to scan through thousands of servers and get a list of the servers that need to be patched.   The messages used here are based on those generated by the above mentioned online tool to which I am very thankful.

function userInit() {
    var hosts = ["", "localhost"];
    var gid = 0;
function VUSER() {
    id = gid ++;
    connect("tcp", hosts[id], 443);
    msg = pack("H*", "16030200dc010000d8030253435b909d9b720bbc0cbc2b92a84897cfbd3904cc160a8503909f770433d4de000066c014c00ac022c0210039003800880087c00fc00500350084c012c008c01cc01b00160013c00dc003000ac013c009c01fc01e00330032009a009900450044c00ec004002f00960041c011c007c00cc002000500040015001200090014001100080006000300ff01000049000b000403000102000a00340032000e000d0019000b000c00180009000a00160017000800060007001400150004000500120013000100020003000f0010001100230000000f000101");
    msg = pack("H*", "1803020003014000");
    if (length(data) > 1000) {
        println("host ${hosts[id]} is vulnerable");

 Here is a brief description of the script.
  • Line 7,  set up a TCP connection to the HTTPS server
  • Line 8, 9, send the client-hello message
  • Line 10,  receive the messages (server hello,  certificate ...) from the server,  note that in general, the server messages may not always come in one packet, so we may need to detect the end of server messages
  • Line 11,12, send heartbeat message, this is the message that tests whether the server is vulnerable.
  • Line 13, 14, wait for at most 2 seconds (2000ms)
  • Line 15, 16,  if the received data is greater than 1000 byte, the server is vulnerable
    At NetGend,  we are proud that our performance test platform is not only massively scalable, it is also so flexible and easy to use that we can use it to solve problems in areas seemingly not related - such as efficiently scanning for the Heartbleed vulnerability, saving our customers tremendous time and money.

  The author would like to thank his friend David Franklin at ventasconsulting for the suggestion to write a blog on this. 

No comments:

Post a Comment