What We Learned in Scanning for Heartbleed Vulnerable servers

    According to many press or analysts, the Heartbleed vulnerability is one of the most serious vulnerabilities ever discovered.  While companies with dedicated IT/security staffs can act quickly to patch the vulnerability as soon as the news broke,  many smaller company web sites are still unattended from security standpoint and amazingly might not even be aware of it for days after the alert was issued.

    As a service to IT community, and, honestly to test the power of our platform, NetGend set out to scan the internet to identify vulnerable sites/servers and notify the owners.  Our first step was to find a list of servers. After some researches,  we found that Amazon AWS has a list of the top 1 million sites in the world.   Amazon also maintains the public IP ranges owned by Amazon.

    As mentioned in a previous blog,  a simple PoC Heartbleed scanner worked on our lab servers.  The idea here is to extend the PoC to do internet-scale scanning.  The following blog describes some lessons we learned.

    Detecting Server-Hello-Done is trickier than first thought.  A typically Heartbleed vulnerability scanner works like the following:

• Sends a special CLIENT_HELLO, 
• Waits for the server to send 3 messages:  SERVER_HELLO,CERTIFICATE,  SERVER_DONE,
• Sends a Heart-Beat message,
• Checks if the server reply has a large chunk of data (revealing server side memory)

 The simple-minded script used in the previous blog assumed all the server messages are in the same TCP data packet.  As it turned out,  it is a lot more complicated in the real world - The "CERTIFICATE" messages can have dramatically different sizes and the 3 message can be packaged in the same or different SSL records, which in turn can span over multiple TCP data packets, with the record boundary and the packet boundary not aligned.  We had to enhance the Server-Hello-Done detection routine to take care of all the cases, but fortunately doing so is  relatively easy on NetGend platform.

    The DNS resolution is rate limited.  Before we started scanning the top 1 million websites,  we thought that we could scan them at a rate of thousands of servers per second,  which is easily supported on our test platform.  It turned out we had to fight with the DNS resolution.  
    We used the google DNS server (8.8.8.8), thinking it's the most powerful DNS server in the world and it could easily handle thousands of DNS requests/second.  Well, in fact, this service is rate-limited. In our experiment, we encountered a limit of about 100 DNS resolutions per second.  This greatly slowed down our scanning.

    3.  There are still a considerable number of servers not patched.  Even though many web sites in the top 1 million list were patched on the first day,   there were over 20,000 servers that had still not been patched after 2 days.  When we tried again about a day later,  25% of the unpatched servers has been patched.  A week after the alert was issued,  a site that boasts of 100 million subscribers was still not patched.   Another interesting thing we found was that, among the top 1 million web site, at least 750K have HTTPS support.

    During our scanning effort, we were glad to see some web sites had started monitoring the attempts to exploit this vulnerability.  Our scanning effort has triggered alarms in some of them companies or organizations.  After explaining the purpose to them and they unblocked our IP from their site.

    To avoid triggering false alarms,  we included some thing in the message to communicate that we did not have an evil intention:  we set the 32-byte random field of the SSL CLIENT-HELLO message to contain the string "test done by netgend.com".

   In addition to the top 1 million sites, we also looked at the IP ranges for Amazon web services.  We were able to find thousands of HTTPs servers that were vulnerable.   There are two surprises:  One is that AWS owned a whopping  7 million public IP addresses;   The other one is more interesting: Many of the vulnerable servers led us to the owner's main web site:  when you point your browser to the server, either the content reveals the name of the main site or the server redirects the browser to the main site.   It turns out that  for many vulnerable servers, the server(s) for the main web sites are patched. It's unclear whether the admins for the main site forgot about the other servers that they own or they simply haven't got a chance to patch these servers yet.

   We have notified many site admins of their vulnerabilities. But, unfortunately, it's a manual process.  To speed things up, we have notified both CERT and Amazon AWS in the hope that they can carry on the task of notifying the owners/admins of the vulnerable sites.

   At NetGend, we are proud that our test platform is so powerful that it can do interesting security testing in addition to the advanced performance testing.  We are also happy that it can be put to good use - like scanning for vulnerable servers and notifying the owners.  If you are interested in learning more, please drop us a line:  info@netgend.com.