Monday, March 31, 2014

Replay server log while observing the timestamp


   One of the common requests from the companies in need of performance testing is to replay the server logs.  Replaying server logs can give the server(s) a very realistic test because the HTTP requests are actually from the real world.  To make this approach even more realistic, it would be nice to replay the server logs according to the time stamps of log entries. Multiple such questions (1 2) were asked in the StackOverflow forum.   In this blog we will explain in detail a script implemented on the NetGend performance test platform.

    Before diving into the script, I want to point out that there are implementations, by using scripting languages such as Ruby, that are quite involved from a test engineer's point of view. In addition, these implementations do not support many concurrent connections (say, 5000) which are required to accurately make use of the time stamps in the server log.  Why is it important to have many connections?  Sometimes, there are bursts of requests to the server, and, without enough connections, you will not be able to generate sufficient requests to simulate the burst.

    There are some solutions on platforms such as JMeter, but they require installing complicated plugins and/or having deep knowledge of the relevant programming language - again above the heads of many testers.

   In contrast, the NetGend script is refreshingly simple.   In the following example, we assume the log only contains two fields:  timestamp, in seconds (with decimal places), and the URL.  More complex logs can be handled similarly.
 1396037194.011 http://www.example.com/sports  
 1396037194.021 http://www.example.com/update/worldnews  

  Here is the script. As you see, it is quite simple.  Note that in this test scenario, “virtual users” are different from what they mean in other test scenarios: they are not used to emulate real users, but instead they are used as workers to get a job and execute it.   So, a "VUSER" should be understood as a worker”.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
function userInit() { 
     var fd = openFile("srvLogs.txt"); 
     var startTime = 0; 
     var startTick; 
function VUSER() { 
     while (1) { 
          line = getLine(fd); 
          if (line == "") { break; } 
          b = qsplit(line); 
          if (startTime == 0) { startTime = b[0]; startTick = currentTick;} 
          delay = (b[0] - startTime) * 1000 - (currentTick - startTick);  
          if (delay > 0) { sleep(delay); } 
          action(http,b[1]); 
     

  • Line 2-4, we open the server log file and declare two global variables, 
    • startTime:  holds the first timestamp in the server log.
    • startTick:   holds the time (in ticks, 1tick = 1ms) on the test platform when it's ready to process the first log entry.
  • The VUSER() function just consists of a while loop.
  • Line 8,  each worker gets a line from the server log file.
  • Line 9,  if it's empty, then come out the loop since we have reached the end.
  • Line 10,  split the line into fields.
  • Line 11,  set the starting timestamp (startTime), we will set it only once.
  • Line 12,  calculate the delay that we need to sleep before running.
  • Line 13, sleep according to the delay from previous step.
  • Line 14, send the HTTP request.

  We can start 10,000 such workers, each following the above logic.  Overall, the test platform is  replaying the server log while following timestamp,  even if there is a big burst of requests.

  Note that the script will work not only with Apache server log, but also with other server logs such as Nignx, MS IIS.

   Application Development and Testing Teams will require many different scenarios for replaying server logs. Based on the built-in flexibility and power of the NetGend platform, we are confident that we will be able to handle them all. Give us a call or drop us a note with any questions, or if you would like to use NetGend in your project.

Friday, March 28, 2014

Test a performance test tool using keyword search


    There are many performance test tools, when choosing the right one for your project,  it's important to know how scalable, flexible the test tool is.  It's relatively straightforward to find the scalability of a test tool  - how many virtual clients it can emulate on one load generator. So in this blog we will focus on testing the flexibility of a test tool.   It turns out there is a simple test scenario that can give you a quick idea how flexible a test tool is - emulating keyword search.


   Keyword search test scenario is to emulate internet user typing a searching phrase in his/her browser. As he/she is typing, a partial keyword is sent to the server and the server will return a list of results matching that partial keyword so the browser can give recommendations as the user is typing.  You may be surprised that this simple yet interesting test scenario are not easy on many test tools  - some test tools will ask tester to write code that is hard for them to understand or create.

  It's fairly simple on NetGend platform.  In the following script, we emulate users searching for the keyword "country music" with a simple loop:
 function VUSER() {  
     keyword = "country music";  
     len = length(keyword);  
     for (count = 1; count <= len; count ++) {  
         partial = substr(keyword, 0, count);  
         action(http,"http://localhost/?q=${partial}");  
         sleep(300); //sleep 300ms   
     }  
 }  
    This script basically sends a bunch of HTTP requests with parts of the keyword "country music".  The partial keywords were constructed using the function "substr(keyword, 0, count)". This function returns a substring of the input (keyword in our case) starting from position 0 (the first position) with count characters.

    In the above script, each emulated user will pause 300ms between typing each characters, what if you want to be a little more realistic by pausing a random amount of time?  That's easy, just replace the sleep(300);" in the above script with the following:
     x = randNumber(300, 2000);  
     sleep(x); //sleep random time between 300ms to 2000ms   

  What if you want to emulate a user typing a few keys and stopping after he/she sees the recommendation that he/she likes?  That's easy too, just change the line "len = length(keyword); " in the above script to the highlighted line in the following.
 function VUSER() {   
   keyword = "country music";   
   len = randNumber(1, length(keyword));   
   for (count = 1; count <= len; count ++) {   
     partial = substr(keyword, 0, count);   
     action(http,"http://localhost/?q=${partial}");   
     x = randNumber(300, 2000);  
     sleep(x); //sleep random time between 300ms to 2000ms    
   }   
 }  

   Finally, what if we need to test a set of keywords?  We can read the keywords from a file.  The following script shows we can read the keywords from a csv file in the userInit() function which executes before any virtual user runs.
 function userInit() {  
      var keywordRows = fromCSV("Phrases.txt");  
 }  
 function VUSER() {   
   keyword = getNext(keywordRows);  
   len = randNumber(1, length(keyword[0]));   
   for (count = 1; count <= len; count ++) {   
     partial = substr(keyword, 0, count);   
     action(http,"http://localhost/?q=${partial}");   
     x = randNumber(300, 2000);  
     sleep(x); //sleep random time between 300ms to 2000ms    
   }   
 }   
    The "var" in above script means that the variable "keywordRows" is visible to all the virtual users. The reason why we use keyword[0] is because getNext() returns a row of the csv file. A row can contain multiple keywords and keyword[0] means the first one.   We can emulate users trying all the keywords in a row,  but it's left as an exercise for the reader;-)

    Hope you get a good idea on how to test the flexiblility of a test platform using the keyword search test scenario.  At NetGend, we are proud that our platform is very flexible and  easy to use in addition to being hyper-scalable - it can emulate 1 million such virtual clients on on box.

Tuesday, March 25, 2014

Performance test on real time HTTP communication: Websocket


    Over the years,  there are many enhancements to HTTP.   One of them is Websocket.

   Websocket makes the communication between a client and a server fully duplex - each part can independently send a message at any time, which is essential for the real-time communication.  This is in contrast to the old style where the client sends a request to the server and waits for a reply.  In order to get real time updates from the server, the client has to either poll the server with requests or send a request to the server with a long time out and repeats it after it receives a message. None of these is very efficient in terms of message size or message count.
 
    Here is a great video that shows the difference that websocket makes on an online drawing web app.  The following is a snapshot in the video.   There are 3 windows, a user draws a graph on the left one and expect the graph to also show up on the other two windows.

  • The middle one is developed using websocket
  • the right one is a standard web app (no websocket)
     One can see the the middle one catches up with the graph that user draws on the left, while the right window is lagging behind.




 








    How do we run performance test on the websocket based web apps?   There are many performance test platforms but unfortunately not many support websocket.    Luckily it's supported on the NetGend and it's fairly easy too!   The following is a simple script that shows how to initiate a connection and do basic operations such as send and receive data.

 function VUSER () {   
    connect("websocket", "ws://echo.example.com");   
    send("hello1");   
    send("hello2");   
    recv(reply);  
    println(reply.msg);   
    recv(reply);  
    println(reply.msg);   
 }  
    The first step is to connect to a server that talks websocket:
           connect("websocket", "ws://echo.example.com");
After that, you can use the function "send()" to send data and the function "recv()" to receive data.  The received message is in reply.msg, you can process it in man different ways,  such as substring matching, regular expression matching etc. NetGend is packed with features for user to process these messages.

     Some of the websocket based apps are for streaming.   Examples of the streamed data include social feed data and  financial ticker data.  The following example shows how easy it is to start testing streaming over websocket - by using a while loop.

 function VUSER () {    
      connect("websocket", "ws://echo.example.org");    
      send("hello");    
      send("exit");  
      while (1) {  
           recv();  
           //println(reply.msg); you can do other processing on the received data.
           if (reply.msg == "exit") { break; }  
      }  
 }   

    Another popular application of the websocket is the location updating, used in many location aware social media apps.  The following is a simple example where we emulate clients who starts from a point and move randomly every 10 or 20 seconds and send location updates to the server.

 function VUSER () {    
      connect("websocket", "ws://locationUpdate.example.org");    
      oc.x = randNumber(10.0, 12.0, "float");
      oc.y = randNumber(20.0, 25.0, "float");  
      while (1) {  
           n = randNumber(10000, 20000);   
           sleep(n); //sleep 10 to 20 second  
           loc.x += randNumber(-0.01, 0.01, "float");  
           loc.y += randNumber(-0.01, 0.01, "float");  
           send(toJson(loc));  
      }  
 }   

    As the popularity of the websocket grows, it's certain that there will be more applications built with it. At Netgend, we are glad that we got the performance testing of it covered.

Saturday, March 22, 2014

Binary data for HTTP request made easy


   In many cases the data transferred over HTTP are ascii,  so they are easy to process for most performance testing platforms including JMeter.  However, occasionally we may need to send binary data to the server, as shown in a stackoverflow question.   This is where it's not quite easy any more on many testing platforms.  The following is a perl script that the author of the question wrote for one HTTP session.

 #the perl script that sends the time data
 $date_time = sprintf "%08X", time();  
 $BODY_TEMPLATE = "00${date_time}0015";  
 $body_len = (length (sprintf($BODY_TEMPLATE,0,0))) / 2;  
 # Here I set $TARGET & $HOST  
 $MSG_HEADER = "POST \/unil?URL=${TARGET} HTTP\/1.1\r\nHost: ${HOST}\r\ncontent-type:application/binary\r\ncontent-length: ${body_len}\r\n\r\n";  
 $body = pack ("H*", $BODY_TEMPLATE);  
 $message_to_send = $MSG_HEADER . $body;  
 # at this point I sent the entire message over a TCP socket I previously opened.  

    This test scenario is actually quite easy to describe:  the client needs to create a message with the time (in seconds) in the form of a 4-byte integer, with one byte (0) in the front and two bytes (0x0015) in the rear,  and send the 7-byte message as the POST body.

    The recommended solution on JMeter involves knowing Bean shell pre-prosessor and some knowledge on dealing with jmeter variable(s).  There is also a Jmeter plugin that can potentially send any raw HTTP request.  But it requires installing the plugin and non-trivial skills to be able to get the binary data into the POST body.

   On NetGend platform,  the following simple script will take care of this test scenario:

 function VUSER() {   
      http.POSTData = pack("CNn", 0,time(), 0x15);  
      action(http, "http://www.example.com/unil?URL=${TARGET}");  
 }   

    The key part of this short script is the "pack()" function.  "pack" is the word used to describe the process of converting data into binary format.  For example, the two characters "35" can be packed into one byte "5" (whose ascii code is 0x35).

     In the above Netgend script, we packed 3 fields to a binary data using the format string "CNn",

  • The first part is "C",  it will pack the first field, 0,  to a byte,  
  • The second part is "N", it will pack the second field, time(), to a 4-byte integer in the big Endian order. Here the function "time()" returns the number of seconds since Jan 1, 1970,  
  • The last part is "n", it will pack the last field, 0x15, to a 2 bytes integer in the big endian format.   
    In addition to the above 3 formats, a lot more are supported.  For example, "V" will pack a number to an 4-byte integer in the little Endian order.

   This little script is not only simple, it's also very powerful:  it can easily emulate tens of thousands of HTTP clients from one load generator.

   The "pack()" function has a sister function "unpack()" which will do just the opposite: convert a binary data to  number(s) or string(s).  "unpack" can be very useful in processing server messages  that have binary data and it apply not only to HTTP bodies but also to any TCP data in general.

   Together, "pack" and "unpack" make dealing with binary data very easy on NetGend platform.

Sunday, March 16, 2014

Why is mobile web browsing slow


    We all know browsing web via wireless is slow and we think we know why: latency introduced by the wireless network.  But the latency is at most about 150ms,  so on the surface it shouldn't affect the overall time that much.  Why do we feel an obvious slow down?

    I did an experiment and dived into the network packets for a HTML page download.  Turned out it was something that I was not aware of - TCP congestion control.

   The experiment was done by browsing a test web page on my server from my android phone (AT&T).  First,  the WiFi was turned off  so that my cell phone only uses the wireless connection.  Then I started wireshark, the famous packet sniffer,  on the server side.   The server is a libevent based server running on Ubuntu 12.04 server with the kernel version 3.2.0-59-generic.   The test was done when the (dedicated) server has very little load (<1% CPU).

   I grabbed the TCP session for downloading a HTML page and summarized the packets in the following.  The IP address of the server was changed to 11.11.11.11 to protect my server's identify (don't want others to test my server :-)).  To make it easier to read, I removed the TCP-ACK packets.   So other than the first two packets, all the others are TCP data packet.


 Time(Sec) Source IP    Destination IP 
 0.000000 107.77.66.120 11.11.11.11    TCP SYN
 0.000052 11.11.11.11 107.77.66.120    TCP SYN_ACK
 0.046860 107.77.66.120 11.11.11.11    HTTP GET /TestWebPage
 0.047456 11.11.11.11 107.77.66.120    data
 0.047484 11.11.11.11 107.77.66.120    ...
 0.093891 11.11.11.11 107.77.66.120  
 0.093901 11.11.11.11 107.77.66.120  
 0.094011 11.11.11.11 107.77.66.120  
 0.140421 11.11.11.11 107.77.66.120  
 0.140509 11.11.11.11 107.77.66.120  
 0.186898 11.11.11.11 107.77.66.120  
 0.233355 11.11.11.11 107.77.66.120  
 0.233463 11.11.11.11 107.77.66.120  
 0.279821 11.11.11.11 107.77.66.120  
 0.326314 11.11.11.11 107.77.66.120  
 0.326419 11.11.11.11 107.77.66.120  
 0.372772 11.11.11.11 107.77.66.120  
 0.372883 11.11.11.11 107.77.66.120  
 0.419273 11.11.11.11 107.77.66.120  
 0.419378 11.11.11.11 107.77.66.120  
 0.458092 11.11.11.11 107.77.66.120  
 0.466553 11.11.11.11 107.77.66.120  
 0.466571 11.11.11.11 107.77.66.120  
 0.513158 11.11.11.11 107.77.66.120  
 0.513180 11.11.11.11 107.77.66.120  
 0.514254 11.11.11.11 107.77.66.120  
 0.559861 11.11.11.11 107.77.66.120  
 0.559893 11.11.11.11 107.77.66.120  
 0.560944 11.11.11.11 107.77.66.120  
 0.606500 11.11.11.11 107.77.66.120  
 0.606517 11.11.11.11 107.77.66.120  
 0.606523 11.11.11.11 107.77.66.120  

    Looking at the list of packets and their timing,  I am reminded of driving on a congested road - stop and go and stop:
  • at time around 0.047, it sent 2 packets, 
  • at time around 0.093, it sent 3 packets
  • at time around 0.140, it sent 2 packets
  • ...
    Why can't the server keep sending data to the client?  First, there is the TCP window,  it determines how much data a host can send without getting acknowledgement. In our case, the TCP window size started with 14848 and it growed over time.  If this were the only limitation,  the server should be able to send 10 data packets in a burst. Why did the server only send 2 or 3 packets at a time?  Turned out it was also limited by the TCP congestion window.  Typical TCP congestion window will allow a host (the server in our case) to send 2 full size TCP data packet in the beginning,  it will grow or shrink (if there are dropped packets) over time. 

   So the congestion window on the server side is obviously a more severe limiting factor.  I am sure there are people who don't like it, but it's there for a reason: congestion control.  When there is a network congestion, it will keep all the TCP sessions moving, albeit a little slowly. This is much better than keeping sending packets to make congestion worse and no sessions are moving.


    The experiment was carried out at a time when there is not much wireless usage (early Sunday morning) and hence the latency is pretty low,  the round trip time between server and my mobile phone is only 46ms.  However downloading the entire HTML page still took about 600ms, which is more than 12 times the round-trip time.

  In normal operating hours, the round trip time between mobile device and server can be 200ms or more and the HTML page downloading can take more than 2.4 seconds and it could be slowed dow even further by :

  • dropped packets
  • SSL handshake
   On top of that,  thinking about the other works a mobile browser has to do:  download the javascript files, css files and yes, image files.   Now I start to understand why mobile browsing can be much slower.

   After this experiment, I not only have a better understanding of why mobile internet browsing is slow,  I appreciate the network sniffer, wireshark,  even more.  I hope wireshark becomes a good friend of every performance professionals.