Statistics in Performance testing

    In performance testing, statistics is very important.  For instance, the following statistics will give user an idea how the server under test is performing,

• max concurrent connections
• transaction rate
• latency
• bandwidth
• ...

     In this blog, we are going to cover some "new" features in statistics that netgend javascript supports,

User-defined Counter

    For HTTP transactions, sometimes it's enough to see the HTTP requests get responses,  sometimes we need to see whether the HTTP responses are all 200 OK.   There are cases when even the 200 OK responses are not enough, for example,  when you try to submit a login form, you got 200 OK but the response text tells you login is unsuccessful.  How can you catch this by watching statistics charts?

   This is where user-defined counter can help.  In netgend javascript,  you can define your own success counter.  

 http.POSTData = "user=${username}&pass=${password}";  
 action(http,"http://www.example.com/login?");  
 if (contains(http.replyBody, /Welcome/)) {  
   stats("success");  
 }  

    The above simple logic checks whether a response contains "Welcome" and increment the counter only when it does.  You can add it to your script and watch the statistics chart in GUI.  It will make you happy if you see the curve for user-defined counter matches with the request rate curve.  

   This can be very useful if you are testing a non-HTTP protocol, where there is no well defined transaction. You can use the user-defined counter to find out how well the server is responding to the requests. 

User-defined Latency

   In addition to counter, we measure different latencies, including 

• TCP connection latency,  
• First HTTP response latency (the time between request and first packet in response)
• Last HTTP response latency (the time between request and last packet in response)

    However, there are cases when we may want to measure a range of operations.  For example, certain web portal may need multiple HTTP transactions,  what if we want to measure the total time?     

     Our user-defined-latency can help here.  You just need to add two lines of script around your operations.

 start = currentTick;  
 //do a bunch of operations here  
 stats("latency", currentTick - start);  

     Note that the variable "currentTick" will contain the latest timestamp.
 
     With these extra lines,  statistics window will show the user-defined latency in addition to the other latencies. 

Dynamic Rate Adjustment

     When running a performance test, we sometimes start with a low transaction rate,  set up increment schedule like:   increment the rate by x for every y seconds.  While this will work,  there are cases where it's better to adjust the rate manually first, especially when we are not familiar with the system capacity yet:

• The rate increment is too slow, based on the feed back from latency and other statistics, we would like to increment the increment rate. We are torn between restarting the test (with faster increment) or being more patient and watch the rate slowly creep up. 
• The rate increment is too fast,  latency starts to grow bad while rate is still increasing.  

     In Netgend platform, we support dynamic adjustment of rate. You can start with certain rate and increment the rate manually based on the state of the system, if the latency is healthy, transaction success rate is good (remember we have user-defined counter for success),  you can increment the rate by a user-defined amount, otherwise, you can decrease the rate to find out the tipping point of the system under test.

     Of course, after you have gotten an idea on the capacity of the system, you are welcome to set up a test with automatic rate increment (increment by x for every y seconds), which is also supported.