Early performance testing, 1 oz prevention >= 1 lb cure

   We all know testing is an essential part of product development, performance testing is a big part of testing, especially when we are dealing with the web based applications where customers will be lost if server crashes under high volume or response time becomes too long.  Unfortunately many project involves the performance testing only in the end, as part of the acceptance testing.  Common as it seems, there are serious problems with this approach.

   In some cases, performance test result shows that considerable amount of performance improvement is needed for the release of the product. Now developers have to search all over the code to find places that can be optimized. It may seem like a bug hunt,  but it's different because it is not about  logic errors.  Afterall, logic of code may be all right after going through rounds of unit-testing and functional testing. Developers may have to do trial-and-error by making some changes and see how much performance gains they get. The target for performance may be so high that it will take weeks if not months to get there.  I have seen some developer staring at the code so hard for optimization and they found some real bugs missed in unit-test or functional-test :-).  Obviously it's not the reason why performance test should be done at the end.

   In some other cases, performance testing shows that performance number is seriously lower than expected and after some experiments, a harsh reality comes to surface:  a part of design is fundamentally wrong.  After month's of development and testing, things have to go back to square one. This may seem impossible but it does happen from time to time. I was involved in a project, where the performance bottleneck turned out be disk IO. Simple as it may appear, it was found out after

• trying different web servers, thinking asynchronous server can solve the problem
• trying various database optimization, hope one silver bullet can save them.

The cold and hard fact found by the performance testing is that:  the bottleneck is on retrieving records from disk. How fast the records can be retrieved from disk depends on the access pattern, the database and the disk. A totally different database had to be used for that project.

     Now, why isn't early performance testing commonly practiced? One of the reasons is that

it requires a non-trivial amount of resources. Some times, such performance  tests involve some programing, so some developer resource has to be reallocated to do performance test development.  Is it worth it?  Very likely.  It can be a perfect application of the classic principle of "one ounce of prevention may worth a pound of cure".  An added benefit is that  testing team may also benefit from the tools/procedures created by the developer.

    Another possible reason is, performance testing requires a tool that is neither simple

to use nor cheap/free. In today's explosion of software products, new tools are coming up like

mushrooms after a rain, just check around and you will find the pleasant surprises. For

example, in the area of application performance testing,  new tools and low-cost/free services

like gatling, loader.io, NetGend are all new kids in the block and waiting to help you

do the early performance testing.

     Looking back on some of the projects, I feel they could benefit tremendously if we have done performance testing from the start - when the project was still in its infancy. We could have found out whether the major decisions (like the choice of database) were fundamentally wrong early on.  Just like curing a major disease (think cancer), making changes in the early stage is much easier than making changes in the late stage:

• there is not a lot of time and effort invested,
• the code structure is relatively simple.  Layers of patches on the code base during the course of project can make it very hard, if not impossible to hunt bugs, especially performance bugs,
• If there is a performance bug introduced in a build, it’s much quicker to find the delta between the builds and zero in on the bug by applying exclusion on the delta.  It often happens that when the delta is reasonably small, developer(s) can spot the performance bug immediately.

     Is early performance testing possible when development has just started?  Not always, but early performance testing should be done as soon as it becomes possible - definitely before the end of the project.  Once it becomes possible, performance test should be performed regularly just like regular testing to find bugs introduced in various builds. Performance bugs are just as nasty as functional bugs and sometimes performance bugs can be harder to hunt since they are typically correct in logic.

     Serious performance issue can often derail a product, cause a project to miss its deadline.  Are you ready to use one ounce of performance testing and save one pound of cure?

Random element in an array

    In performance testing, we sometimes need to deal with a list of values.  For example, when we test an e-commerce web site, we may need to extract a list of products from a page.  With the list/array extracted, we can do operations such as randomly picking an item to view or to add to cart.  On some platform, this logically simple operation can be a daunting job.

    Saw an interesting blog by Howard Clark on testing Peoplesoft Financials 9.0 using loadrunner: "Randomly Selecting an Array of Values and Using That Value As A Parameter".  Here is the script to pick a random BUID copied from that blog. It's listed here just to show how complex it is - you don't have to read it.

 int TotalNumberOfBUIDs;  
 char TotalNumberOfBUIDschar[3]; //working variable  
 char *AvailableBUIDsparam; //working variable  
 web_reg_save_param(“AvailableBUIDs”,  
 “LB/IC=class=’PSSRCHRESULTSODDROW’ >”,  
 “RB/IC=”,  
 “Ord=All”,  
 “Search=Body”,  
 “RelFrameId=1″,  
 “Notfound=error”,  
 LAST);  
 TotalNumberOfBUIDs=atoi(lr_eval_string(“{AvailableBUIDs_count}”));  
 TotalNumberOfBUIDs = rand() %TotalNumberOfBUIDs;  
 lr_output_message(“%d”, TotalNumberOfBUIDs);  
 itoa(TotalNumberOfBUIDs, TotalNumberOfBUIDschar, 10); //working variable conversion  
 lr_save_string(TotalNumberOfBUIDschar, “BUIDindex”);  
 AvailableBUIDsparam = lr_eval_string(“{AvailableBUIDs_{BUIDindex}}”);  
 lr_save_string(AvailableBUIDsparam, “BUIDs”);  
 lr_save_string((lr_eval_string(lr_eval_string(“{BUIDs}”))), “BUID”);  
 “Name=VCHR_ERRC_WRK_BUSINESS_UNIT”, “Value={BUID}”, ENDITEM, //application of the value that was captured and then randomized  

    While I was impressed by the author's skills in C,  I couldn't help wondering how an average test engineer can do this.  They know their test subject well, but writing such complex C program can be well above their heads.   On Netgend platform, it's so much easier.

 AvailableBUIDs = substring(str, "class=’PSSRCHRESULTSODDROW’ >", "<", "all");  
 TotalNumberOfBUIDs = length(AvailableBUIDs);
 BUIDindex = randNumber(0, TotalNumberOfBUIDs-1); //indexing is 0 based  
 BUID = AvailableBUIDs[BUIDindex];  //variable "BUID" now contains a random value.

    Even though we tried to use the same variable names as in loadrunner script, our script is still much shorter and easier to understand.  Here are some of the reasons for the simplicity:

• In a NetGend script, all variables are local to an instance/Vuser.  There is no need to use API to get the value of a variable (called "parameter" in Loadrunner) into code space or write the value back to the variable after some operations in the code space.
• NetGend supports the array variable.  In this case, the variable "AvailableBUIDs" holds the array of extracted BUIDs.
• You can get the length of the array by the function "length()".
• You can get any element in the array by indexing using [].  For example: AvailableBUIDs[BUIDindex] gives the value of "BUIDindex+1"th element. (it's 0 based)

    Note that the above short script (4 lines) can be made even shorter.   The readability may suffer a little bit but it's still fairly easy to understand. 

 AvailableBUIDs = substring(str, "class=’PSSRCHRESULTSODDROW’ >", "<", "all");  
 BUID = AvailableBUIDs[randNumber(0,length(AvailableBUIDs)-1)];  

    A good performance test platform needs to make complex test scenarios simple, not the other way around.

HTTP Performance testing made simple

    HTTP is the most popular protocol because it's the foundation of any web site.  Over the years,  it has gotten more complex because people tweak it all the time.   So has performance testing.   There are lots of options on many platforms when testing HTTP/Web server.  As we shall see,  performance testing on HTTP can actually be fairly simple, even when you are emulating 50,000 HTTP clients (VUsers).

    This is what a pair of basic HTTP transactions look like in javascript on Netgend platform.

 action(http,"http://www.example.com");  

 #HTTPS transaction very simple too, just change "http://" to "https://"
 action(http,"https://www.example.com)  

     Piece of cake!

    A little more advanced example is to emulate a user clicking through a sequence of pages.

 for (id = 1; id < 5; id ++) {  
   action(http,"http://www.example.com/docs/tutorial${id}.html"); 
   sleep(10000);
 } 

    Here the "sleep 10000" simulates a user spending 10 seconds (10000ms) reading the content.  You can make the time to be random, i.e. replace the "sleep 10000" with the following two lines to make it more realistic.

   x = randNumber(2000, 30000);  
   sleep(x);  

     HTTP POST is important because it's used in submitting a form, login, registrations and uploading..
This example shows how easy it is

 http.POSTData = "username=jsmith&pass=letmein";  
 action(http,"http://www.example.com/login?");  

    With the support for variables in netgend javascript, we can add some variable part to the POST data.

 name = randString(5,10);
 http.POSTData = "username=${name}&pass=letmein";  

     HTTP header is absolutely important,  with netgend javascript, you can add any HTTP header,  in particular, you can add

• REFER header (which is useful in server security protection against CSRF) 
• User-Agent header so that your client will emulate an IE, or Firefox or Chrome.

 httpHeader.REFER = "http://www.example.com/support";  
 httpHeader."User-Agent" = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)";  
 name = "John";
 http.POSTData = "username=${name}&pass=letmein";  
 action(http,"http://www.example.com/login?");  

     Note that in the scripting, "User-Agent" header is enclosed with double quote, this is to prevent the "-" in the name from being mis-interpreted (as a minus sign).

     It's common to have a test reading data from a csv file and uses that data to "fill" a form and submit to server.  Here is an example:

function userInit() {
     var db = readCSVFile("data.csv");  //global variable
     var id = 0;                        //global variable
}
function VUSER () {
     r = db[id];  
     id ++;  
     info.name = r[0];  
     info.age  = r[1];  
     info.city  = r[2];  
     info.state  = r[3];  
     http.POSTData = toJson(info);  
     action(http,"http://www.example.com/register"); 
}  

    In this example, each VUser (client) will read a line/row from the csv file.  Each row of csv file may have multiple fields:   "name",  "age, "city", "state".
    For instance,  one row may look like  "John,23,Chicago,IL".   The "toJson" function will turn a dictionary variable ("info" in our case) into a json record like
{"name": "John", "age": "23",  "city": "Chicago", "state": "IL" }

     As a final example in this blog, we show how to generate xml message as HTTP Post data.

 function VUSER () {  
      name = "John";   
      age = 23;   
      city = "Chicago";   
      state = "IL";   
      http.POSTData = q|<user>   
      <name>${name}</name>   
      <age>${age}</age>   
      <city>${city}</city>   
      <state>${state}</state>   
 </user>|;   
      action(http,"http://www.example.com/register");  
 }  

    It looks very intuitive, doesn't it?   By the way, you can expand this example so that it will read data from the csv file just like the previous example.   Just don't forget to change the variables in the xml template string from "${name}"  to "${info.name}",   "${age}" to  "${info.age}" etc.

   There are many more interesting examples in performance testing on HTTP/web servers, we will cover them in other blogs, stay tuned.

A server performance testing platform that is scalable, flexible and simple

    Loadrunner has long been the leading product as server performance/stress testing tool. It's flexible, supports lots of protocols and features. but it's also

1. Not scalable - one unit only supports up to a 1,000 VUsers.
2. Complex to use - one needs very good C programming background to take advantage of it.
3. Expensive - one system emulating 1,000 VUsers may cost close to $92,000 for 500 VUser license.

    There are many other performance testing platforms in this area, but they are less flexible compared to Loadrunner and suffer from the same scalability limitation.   Netgend has created a new testing platform with breakthroughs in both scalability and flexibility.

• Scalable - one system can emulate over 50,000 VUsers,
• Flexible,
• Complex parameterizations is made easy, 
• Generates dynamic messages to send to server,
• Processes server response in a very flexible and easy way,
• Emulates any networking protocols, open or proprietary,
• Simple to use - complex parameterizations are made easy on this platform.

The platform runs both in cloud and on hardware such as the following mini PC or a industry server.

The mini PC platform may look small, but it's powerful enough to bring a big server or server farm to its knees.

    Flexibility in client emulation comes from the fact that parameters (a.k.a variables within the scope of a VUSER) are built in.  You can use the value from a parameter and/or do operations on a variable directly.   No need to pull the parameters into code space and do some operations and (optionally) write it back to parameter as on other platforms.

   For example, the following code snippet can emulate many stations who send heartbeat info to master server.

 //javascript syntax is used on Netgend platform
 count = 1;
 stationId = getGlobalId(); 
 //global ID will start from 1, incremented by one with each call to this function.  
 while (1) {  
   send("heartbeat count ${count} on station ${stationId}");    
   count ++;  
   sleep(10000);  
 }  

     As shown in this example,  operation on variable "count" feels natural and creating a dynamic message is easy.

    There are lots of examples in the documentation and will come in future blogs.

    This platform is designed for those users who know some basic programming but don't have complex coding background like seasoned programmers/developers.  These users will be more productive on this platform than experienced programmers/developers on other platforms.

    For more details, please check out Netgend and sign up for free - the cloud based platform is free to use during beta period.

Disclaimer: I work for Netgend.