Saturday, November 30, 2013

Javascript processing in Performance testing

    Thanks to a tweet from Stuart, a guru on performance testing,  I checked out the blog by HP loadrunner that shows how to do authentication with client side javascript evalution.  One key point in the blog is an interesting Loadrunner function called "web_js_run".  This function can be used to evaluate a javascript code snippet.   In this blog, we are going to show that our NetGend platform also supports javascript evaluation, only easier.

    To paraphrase the test scenario (with a slight change) ,  a client needs to do the following 3 steps
  • sends a request to an authentication server to get the challenge string,  the server response is in the format of JSON
  • Extracts the last element (the real challenge string) from the JSON string and concatenates it with a local passcode and do the encryption. 
  • sends the encrypted string to the server to get a token (assume authentication is successful).
    Part of input are two javascript files, crypto.js and calcPassword.js that are used by a browser to do the job.
  /////file crypto.js
  function encrypt(data) {   
    return data.toString().split("").reverse().join("");   
  //simplified encryption to illustrate the point of evaluating javascript on client side

 // file calcPassword.js  
 function extractJson(stringData) {
   var data = JSON.parse(stringData);    
   var index = data[data.length - 1];    
   return data[index]; 
 function getPassword(stringData){    
   return encrypt(extractJson(value));    

    Here is the NetGend script. Note that we have two functions related to Javascript evaluation:

  • evalJS(..) can evaluate a javascript, either from a file (specified by its file name) or just a string of javascript code snippet (default case).
  • JE(..) will evaluate a javascript function, first parameter is the javascript function name.

 function userInit() {  
    evalJS("crypto.js", "file"); 
    evalJS("calcPassword.js", "file"); 
    //now javascript engine will understand functions "encrypt"  "extractJson"
 function VUSER() {  
    challenge = je(extractJson, http.replyBody);
    localPasscode = "abc123"; //can also read from a csv file
    password = je(encrypt, "${challenge}${localPasscode}");  
    password = toUrl(password); //encoding password to URL  
    action(http, "http://localhost:3000/token?password=${password}");  
    token = http.replyBody;

     As a contrast,  here is the original loadrunner script in the blog.  Note that the script would have been much longer if it has to do the concatenation without javascript.
           "File=crypto.js", ENDITEM,  
           "File=calcPassword.js", ENDITEM,  
           "Code='http:/'+'/localhost:3000/token?password=' + encodeURI(LR.getParam('password'));",  
      web_custom_request("get token",   
      return 0;  

    Being able to support javascript in performance testing is important, does your favorite test platform support it? Is it easy?

Thursday, November 28, 2013

Loadrunner script vs NetGend javascript script: a more complete comparison

    After a few blogs with code snippet showing the differences between Loadrunner syntax and NetGend javascript syntax,  I got an excellent request:  "can you compare a full Loadrunner script with a full NetGend javascript"?

    I like this idea because it not only gives a better idea on how the NetGend javascript works in more detail,  it also shows the possibility of translating a loadrunner script into (much simpler) NetGend script.
    The loadrunner script is based on a real one from production test,  it emulates VUsers performing the following actions: 1) login to a conference web site,  2) set up a conference,  3) update the conference  4) cancel the conference and logout.

    To save the size of this blog, I combined the vuser_init, vuser_end and action parts together and trimmed  the number resources for transactions.  By comparing it with NetGend script, you can see

  • Assigning to a parameter like "URL" is much simpler with NetGend Script
  • NetGend functions
    • function names are shorter and easier to remember
    • function parameters are shorter too, they are easier to understand.
  • Variable substitution is very close between the two scripts
    • with Loadrunner, you do it with {VARNAME}
    • with NetGend,  you do it with ${VARNAME}
    • One subtle but important differenence on variable substitution 
      • In NetGend, it happens in all strings (including all string parameters of functions), 
      • In Loadrunner, only certain parameters of certain function are capable of doing substitution.
  • On NetGend platform, when there are many resources to download, it will spawn many sessions to retrieve the resources at the same time.
      Here come the scripts, first one is for loadrunner, the second one is for NetGend. They are a little long.
 Action()   //Loadrunner script
      char Url[20];  
      sprintf(Url ,"");  
      lr_save_string (Url,"URL");  
           "Url=/images/bg_content04.gif", "Referer=http://{URL}/loginAction!getCookie.action", ENDITEM,   
           "Url=/images/bg_loginLeft_bg01.gif", "Referer=http://{URL}/loginAction!getCookie.action", ENDITEM,   
           "Name=password", "Value={Password}", ENDITEM,   
           "Name=vercode", "Value=", ENDITEM,   
           "Name=qsbycookie", "Value=<OFF>", ENDITEM,   
           "Url=/images/bg_welcome01.gif", "Referer=http://{URL}/mainAction.action", ENDITEM,   
           "Url=/images/bg_contentHome01.jpg", "Referer=http://{URL}/mainAction.action", ENDITEM,   
           "Url=../css/jquery.datepick.css", ENDITEM,   
           "Url=../images/bg_button07.gif", ENDITEM,   
           "Name=rv.conferenceId_by", "Value=", ENDITEM,   
           "Name=rv.confStatus", "Value=0", ENDITEM,   
           "Name=rv.confName", "Value=test_per_update", ENDITEM,   
           "Name=rv.dateStart", "Value={Date}", ENDITEM,   
           "Url=../images/bg_H02.gif", ENDITEM,   

      web_reg_save_param ("conferenceid","LB=<a href=\"/reserve/confView.action?conferenceid="",
                              RB="\" reserve="">",
  "Url=../images/bg_content01.gif", ENDITEM, 
  "Url=../images/bg_content02.gif", ENDITEM, 


 lr_think_time (30);



  "Name=rv.conferenceId_by", "Value={conferenceid}", ENDITEM, 
  "Name=rv.confStatus", "Value=0", ENDITEM, 
  "Name=rv.contactphone", "Value=", ENDITEM, 
  "Name=rv.confName", "Value=test_per_update{NameNumber}", ENDITEM, 
  "Name=rv.dateStart", "Value=2013-11-28", ENDITEM, 

 lr_end_transaction("update", LR_AUTO);

  "Name=conferenceid", "Value={conferenceid}", ENDITEM, 

 lr_end_transaction("Cancel", LR_AUTO);   
      return 0;  

     Here is the equivalent script in NetGend Javascript.
 function VUSER() {  //NetGend Javascript
      URL = "";  
      action(http, URL); 
      httpHeader.Refer = "http://${URL}/loginAction!getCookie.action"; 
      spawn( ["/images/bg_content04.gif","/images/bg_loginLeft_bg01.gif"]);  

      http.POSTData = "password=${Password}&vercode=&qsbycookie=<OFF>";  
      action(http, URL);  
      httpHeader.Referer = "http://${URL}/mainAction.action";  
      spawn( ["/images/bg_welcome01.gif","/images/bg_contentHome01.jpg"]);  

      spawn(["../css/jquery.datepick.css", "../images/bg_button07.gif"]);  

      b."rv.conferenceId_by" = "";  
      b."rv.confStatus" = 0;  
      b."rv.confName" = "test_per_update";  
      b."rv.dateStart" = currentTime();  
      http.POSTData = combineHttpParam(b);  
      httpHeader.Referer = "http://${URL}/reserve/conf2_readyForCreate.action";  
      action(http, "http://${URL}/reserve/conf2_reserve.action");  

      conferenceid = substring(http.replyBody, "<a href=\"/reserve/confView.action?conferenceid=", "\" >");  
      httpHeader.Refer = "http://${URL}/mainAction.action";
      spawn(["../images/bg_content01.gif", "../images/bg_content02.gif"]);  

      httpHeader.Referer = "http://${URL}/reserve/conf2_list.action";  
      action(http, "http://${URL}/reserve/conf2_readyForUpdate.action?conferenceid=${conferenceid}");  

      rendv("update", 40); //gather 40 VUsers here before moving on to next step  

      httHeader.Referer = "http://${URL}/reserve/conf2_readyForUpdate.action?conferenceid=${conferenceid}",   
      c."rv.conferenceId_by" = conferenceid;  
      c."rv.confStatus"   = 0;  
      c."rv.contactphone"  = "";  
      c."rv.confName"    = "test_per_update${vuserId}";  
      c."rv.dateStart"    = "2013-11-28";  
      action(http, "http://${URL}/reserve/conf2_update.action");  

      httpHeader.Referer = "";  
      http.POSTData = "conferenceid=${conferenceid}";  
      action(http, "");  

      httpHeader.Referer = "http://${URL}/reserve/conf2_list.action";  
      action(http, "http://${URL}/logoutAction.action");  


Tuesday, November 26, 2013

Know yourself when choosing the right web hosting 

    There are a lot of web hosting companies out there, many claim to have some sort of accelerators that can significantly speed up your web pages.  While some claims are true, it's better to put them to test.

    Recently I looked at a web hosting company,  found one of its customers by going to "testimonials" section and picked a "happy" customer.  When I opened this customer's site in my browser, I am surprised that it was not that snappy (despite the good words from the "happy" customer).

   To get a better measurement, I opened Chrome's developer console and clicked a new link.  The network tab in developer tool showed that the main HTML page itself (not including the resources like css, js, images ...) spent more than 2 seconds just in the "waiting" period.  To those who are not familiar with this term, "waiting" period is the time between transmission of the HTTP request and arrival of first byte of HTTP response comes. Interestingly, if I refresh the page, the waiting time is only a little more than 100ms.   The most likely reason is that data is cold in the database (not in the cache).

   What if I refresh the page multiple times and what would the waiting times be?   Since it's tedious to do it manually, I used a simple script (using netgend javascript syntax) to do the measurement.  Here the URL is changed to keep the web hosting company anonymous.
 function VUSER() {  
      i = 0;  
      while (i < 10) {  
           action(http, "");  
           printf("%.2f\n", http.initialRespTime);  
           i ++;  
    It produced the following data (in ms):

    From this we can observe that:

  • After the first 2 attempts, the server is fairly responsive.  ~80ms is quite fast considering the network latency itself is about 40ms.
  • Occasionally the latency goes a little higher (like the 231ms in the above) but still acceptable, could be because server is busy handling other requests.

     We knew the (most likely) reason why the first waiting time is long, but why was the second waiting time is high too?   I couldn't think of a reason.  Even looked at packet capture when I ran the test and it confirmed the timing reported by test tool.  I can only guess that it's introduced by the acceleration engine of the web hosting infrastructure.

     Based on the numbers, we could see that the accelerator may be working, but does it help if you are a site owner? It depends:

  • If your web site is lightly visited,  your visitor may have a not-so-great experience (every page that he clicks will be cold in database).  The benefit of the accelerator may be questionable.
  • If your site is heavily visited,  majority of the visitors will feel your site is snappy!

     In summary, you, as a site owner, need to know your site before choosing the "right" web hosting service.

Monday, November 25, 2013

Performance testing with different IP addresses

    Recently I saw a web hosting company which has a creative approach on charging:   by page views on a given day,  defined as the number of visitors to the web site with unique IP addresses.    This is interesting in that it protects the site owner from being charged when a hacker uses a tool and generates lots of "hits" to the web site.   Being able to emulate different IP addresses in testing such an infrastructure will definitely be helpful.

   We mentioned in earlier blogs that that a Netgend performance test platform can emulate 50,000 VUsers.  In fact, it can maintain 50,000 concurrent TCP sessions.  So, can all the VUsers have unique IP addresses? The answer is yes.

   To do this, we just need to specify the following parameters:
  • Ranges of IP addresses
  • virtual IP
  • gateway IP. 
  • route to server
    As an example, suppose

  • You connect the test platform to an interface of a router,  the router's interface IP address is,  
  • You want to emulate IP addresses in two ranges:,,  
  • The target servers have IP addresses in
    Your configuration can be the following:

 virtual router IP:  
 gateway IP:  
 route to Servers:  

    Note that the "virtual router IP" above can be considered as a "gateway" to IP addresses in the ranges (client IP addresses).  The physical router will need it to set up route to reach the client IP addresses.

    On many OS, one can assign multiple IP addresses to an interface. But it is not scalable in terms of the time consumed in configuration and the number of IP addresses allowed.   One can also try user-space TCP/IP stack, but it may miss many features of a standard TCP/IP stack on  windows, Linux or Mac.   On the netgend platform we use an innovative method to "enable" an OS to have many different IP addresses   You can ping them to verify.

    Other than testing the above infrastructure, another benefit of using unique IP addresses in performance test is on trouble-shooting.  Suppose you have network sniffer (like Wireshark) that captures all the packets during testing, it will be a nightmare if all the sessions are from the same IP addresses. With different IP addresses for clients,  it will be much easier to keep track of any of the clients - you can extract all the packets related to a client and study them.

   With social media gaining momentum,  I am sure there will be more examples of server infrastructure that can make good use of the IP addresses from clients.   Can your favorite performance testing platform support it?


Thursday, November 21, 2013

Story of a ghetto coder

    Believe or not, I was called a ghetto coder when I was doing testing automation.  The story was like this, a software feature was implemented, but missed an important feature:  "Save As".  So users of the system can't save his/her configurations.  I wrote a tool that extracts relevant part of user configuration on that feature from the system configuration (a big xml blob) and save it to a file and be able restore the configuration from a file.   This workaround served the purpose of "Save As".  I was happy and expected some compliments, but was surprised that one "compliment" was "ghetto coder".    Why?  I did the extraction(from a big xml string) by regular expression :-(     Later on,  I changed the tool to use tdom package - the official and better way on extraction from xml string, but the nick name remained.   I was  actually happy at the outcome since many colleagues got to know this nickname.  Whenever they needed some tools urgently, they came to this "ghetto coder" and they usually got something working shortly after.    I guess the word "ghetto" has turned into "quick and dirty" and it can be helpful in some cases.

     It reminded me of the story when I saw someone asked a question on performance testing:  Given a server response like the following,
   "id" : {
    "idUri" : [ "/id/123123" ]
how to extract the field "/id/123123"?   The answer was given using regular expression:
"idUri" : \[ "([^"]+?)" \]
    While the regular expression works for this particular string, I was worried that if the spacing changes a little bit (like some white space was removed or added, a new line was inserted after "["),  this regular expression may fail.

     With Netgend javascript,  it's fairly easy to extract it
 a = fromJson(str);  
 //now variable[0] contains the desired field  
     Here the structure of the variable[0] follows the hierarchy of the json message, so it's fairly easy to understand.  Note that in particular, the array ("idUri" in this case) is 0 based, so idUri[0] means the first element.
     This will work no matter how the spacing changes. Should there be a second element in the "idUri" array, it can extract that element too (just use[1] instead) .  On top of all these,  this is much easier for someone with basic programming skills than the complex regular expression.

    My experience as "ghetto" coder taught me that it's ok to write "ghetto" code,  but you or the users need to be aware of the possible conditions that can cause it to break. Also, if there is an easy way to do the task in a better way (like using tdom package to extract part of xml string), jump on it.  

Wednesday, November 20, 2013

Splitting string in performance testing made easy

    In previous blogs, we showed Netgend platform can not only emulate a lot more VUsers than Loadrunner, it's also more flexible and easy to work with.   In this blog, we give 2 more examples, all related to string splitting.  Due to lack of support for it in Loadrunner,  there seems to be a lot of such questions asked in various forums.

     The first example is based on Shankar's blog, we need to split a string like the following using delimiter ';' and access various pieces after the split.
    The solution shows that the author has a pretty good background in C programming that many test engineers don't have, here is the proof :-)
  //Loadrunner code snippet from the above blog
 char *value="040;350;07/05/2012";  
 char *temp;  
 temp=(char *)strtok(lr_eval_string("{TokenValue_1}"),";");  
 lr_output_message("the string a = %s",lr_eval_string("{TokenValue1}"));  
 temp=(char *) strtok(NULL,";");  
 lr_output_message("the string b = %s",lr_eval_string("{TokenValue2}"));  
 temp=(char *) strtok(NULL,";");  
 lr_output_message("the string c = %s",lr_eval_string("{TokenValue3}"));  

    Here is how we do it in netgend javascript:
 //assume str contains above mentioned string
 p = splitStr(str, ';');  
    Now the variable "p" will hold all the pieces after the splitting,   p[0] is the first piece, p[1] is the second piece ...
     Much easier, isn't it?  What's more, you can use a variable to index into the pieces
 i = 2;  
 //now p[i] will contain the 3rd piece (indexing is 0 based)  
     Try doing that with Loadrunner :-)

    The other example is a little trickier.  Here we need to split the following string with a variable delimiter:   &#43; or  &#47;
You can image that the Loadrunner solution would be even more complex than the previous example.   With netgend javascript,  it's still fairly easy, thanks to the support for regular expression in the function splitStr().

 //assume str contains above mentioned string
 p = splitStr(str, '&#4[37];');   
   In the above code snippet, the delimiter is a simple regular expression,  it means string "&#4" followed by either  "3" or "7" and then followed by ";".

   There are lots of interesting and challenging tasks related to string manipulation in performance testing, we will show that anyone  with basic programming skills can do them!

Tuesday, November 19, 2013

What makes performance testing so difficult?

    It's interesting to study questions asked in various forums on performance testing.  Most of time the questions are technical like "how to extract certain fields in a JSON message?".  These seemingly simple questions were given non-trivial (and sometimes incorrect) solutions using complex regular expressions.  This must have frightened some users into thinking their skills are seriously insufficient when they ask the general questions like "how would I improve my skills in performance testing?".  It makes me wonder why the performance testing platforms are not made easier.

    Many test platforms are more for developers or someone with considerable amount of programming experience.     To effectively use them,  one has to be able to write non-trivial code and be able to do debugging.  As an example,  let's look at an issue with loadrunner, the leading performance testing tool.

    Since string replacement function is not provided in loadrunner by default,  in this blog Kim gave a good workaround on string replacement.  It can be used to replace all the "+"s in the following string to be "%20":

    While it's impressive to see this nice implementation,  I can't help wondering why it is so complex and how many users are as advanced as Kim.      Logically speaking, replacing parts of a string is pretty simple.   Turns out the difficulties come from the concept of  "parameter" in performance testing.   In performance testing, we need to emulate many virtual users (VUser).  Each VUser will have its own variables, states etc.   Those variables or states are called "parameters" in loadrunner's jargon.   In loadrunner, when you want to perform a operation on a variable,  you have to "pull" its value to code space (a variable in C), perform the operation(s) on the variable in C and then write it back to the "parameter".

      On the netgend platform,  we don't believe it's necessary to require the users to have more than the basic programming skills.  We designed our test platform so that all per-VUser variables can be easily accessed.     As a result,  many functions such as string replacement function are very easy to implement  and are included in the test platform by default.  No need to be creative and hack up an extension or a workaround.

     Here is the syntax for "replace" function:
 replace(<sourceString>, <from-string>, <to-string> [,"all"])  

     The following a simple example on how to use it:
      str = "this is John, John Dole";  
      replace(str, "John", Joe, "all");  
      //str will be "this is Joe, Joe Dole"  
     Note that the last parameter "all" means we will replace all the occurrences of "John".   You can leave out the fourth parameter and it will only replace the first occurrence.

     What's more,  the from-string or to-string themselves can be a variable.
      str = q|this is John, John Dole|;  
      newName = "Mary";  
      replace(str, "John", newName, "all");  
      This can potentially be used to read new names from a csv file and do the substitution dynamically. In Loadrunner, even with the nice implementation of string replacement function mentioned above,  one still needs to do more work if either "from-string" or "to-string" are parameters instead of hard coded strings.

      There are lots of areas where we can use our creativity,  I want to use it where simple things are kept simple. What about you?

Saturday, November 16, 2013

Performance testing on social network, location reporting

    Social networking has became wildly popular, thanks to omnipresent smart phones and apps.  Many interesting smartphone apps depend on user's locations to recommend restaurants or shops.  What does this have to do with performance testing?

    In order for servers to send recommendations, they need to know the current locations of the users.  As a result, the smartphone apps will need to periodically send location updates as users move around. It may not be very easy if your servers need to handle millions of smartphones that send updates periodically and run through some algorithms, query some databases to get the recommendations and send them back to users.  It's really important to have a good performance test on the infrastructure (web servers and their backends) before you open it up to the wide audience.

    How do  we do performance testing on this kind of social networking apps?  Simply emulating a lot of users and sending dummy location updates may not test the servers well enough.  To better test the system, in particular the algorithms,  we need to emulate a lot of users roaming around in a more realistic way and send location updates.

    Netgend platform not only can emulate a lot of users (50,000 on one system),  it can also emulate users flexibly and realistically.  We can implement this scenario with a csv file that represents a map.  As an example, based on the following little map,  our csv file will have
  • 9 intersections (numbered 1,2, ...9, left to right, top to down).  each one with its latitude and longitude.
  • segments of streets (hereby called streets) connecting these intersections.  For example, "1,2" means the street connecting intersections 1 and 2.  "1,4" means the street connecting intersections 1 and 4.  There are 12 streets here.

     Here is a quick look at the file.
 ....some lines skipped  
 ....some lines skipped  

     Now it's time to introduce our script to emulate those users who roam around and send location updates:
 function userInit() {  
 function VUSER() {  
      position = randPosition(); 
      //some steps to sign in, omitted here
      while (1) {  
           move(position, 30);   //move 30 feet
           p = getLoc(position);  
           http.POSTData = q|{"latitude": ${p[0]}, "longtitude": ${p[1]} }|;  
           action(http, "");
           sleep(10000);  //10 second

     In the above script,  we first create a (implicit) map object from the csv file by using the function "loadMap".   This is done only when the test starts.
     In the VUser script part (executed by each VUser), we start with a random position,  then we move 30 feet every 10 seconds and update our latitude and longitude.  For the curious minds,  here is what the two functions "randPosition()" and "move()" do.

  •  randPosition():  it will randomly pick an intersection and then find one of its neighbor intersections, these two intersections determine a (segment of ) street.  The user will move from the intersection initially picked and move to the neighbor intersection.   After that it will pick a random point on this segment of street and return it to user's "position" variable.
  • move(),  will move user's position by specified distance (in feet) and update its internal coordinates.  If user comes to an end of a segment, i.e. comes to an intersection,  it will use new intersection as the a starting point and randomly pick one of its neighbor intersections (but not to the intersection it comes from) and continues the moving. 
    There are a couple of ways to make it more realistic (hence more complex):  make each user move at different speeds;   make each user move toward a random destination etc.  All these cases can be supported on this test platform.

    Social networking is going to be hot, is your infrastructure ready? 


Thursday, November 14, 2013

Processing html message, from regexp kung-fu to DOM parser

    There is no doubt that regular expression (regexp) is useful in performance testing. It can be handy when we need to extract some fields from a server response.  Most performance testing platforms support it, the question is how flexible/easy it is supported?

    To get an idea, let's look at an example based on an interesting question asked in stockoverflow.  Give a sample html message like the following, we need to extract the URLs within "h3" nodes.

     <title>A sample webpage!</title>  
 <h3 class="content-title">  
 <!-- change when this is completed -->  
   <a href="/container/recentIssue.jsp?punumber=201"> Title 1 </a>  
   <a href="/container/recentIssue.jsp?punumber=101"> Title 0.1 </a>
 <a href="/container/mostRecentIssue.jsp?punumber=999">abc</a>  
 <h3 class="content-title">  
 <!-- change when this is completed -->  
   <a href="/container/mostRecentIssue.jsp?punumber=202">  
   Title 1  
     It's easy to come up the regexp to extract an URL:  /href=\"([^\"]+)\"/ ,   the hard part is to do it within a h3 node.
    Multiple solutions were proposed, but it was not clear they will skip those that are outside of a h3 nodes (such as the one with value "999").   The problem may appear simple at first,  but it's not trivial to solve it with only one regexp.    One reason is, within a h3 node, there may be unknown number of hyperlinks with punumber.  It's obvious how to use 2 regexps to solve this problem. First we use a regexp to grab the h3 nodes, then we use the simple regexp /href=\"([^\"]+)\"/  to grab URL in each of the grabbed h3 nodes.  I am not sure how other testing platforms support this type of operation,   it's fairly straightforward on netgend platform.
 function VUSER() {  
      a = regexp(http.replyBody, /(?s)<h3 (.*?)<\/h3>/g);  
      i = 0;  
      while (i < getSize(a)) {  
            b = regexp(a[i], /href=\"([^\"]+)\"/g); 
            i ++;
            j = 0;
            while (j < getSize(b)) { 
                  j ++;
In the above code snippet, we first use regexp to grab all h3 nodes, assign it to array variable "a", then for reach element in "a", we then use regexp to grab all the links.  Note that the "g" after regular expression instructs the regexp engine to get all the matches.  Also note that the "(?s)" in the first regexp means we are going to do match with the string being considered as one line.

    Of course, in real world, requirement may be more complex. For example, we may need to do a HTTP transaction for each of the URLs extracted.  It's not hard on netgend platform.  We just need to replace the "printlin(b[j])" with the following line:
 action(http, "${b[j]}");  

     Extraction of fields using regular expression is handy, but it's not robust, for example, when the relative positions of attributes change, some regexps may give bad surprises.   In the case of html message, it's better to use DOM parser function "fromHtml".   This function takes the HTML message as one argument and a XPath as another argument.  It's easier  (provided that you know the XPath expression) and more robust.  Here is a solution to the same problem using "fromHtml" function.
 function VUSER() {  
      action(http, "");  
      a = fromHtml(http.replyBody, '//h3/a/@href');  
      i = 0;  
      while (i < getSize(a)) {  
            i ++;
     Much simpler, right?  But you would need to learn a little bit of XPath.   In case you can't use XPath (for example, the message is not HTML or XML),  you can apply your  regexp Kung-fu on netgend,   a platform that can emulate 50,000 concurrent sessions.

Wednesday, November 13, 2013

Turn headache into pleasure, performance testing on JSON/JSONP messages.

    In previous blogs, we have touched on JSON messages processing.  Here we want to talk more about it partly because JSON has become the standard messaging format between clients and servers, you will be guaranteed  see more of it in performance testing.

    We showed how easy it is to parse JSON message and access any of its fields on netgend platform .  What about JSONP message?

    First let's find out what JSONP message is.  JSONP message can be seen as a workaround for  browsers' single origin policy (SOP):   to get some data from another web server (which is prohibited by SOP), client pretends to ask the web server for a piece of javascript code (not restricted by the SOP),  the web server will "pretent" to send you a piece of javascript code which is actually a JSON message wrapped in a simple javascript statement or function call.  Clever, isn't it?

    A simple JSONP message looks like the following.  In practice, the JSONP message can be much more complex.
 parseResponse({"Name": "Foo", "Id": 1234, "Rank": 7});  

    Now we know a JSONP message is closely related to a JSON message,  how do we process an incoming JSONP message in performance testing?   Of course,  you can use some regexp kungfu to extract the JSON message from a JSONP message and call "fromJson(..)" function on it.  It's not necessary though,  the implementation of the function "fromJson(..)" will detect whether the message is a JSONP message, if so, it will extract the JSON part and parse it.   In another word,  "fromJson(..)" function works both on JSON messages and JSONP messages.

 action(http, "");  
 a = fromJson(http.replyBody);  
 //the http.replyBody may be a JSONP message  
 //now you can access fields in the JSONP message and use it to fill a html form  
 form.destinationCity.value = a.suggestion.fullName;  

    So far, we have talked about processing incoming JSON message,  what about generating a JSON message to send to a server?  In that case, we can use toJson() function. = "Chicago"; = "Mary Dole";  
 a.age = "23";  
 msg = toJson(a);  
 //msg will contain string
 //{"city": "Chicago", "name": "Mary Dole", "age": 23}  

    toJson() function is nice, but sometimes we need to generate a JSON message based on a JSON message template,  like the following.
Many testing platform supports this.  However,  you have to escape all the double quotes in the template:
    This JSON template is very short and it's already tedious to escape all the double quotes.  You will be delighted to know that on netgend platform, you don't have to do that.
 city = "Chicago";  
 name = "Mary Dole";  
 age = 23;  
 str = q|{   
 http.POSTData = str;  
 action(http, "")  
     In the above, all the variable substitutions between "q|"  and "|" will happen as expected,  all space, newline and double quotes are preserved,

     Processing incoming JSON message and generating outgoing JSON message can be a headache on other testing platforms, but on netgend platform,  it's a pleasure.

Tuesday, November 12, 2013

Performance testing on forms

    HTML form is an important part of internet, without it, we will not be able to interact with web server, like placing an order, booking a hotel/flight etc.

    In previous blog, we talked about performance testing on search forms.   But it didn't cover the more complex cases such as select, radio button etc which are typical on HTML forms.  In this blog, we are going to focus on these complex cases and show that the netgend javascript function "getFormField()" can easily take care of them.

   First let's warm up with something simple, this can be seen as a review of previous blog:   suppose we have a form with some text fields to be filled, like "firstName", "lastName", "city", we can generate http request like the following:

 action(http, "");  
 a = getFormField(http.replyBody, "registerationForm");  
 a.firstName.value = "john";  
 a.lastName.value = "smith";  = "chicago";  
 http.POSTData = combineHttpParam(a);  
 //post data will look like "firstName=john&lastName=smith&city=chicago"
 action(http, "");  
    In the above code snippet, second parameter to getFormField (in our case "registerationForm") is used to identify the right form in a HTML page.  When HTML page only contains one form, you can leave this parameter empty.

   Of course, this will be more interesting if the fields are read from a csv file.   Suppose we have a file called "users.csv", each of its rows contains first name, last name and city.
 function userInit() {  
  var list = readCSVFile("users.csv");   
  var i = 0 ;  
 function VUSER () {  
  item = list[i];   
  //here item is an array variable that contains the fields of the row in csv file  
  i ++;   
  action(http, "");  
  a = getFormField(http.replyBody, "registrationForm");  
  a.firstName.value = item[0];  
  a.lastName.value = item[1];  = item[2];  
  http.POSTData = combineHttpParam(a);  
  action(http, "");   

     Note that it will work for both textfield and text area.  The difference from user's point of view is that text field typically has one line, text area can have multiple lines.  In HTML, text field and text area are quite different. Luckily, our users don't need to worry about it -- they are taken care of by netgend test platform.

    What if the form has radio button?   In performance testing we would like to randomly pick a option. Let's suppose, we have radio button with name "vehicle" and it has 4 options: "bike", "car", "pickup", "truck",

 a = getFormField(http.replyBody, "registrationForm");  
 a.vehicle.value = randNumber(1, a.vehicle.size);  
 //...some more assignments to a...  
 http.POSTData = combineHttpParam(a);  
   Here the variable "a.vehicle.size" holds the number of items for a.vehicle, in this case, it's 4.  A good news here:  the above code snippet not only works for radio button, it works for "select" field (a.k.a dropdown box) of a form too.

    What about checkbox. Checkbox is different from radio button in that user can check zero or more items.  So in performance testing, we need to emulate selecting a subset of checkboxes.  How do we do this? randSequence(min, max) function can help.  It will return a subset of the sequence (min, min+1, ...max).  Suppose we have a checkbox group called "area", with items: "n", "nw", "w", "sw", "s", "se", "e", "ne", "c".

 a = getFormField(http.replyBody, "registrationForm");  
 a.area.value = randSequence(1, a.area.size);  
 //...some more assignments to a...  
 http.POSTData = combineHttpParam(a);  

    Finally we put all of them together,  suppose we have a form that has
  • select field called "color", 
  • radio button group called "vehicleType" and 
  • checkbox group called "make".   
Here is code snippet to do performance testing:

 action(http, "");  
 a = getFormField(http.replyBody, "findVehicleForm");  
 a.color.value = randNumber(1, a.color.size);  
 a.vehicleType.value = randNumber(1, a.vehicleType.size);  
 a.make.value = randSequence(1, a.make.size);  
 http.POSTData = combineHttpParam(a);  
 action(http, "");  

    Not bad for a "complex" form, right?

Monday, November 11, 2013

Syntax comparison between JMeter and Netgend

Jmeter is a great open source performance testing platform.  It's used by a large community of users and gain popularity at the cost of loadrunner, as shown in this report thanks to the job site Indeed.

    The Jmeter GUI is nice. But to gain performance advantage, one may have to run it on command line as suggested by this tip.  Also, you may have to be careful on how many threads to have, because  hundreds of active threads may affect performance. This creates a problem:  when HTTP transaction latency is high, say, hundreds of milliseconds, one needs to have a large number of VUsers (threads) to bring up the transaction rate, but a large number of threads may affect performance.

    These are not the toughest problems with JMeter,  the toughest problem is with parameterization or correlation:  user needs to be really good at Java programming to do it.   For many users with only basic programming background, this can be a serious challenge.

    Netgend javascript makes things simpler:
  • Don't have to worry about the number of threads,  one system can support about 50,000 VUsers.
  • Complex parameterization/correlation can be simple. 
  • Can view real time statistics on GUI while many VUsers are running.

    The following question asked on a forum provides an excellent example.   The question is on processing the following "complex" JSON message and extract the values of two fields "equipmentPartId",  "compositePartName".

       "compositePartName":"SRW224G4P-K9-AU ",  
    "status":"validation ok",  
    "messages":", Non Configurable"  
       "compositePartName":" UC560-T1E1-K9 ",  
    "messages":", Non Configurable"  
    Here is a solution given for JMeter.
 import org.json.JSONArray;  
 import org.json.JSONObject;  
 String jsonString = prev.getResponseDataAsString();  
 JSONArray equipmentParts = new JSONArray(jsonString);  
 print("parsed json");  
 JSONArray parts = new JSONArray();  
 for(int i=0;i<equipmentParts.length();i++ ){  
      JSONObject equipmentPart = equipmentParts.getJSONObject(i).getJSONObject("equipmentPart");  
      JSONObject allAttributes = equipmentPart.getJSONObject("allAttributes");  
      JSONObject part = new JSONObject();  
      // add more here  
 vars.put("jsonResponse", parts.toString());  
    An ordinary user have to really focus his/her attention to be able to follow it.  With Netgend javascript, it's as simple as the following:

 a = fromJson(data); 
 parts = [];
 for (i = 0; i < getSize(a); i++) {
      part.partId   = a[i].equipmentPart.allAttributes.equipmentPartId;
      part.partName = a[i].equipmentPart.allAttributes.compositePartName;
      push(parts, part); 
    Here the compound variable "a[i].equipmentPart.allAttributes.equipmentPartId" is based on the JSON message structure: at highest level, this JSON message is an array,  the each element in the array can be represented by a[i],  in each element, we have hierarchy like equipmentPart -- allAttributes --equipmentPartId.  Now you understand where the compound variable comes from.

   We believe parameterization/correlation should be made so simple that one doesn't need to be a guru on programming to do performance testing!   

Sunday, November 10, 2013

Performance testing on search form

    Searching is as important to an e-commerce web site as google is to the internet.  With a good, responsive search form, user can quickly find the item/product to purchase or book.

   So how should we do performance testing on search form?  simply sending HTTP POST or GET request with something like "search_block=examples" may not work, since there may be some hidden fields present in the form data like "search_block=examples&cpId=12001&_sessionId=100012". That's because the form may look like
 <form action="blabla">  
 <input type="text" name="search_block"/>  
 <input type="hidden" name="cpId" value="12001"/>  
 <input type="hidden" name="_sessionId" value="100012"/>  

   How to send search request and keep the hidden fields?   With netgend javascript, we can use the function "getFormField".  Here is an example.
 function VUSER() {  
      action(http, "");  
      a = getFormField(http.replyBody);  
      a.search_block.value = "examples";  
      http.POSTData = combineHttpParam(a);  
      action(http, "");  
   In the above, "getFormField(http.replyBody)" will parse the form in the http reply body and produce a compound variable with multiple fields, each field has a name and value (if present) from the form. We will assign a value to the "search_block" field and "combine" the fields in the compound variable "a" and create a string like

   Of course, to make it more effective, we can read a list of query phrases from a csv file. This is covered in previous blogs and will be left as an exercise for the reader.

   In addition to query string, sometimes we need to create a pair of random dates when doing search.  This happens, for example, when we reserve a hotel room or air plane tickets.   It's quite easy with netgend javascript.  Here is an example that shows a random start date is picked between now and 7 days later, an end date date that's 2 days later.

 t = time();  
 t += randNumber(0, 86400 *7);  
 a.startDate = toDate(t);  
 //variable "a" will contain "11/18/2013"
 a.endDate = toDate(t + 86400 * 2);  
 http.POSTData = combineHttpParam(a);
 //http action
    For web site that's particular about format of date, you can easily change it using the optional second parameter of "toDate" function.
 a.startDate = toDate(t, "%Y-%m-%d");  
 //will produce something like 2013-12-16  

    Last thing we cover in this blog is related to "suggestions" in search form. We frequently see some suggestions when we are typing queries.  How did the suggestions get to browser?  Turns out that behind the scene, there are ajax calls to server to fetch suggestions as user types the characters in the text box.

   How do we emulate clients talking to server for suggestions?  It's only a few lines on netgend's javascript.

 query = "computer";  
 for (i = 1; i < length(query); i ++) {  
      partial = substr(query, 0, i);  
      //variable "partial" will contain the first i characters of the query string
  Here we use the variable "partial" to hold progressively more characters in the query string and use it to do transaction with server.

   There are lots of interesting questions on performance testing on search forms, we will cover  them in future blogs.

Thursday, November 7, 2013

Performance comparison between Javascript, python and perl

    One of our projects has heavy processing on json string.  Python was the language of choice, but I was curious on how the other languages perform, so I did a simple comparison against  2 other languages (actually their interpreters): Javascript, Perl.

   We are not surprised that Javascript is the fastest because 1) JSON is a native data format in javascript  2)  we had heard V8 engine (which Node is based on) is the fastest interpreter of all scripting languages.   We know this doesn't mean we have to change our  choice of language to be javascript right away, but it does give some idea how much performance gain we may get.

    Disclaimer:  this is just one test and focus on one type of message processing,   so it's far from being comprehensive.  Don't want to start a war here :-)


  • CPU:   Intel(R) Core(TM) i3 CPU M 380  @ 2.53GHz
  • OS: Ubuntu 11.04 desktop 64bit
  • Node (for javascript):  v0.4.9
  • Python 2.7.2+,  json module version 2.0.9.
  • Perl 5.12.4,  JSON module version 2.53

Test benchmark

  • Take the following json string, convert it to a object (specific to that language/intepreter under test)
    {"addr": {"city": "austin", "zip": 12345}}
  • replace the "city" field of the object to be "houston"
  • encode the object into json string.
  • do the above 1 million times and measure the total time it takes.


LanguageTime taken
Javascript (Node)2.85s

Here are the code snippets for each of the three languages.


 import time  
 import json  
 import os  
 #purpose: test performance of python.  
 a = '{"addr": {"city": "austin", "zip": 12345}}'  
 ts = time.time()  
 for i in range(0,1000000):  
   x = json.loads(a)  
      x["addr"]["city"] = "houston"  
      y = json.dumps(x);  
 print time.time() - ts  


 //==========process a simple json string in nodejs, took 2.845seconds=============  
 a = '{"addr": {"city": "austin", "zip": 12345}}';  
 ts = new Date().getTime()  
 for (i=0; i<1000000; i++) {  
   b = JSON.parse(a); = "houston"  
 console.log(new Date().getTime() - ts)  


 #==========process a simple json string in perl, took 4.377seconds=============  
 use JSON;  
 use Time::HiRes qw( usleep ualarm gettimeofday tv_interval nanosleep  
            clock_gettime clock_getres clock_nanosleep clock  
            stat );  
 $json = JSON->new->allow_nonref;  
 $a = '{"addr": {"city": "austin", "zip": 12345}}';  
 $start = getTS();  
 for ($i=0; $i<1000000; $i++) {  
   $b = $json->decode ($a);  
      $b->{addr}->{city} = "houston";  
      $c = $json->encode($b);  
 print getTS() - $start,"\n";  
 sub getTS {  
      my ($seconds, $microseconds) = gettimeofday;  
      return $seconds + (0.0+ $microseconds)/1000000.0;  

Monday, November 4, 2013

Extracting fields from server responses

    In performance testing, we frequently have to process server responses.  One part of the processing  is extracting fields from the response messages and saving them to parameter(s) to be used later.
    In Loadrunner, the most common way of extraction is by using boundary method:  you provide the left-boundary of a field and a right-boundary.  It returns the value contained in-between. This could be a good choice when the response data is in free format, but if the response data has a structure, like a HTML form, Netgend scripting provides an easier and more intuitive way for data extraction.

   The first example here is based on an example from this blog (thanks for it!). We need to extract the value for "ID1" field from the following message:
 <input type="hidden" name="ID1" value="11111">  
 <input type="hidden" name="ID2" value="22222">  
 ... (more lines) ...  
    Of course, one can extract the value in this case as
 Left-boundary: ID1\" value=\"  
 Right-boundary: "  
    In netgend javascript script, we can do it as
 a = getFormField(str);  
 #now variable a.ID1.value will contain the value "11111".
    Much cleaner, right?

    In case you really want to use boundary method, netgend javascript supports it for feature compatibility :-)
 a = substring(str, "ID1\" value=\"", "\"");  
    Note that in netgend javascript, boundary can potentially be dynamically changed, while in the case of Loadrunner, it's static (fixed in test creation time).
 left  = ... ; #may be extracted from previous server response   
 right = ... ; #may be extracted from previous server response   
 a = substring(str, left, right);   

    The second example involves CAPTCHA. We as a human being,  can "see" the characters better than a machine/program.   However, there are some web sites that use a much weaker format:   In their registration forms, there is a question like  "what is sum of 14 + 9:  [      ]":
This may deter some casual hackers, but it will not deter sophisticated hackers from registering lots of accounts.

    Suppose we need to test this,  we can use the boundary method mentioned earlier, but it will be very tedious:  we will extract both numbers into their parameters, do the arithmetic operation on the two parameters and write it back to a parameter.

    In netgend javascript script, it's very trivial:
 contains(http.replyBody,  /what is sum of (\d+)\s+\+\s+(\d+)/);  
 answer = g1 + g2;  
 //now you can send POST data like "...userAnswer=${answer}"  
     Note that in the above, the two variables "g1", "g2" will contain the two numbers. We just add them up and assign the sum to the variable "answer".  In case you wonder,  the values in "g1", "g2" are strings,  can you add them as if they are numbers?   The answer is yes! If they are really numbers,  like "14" and "9" in our case, you can performance the arithmetic operation.

     Of course, we are not hackers :-)  We are the proud performance testing engineers!

    What if we need extract some values from a response that looks like xml? More precisely, suppose we have a response that looks like the following, how do we extract name of the first employee in the response?

       <name>John Smith</name>  
       <name>Jane Dole</name>  

    You may be tempted to use regular expression, it may work, but it may get you the wrong answer (for instance, get you the name of the second employee).   Here is a much simpler solution.
 a = fromXml(str);  
 //now a.employees.employee[0].name.value will contain the value "John Smith" 
    Here the compound variable "a.employees.employee[1].name.value" come from the xml structure, reader can quickly see the easy mapping here.

   Different test scenarios may require extraction of data differently. Based on the above examples, we can tell they are not going to be too hard for netgend javascript :-)

Saturday, November 2, 2013

JSON message processing in performance test

    We all know that JSON as a message format has become more popular,  databases such MongoDB, CouchDB support it, so do many web services.  So it's not a surprise that performance test tools need to deal with it.    However,  it's not easy to process JSON messages on many existing test platforms,  in some cases, users are desperate enough to use regular expressions!

    Here we are going to show how easy it is to process json messages in netgend javascript, even for user with modest coding background.  For those who have read previous blogs which touched on the subject of processing JSON messages,  this blog is a continuation and it will cover some more examples on how to use netgend javascript to process JSON messages in performance testing.

    Before diving into the examples,  the author would like to thank the sites where examples/questions were taken from. Readers of this blog are encouraged to read more at those sites and try to solve questions/challenges raised there.

Example 1

    This example is based on a stackoverflow question:   given a JSON string like the following, find the id's whose value is not null.
 {"id":"456","value":null,"description":"something else"},
 {"id":"789","value":"def","description":"something more"}]
    The solution is fairly simple.  In the following solution, we assume variable "str" contains the above JSON string.
 a = fromJson(str);  
 i = 0;  
 while (i < getSize(a)) {
      if(a[i].value != "") {  
      i ++; 
    Note that function "fromJson" parses the JSON string and assign to variable "a",   "a" will contain an array of json objects, each containing "id",  "value" and "description" fields.  The variable "a[id].value" means the "value" field of id'th item of the array  "a".   Finally, as a reminder, all the variables here are local to the VUser.

Example 2

    Based on another stackoverflow question (there have lots of questions, don't they?):  given a JSON string like the following, print all id's of the rows whose type is "B".
    The solution is very similar to example 1.
 #assume str has the content of above json string
 a = fromJson(str);   
 id = 0;  
 cnt = 0; 
 while (id < getSize(a)) {  
    if (a[id].type === "B") {   
      cnt ++;   
    id ++;   

Example 3

      Suppose we have the following JSON string as HTTP response,  we need to get value of certain fields.
   "response_time": 0.014376163482666016,  
   "applications": [  
       "api_key": "blted0e7982e1cf62a8",  
       "name": "gta",  
       "uid": "gta",  
       "account_name": "jack"  
       "api_key": "blt1423c40d23e4a423",  
       "name": "cellapp",  
       "uid": "cellapp",  
       "account_name": "max"  
    The JSON structure is a little more complex, so are the compound variables like "resp.applications[0].api_key".  However, it should be very easy to map the structure of the variable to the fields in JSON string.
 resp = fromJson(http.replyBody);  
 //now you can access fields in the json string.  
 //In the following expressions, "application" is an array of two item, first item has index 0  
 //The above will have value "blted0e7982e1cf62a8"  
 //the above will have value "cellapp"  

Example 4

     Again based on an example in stockoverflow.  The JSON string in this example is long and complex.  Here we need to find the names of two nodes after "CURRENT", in our case, we need to find  "36735928", "37851136". 
 "payload": {  
       "userName": {  
             "firstName": "Peter",  
              "lastName": "Pan"  
       "reservationLists": {  
               "CURRENT": {  
                    "36735928": {  
                           "startDate": "2013-10-03",  
                            "destination": "Balearen",  
                            "mainProductType": "Mietwagen",  
                           "allProductTypes": [  
                    "37851136": {  
                           "startDate": "2013-10-14",  
                           "destination": "Nevada, Las Vegas",  
                           "mainProductType": "Camper",  
                          "allProductTypes": [  
                 "PAST": {  
                     "12061210": {  
                           "startDate": "2012-05-05",  
                           "destination": "Loire / Nivernais",  
                            "mainProductType": "Boot",  
                            "allProductTypes": [  
    Despite the long JSON string, the solution is quite short and easy to understand.
 //assume variable "jsonstr"contains the complex json string in the question  
 a = fromJson(jsonstr);  
 b = keys(a.payload.reservationLists.CURRENT);  
 //now b[0] will contain first node name ("36735928" in this case)   
 //and b[1] will contain second node name ("37851136" in this case)  

Example 5.

    Finally an example JSON from Alex Ersenie's blog,  we need to print all the child categoryId's.
       "name":"Root Catalog",  
          "children":[ ]  
          "name":"Might Like",  
          "children":[ ]  
          "name":"Recent Additions",  
          "children":[       ]  
          "name":"Hero Products",  
          "children":[ ]  

     The solution, as usual, is fairly simple.
 a = fromJson(str);  
 b = a.categoryTree.categories[0].children;  
 for (id = 0; id < getSize(b); id ++) { 

    Now that we have shown processing complex JSON structure can be so simple,  I am sure you may want to apply this technique to more challenging issues encountered in your performance testing!