What makes performance testing so difficult?

 
    It's interesting to study questions asked in various forums on performance testing.  Most of time the questions are technical like "how to extract certain fields in a JSON message?".  These seemingly simple questions were given non-trivial (and sometimes incorrect) solutions using complex regular expressions.  This must have frightened some users into thinking their skills are seriously insufficient when they ask the general questions like "how would I improve my skills in performance testing?".  It makes me wonder why the performance testing platforms are not made easier.

    Many test platforms are more for developers or someone with considerable amount of programming experience.     To effectively use them,  one has to be able to write non-trivial code and be able to do debugging.  As an example,  let's look at an issue with loadrunner, the leading performance testing tool.

    Since string replacement function is not provided in loadrunner by default,  in this blog Kim gave a good workaround on string replacement.  It can be used to replace all the "+"s in the following string to be "%20":

 This+is+a+string  

    While it's impressive to see this nice implementation,  I can't help wondering why it is so complex and how many users are as advanced as Kim.      Logically speaking, replacing parts of a string is pretty simple.   Turns out the difficulties come from the concept of  "parameter" in performance testing.   In performance testing, we need to emulate many virtual users (VUser).  Each VUser will have its own variables, states etc.   Those variables or states are called "parameters" in loadrunner's jargon.   In loadrunner, when you want to perform a operation on a variable,  you have to "pull" its value to code space (a variable in C), perform the operation(s) on the variable in C and then write it back to the "parameter".

      On the netgend platform,  we don't believe it's necessary to require the users to have more than the basic programming skills.  We designed our test platform so that all per-VUser variables can be easily accessed.     As a result,  many functions such as string replacement function are very easy to implement  and are included in the test platform by default.  No need to be creative and hack up an extension or a workaround.

     Here is the syntax for "replace" function:

 replace(<sourceString>, <from-string>, <to-string> [,"all"])  

     The following a simple example on how to use it:

      str = "this is John, John Dole";  
      replace(str, "John", Joe, "all");  
      //str will be "this is Joe, Joe Dole"  

     Note that the last parameter "all" means we will replace all the occurrences of "John".   You can leave out the fourth parameter and it will only replace the first occurrence.

     What's more,  the from-string or to-string themselves can be a variable.

      str = q|this is John, John Dole|;  
      newName = "Mary";  
      replace(str, "John", newName, "all");  
      println(str);  

      This can potentially be used to read new names from a csv file and do the substitution dynamically. In Loadrunner, even with the nice implementation of string replacement function mentioned above,  one still needs to do more work if either "from-string" or "to-string" are parameters instead of hard coded strings.

      There are lots of areas where we can use our creativity,  I want to use it where simple things are kept simple. What about you?