Test Application Monitoring Software

    Today’s blog is addressed to the infrastructure folks who have the job of making sure all of our applications and monitoring systems are functioning effectively. The NetGend load testing tool is awesome for testing applications at scale, but we will discuss another use – how do we generate synthetic yet realistic load to test monitoring, filtering and scanning systems that sit in our infrastructure?

    In today's internet-based economy, it's essential to ensure that servers are running efficiently and as-expected.  It is important to periodically check the health of the servers.  Application monitoring software is a key tool in validating one’s performance and business-readiness.  Another example of non-traditional load testing is with filters and proxies, which also have a role in ensuring the protection and configuration and enforcement of various policies – application blocking, logging, etc… (We will talk about this in more detail in an upcoming blog.)

    How do we check the checkers? In the case of application monitoring software, just like any other software, it is important to load test it during development and after functional testing phase.   A natural question is, how do we emulate a large number of servers while injecting a sufficient/configurable number of abnormal conditions?

   It turns out that NetGend, in addition to being a great performance testing platform,  can also emulate the servers-under-monitoring.  More specifically,  it can emulate the agents running on the servers and emulating the sending of the server statistics such as CPU, memory,  etc to the application monitoring software.

   Here is an example of NetGend script that can be used to emulate thousands of agents.   From the standpoint of TCP/IP, each of the agents is simply a TCP client, sending data to the TCP server (the monitoring system) repeatedly.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

function VUSER () {     connect("tcp", "monitorserver.example.com", 3000);     isGoodServer = randNumber(0, 50);     cpuLimit     = 100;     memoryLimit  = 100;     if (isGoodServer) {         cpuLimit    = 80;         memoryLimit = 90;     }     cpu    = randNumber(1, cpuLimit);     memory = randNumber(20, memoryLimit);     while (1) {         cpu    = randWalk(cpu, 0, cpuLimit, 10);         memory = randWalk(memory, 0, memoryLimit, 10);         send("${cpu},${memory}");         sleep(1000);     } }

Here is a brief explanation of the above code:

• Line 2,  sets up a TCP connection to the server.
• Line 3,  decides whether the instance is going to be a "bad"/"unhealthy" server, in which case, CPU or memory usage may shoot up to 100%
• Lines 4-9,  sets the upper limit. For healthy ones, we set the upper limit to be 80% on CPU and 90% on memory.  
• Lines 10-11, sets the starting value for cpu and memory.
• Lines 13-16, updates the cpu and memory percentage and sends the information to the server.

   This script is so concise and efficient thanks to our function randWalk(), which gets its name from the phrase "random walk".  This function takes 4 parameters

<currentValue>, <lowerLimit>, <upperLimit>, <step>.  

The returned value can go either up or down from <currentValue> by up to <step> amount.

   This brief example reports only the statistics on CPU and Memory usage, but it can be easily extended to report other statistics as well, such as Disk usage, IO, bandwidth usage etc.  Also, some monitoring software may expect the agents to report the statistics in the form of HTTP/HTTPs requests or other protocols over TCP or SSL/TLS.  Rest assured,  NetGend supports all of the above transport mechanisms.

   At NetGend, we are proud of the flexibility and scalability of our platform, we are especially happy that it can be used to test Application monitoring - a sister software to application performance testing.  Also, if you are interested in how this can be used to generate load for proxies or Big Data analysis, please don’t hesitate to  please drop us a line  info@netgend.com.