Some notes I wrote sometime ago while working on Performance testing using JMeter for a client:
Definition of Terms
Performance : The application’s ability to meet a certain goal, such as a desired response time & latency, level of throughput, concurrency, etc.
Capacity: The total load the application can handle – Quantity of Data, Number of Users, User Activity, Size of related Datasets.
Scalability: The application’s ability to maintain performance as it becomes larger by some metric (more servers, for example). Performance in broad sense is capacity and scalability together.
Availability: The percentage of time the application is able to respond to requests. This is usually measured in “nines”; for example, “five nines” means the application is available 99.999% of the time, which translates to roughly five minutes of downtime per year.
Fault tolerance : The application and overall system’s ability to handle failures gracefully. Even a system designed for high availability can fail. When it does, a fault-tolerant application can continue to provide as much functionality as possible, rather than being entirely unavailable.
We can use Performance Testing for measuring Performance (duh!), Capacity, and Scalability. The Stress test measures the Availability and the Fault Tolerance of the application.
Performance Testing should occur after the functional testing is mostly complete and the software has become quite stable. Functional defects may be revealed during the performance testing, but that is not considered as the objective for Performance Testing. The objective of the performance testing is to validate the software “Speed” against the business need for the “speed” as documented in the software requirements. Software speed is generally defined as some combination of response time and workload during peak load times.
To make the application reveal its true production speed, the tests must be executed in a testing environment that approximates the intended production environment as closely as possible. In essence, the performance test provides data that allows us to meet the requirements in Scalability of the application. Performance testing is achieved by a series of tests that introduces increasingly more workload of increasingly more complex business transaction mixes.
The objective of the stress testing is to measure the fault tolerance and recoverability of the software application. The stress testing provides valuable insights in achieving the availability requirements. The stress test needs to be done on various components that are depicted, preferably in a, deployment diagram.
The test plan for stress testing includes shutting down the various components of the application deployment such as web server, load balancer, nodes, database, etc. while the system is at peak workload.
Recoverability and Fault Tolerance
To test whether the recoverability of the system, the application must be under peak workload and we reboot/restart one of the component and observe the system’s behavior. For example, while the system is under peak load using the JMeter script, the database can be shutdown forcefully, and then need to be restarted. We need to check how the system behaves in such case. Ideally no data should be lost and the user should be notified and the application should try to restore its state gracefully. Same can be done by restarting one of the tomcat nodes in the cluster as well as the load balancer.