When running performance tests, it is important not to run just a single test in isolation, but blend a number of realistic scenarios.
For example, consider a website which allows users to search ecommerce products and to refine their results. If we only load tested the search functionality, then we would simply be testing how quickly the search index can be read under heavy load.
For a realistic test, we also need to see what happens when product content is updated or published, as this will cause the search index to be rebuilt. This ensures that different elements of the underlying solution are tested together, and not just in isolation.
Defining blended scenarios
Imagine a website providing a job search function. It might be reasonable that the site has 20,000 registered users who are actively seeking a new role, and that around 1 person in 5 accesses the site each morning within a 2 hour window (e.g. following receipt of an overnight email showing them potential new roles).
In this case, it would be reasonable to assume that the peak load is defined as 10% of registered users accessing the site within a one hour window – so in this case, the web application needs to be able to support 2,000 visitors per hour.
Given such a case, we would suggest testing the following scenarios
- Scenario 1: 80% of users login and view their overnight job matches (1,600 candidates per hour)
- Scenario 2: 20% of users to login and search (400 candidates per hour)
- Scenario 3: Several staff members login and add or update job vacancies.
What each scenario tests
Each scenario is testing a different function of the web application.
Scenario 1 tests the raw database performance. Applications that provide job shortlisting often rely on extremely large database tables to track view history, in order for users to only get notified about the positions they haven’t previously seen. These databases can store often millions of rows of data.
Scenario 2 tests how quickly a search index can be read under load, as well as any data it needs to retrieve from the database (such as job summary or description). In order for a realistic test, we also need to see what happens when new jobs are published, as this will cause the search index to be rebuilt.
Scenario 3 allows staff members to login and modify data. This could affect potentially affect both the shortlist database as new positions are added (especially when shortlists are built automatically) as well as causing the search index to be rebuilt. We would include this scenario within the load tests to make sure that this is a process which happens in the background and has with no noticeable performance impact on users who are accessing the site.