In our previous article, “How to choose the right KPIs for your business goals?” we emphasized the importance of measuring quality from a holistic perspective and connecting it to the business goals. We introduced Key Performance Indicators (KPIs) to measure quality, eliminate wasteful activities, and evaluate the success of individual metrics.
This article will explore the correlations between methods and metrics, illustrating why they are crucial for successfully implementing KPIs. We will discuss three main methods: Risk-Based Testing, Agile Development, and Test Automation.
Method 1: Risk-based testing
Risk-based testing is a holistic approach that connects testing across all processes in the SDLC (Software Development Life Cycle) with business risks. It maintains a constant testing effort of complex systems by focusing on high-risk areas first, preventing serious errors and improving quality.
To implement a risk-based testing methodology, the following three steps are necessary:
- Identification of product risks.
- Risk analysis is based on the probability of occurrence and extent of damage to create risk prioritization.
- Planning, design, and execution of all quality activities according to prioritization, including continuous re-evaluation of risks.
The last step, in particular, is crucial so that we do not have excessive quality management which has only one task: documenting and tracking risks, and that we avoid having a long list of defined risks that no longer play a role in daily implementation.
Key metrics for risk-based testing include:
- Percentage of rejected failures,
- Rate of critical failures (in production),
- Risk-based (regression) test coverage.
Method 2: Agile development
Agile development goes beyond speed in software development. It entails integrating the QA team from the start of requirements gathering and fostering cooperation between development and testing. It also relies on a competent team, open communication channels, and a supportive environment. Agile development incorporates a risk-based view of testing and software development. It’s ideal for handling complex systems with a minimum viable product and incremental approach, facilitating the integration of new technological challenges and shifting the overall effort left in line with Lean principles.
Key aspects of introducing meaningful KPIs in an Agile framework:
- Iterative and incremental approach.
- Efficient communication.
- Rapid adaptability and feedback.
Metrics to measure progress:
- Deployment frequency per sprint.
- Deviation from budget planning, time deviation from the project plan, and deviation from test effort estimation.
- Deviation from average development cycle time.
Method 3: Test Automation
Applying lean principles to test automation methods and processes can lead to overall quality gains through optimization and focus. Many companies are structuring their tests as a testing diamond.
The focus is to create a regression test set at the integration level, which considerably simplifies test maintenance and verifies interaction with other modules, since most problems and risks are found at the integration level. In our product development for the QACube platform we also follow this principle. We reduced wasteful maintenance at the unit test level and errors that are found during the system integration test while improving overall quality.
In this case, test automation serves as a long-term solution to increase efficiency in testing and create additional capacity. Key metrics for test automation include:
- Time saved through automation,
- Frequency of regression testing per sprint.
Conclusion
By taking a holistic approach and measuring progress at various levels, Sixsentix experts seen improvements that result from synergies across teams in the software company we worked with.
Linking the business goals to the process level provided transparency, enabling faster, more informed decision-making and strengthening team understanding. Ongoing risk measurement allowed for the prioritization of limited resources to the correct issues at the right time, resulting in the project being delivered without budget variances.
The adapted test automation strategy stabilized the regression test set, enabling the team to catch more bugs earlier and reducing maintenance efforts. Overall, we saw a reduction in critical production defects throughout the transformation.
The first major accomplishment of the transformation project was the discovery of a demand trend for a sub-product through analysis in the specialist area. This extension was successfully introduced in a relatively short time with high customer satisfaction, and a level of responsiveness that would have been unthinkable.
It's worth mentioning that we encountered obstacles in implementing the KPIs, which we will share in the next article. So stay tuned!