Blog by Dennis Elenburg, Customer Solutions Engineer, MKS a PTC Company
A problem that plagues all software development organizations is never having the time to test everything you want to test. Shipping a quality product or bug free application means making hard choices when allocating limited testing resources. Testing faster and more efficiently is always a good idea, but are you testing the right things? Can your testing organization easily access the product requirements and create defects from failed tests, both automated and manual? Do your testers write bug reports that help your developers quickly fix the problem?
You Get What You Measure
Quantity of tests executed doesn’t necessarily correlate to product quality. This is especially true when automation is involved. If the metric used to assess quality is “number of tests performed,” a tester who automates 1,000 test cases may be rewarded and respected more than the lowly manual tester who is only completing 10 tests in the same period of time. However, if those 1,000 tests pass without uncovering any bugs this may be a false quality indicator. If the manual tester discovers two bugs and an unmet requirement in his 10 manual tests, who contributed more to the quality process?
Test automation is utterly incapable of interpreting the meaning of test failures. Automation speeds up the execution process and comparison of results, but a knowledgeable tester must still determine if a failure was due to the test script or if a defect truly exists. Then, a well written bug report will not only note the failure but indicate how the system behavior doesn’t conform to the requirements. This improves developer productivity in fixing the defect. Efficient knowledge work connects testing, requirements, defects, and even the code.
High value software testing and defect analysis is performed by testers who understand product requirements, have the skills to map tests to requirements, and who can write bug reports that help developers fix defects quickly. This is collaborative knowledge work. You can boost the effectiveness of your knowledge workers by providing them with tools to help them connect the dots. More automation may speed things up, but to really enhance product quality, total visibility and end-to-end connectivity across the software development lifecycle is imperative.
Highly Effective Testing is Collaborative
As a former QA manager who struggled under dictates from senior management to “automate, automate, automate,” I can empathize with the desire to run more tests. Test automation vendors make lots of promises, and automation can be great, but it is easy to lose sight of the importance of connectivity when focused on a major automation initiative. How do you translate your test results back into useful knowledge for improving product quality?
A quality metric of “number of tests passed” is not enough, particularly if you’re automating hundreds or even thousands of tests. You need metrics on all aspects of your software development process and how they relate to each other: test coverage against requirements, defects per test session, and even defects in requirements are just a few examples. A whole host of metrics are possible when you have total visibility and connectivity across your entire software development process.
If everyone can see the complete lifecycle of the requirements from inception to final delivery, including any defects along the way, then quality becomes a shared responsibility across the enterprise. Silos come down and collaboration improves. Quality depends on everyone doing their part, and that starts with equipping your knowledge workers with access to the information they need to do their jobs. Do you have total visibility into your entire software development organization? If not, let us show you how…
Please feel free to leave comments or answer any of the questions asked throughout this blog.