The SQA2 Blog: BBT - Behavioral Based Testing
Requirements Test Coverage
Requirements test coverage is a highly critical piece of the development and testing phase as it allows the stakeholders to gain confidence in the application. The higher the coverage and success of those tests, the more confident the team, and more importantly, the stakeholders are in the application. So why would you trust such an important piece of the puzzle to a gut check? Quantifiable requirements test coverage provides the level of confidence that no gut check can.
Let’s put ourselves in the shoes of the QA professional responsible for testing. So, you understand the requirements, you know the application, and the scope and impact have been called out – no problem. You got this, right? Keep in mind you need to cover the new requirements, the existing features, integration with other upstream or downstream applications, edge cases, negative path tests, positive path tests, and on and on and on…
Okay, take a breath. This is a bigger effort than we thought. Don’t let it freak you out though. Let’s think about this for a minute. How do you keep track of all of this? How do you know you covered everything? A gut feeling? A really long Excel spreadsheet? How do you prove it? There are many ways we go about developing requirement test coverage, but are they the right way? You have to be able to prove to business that everything works, and everything was tested.
Just test the critical path…
Some schools of thought say that you can’t test everything. This is a highly adopted approach when companies just don’t have time to test everything before their fancy new tool needs to hit the market. “Just test the critical path now and we’ll come back to test the rest later.” Sound familiar? We shouldn’t settle for that.
Not wanting to give up on this new feature, we get under way by writing some test cases. Make sure it does this. Make sure it does that. Make sure it can’t do this and that. How do we know we covered all of the requirements? For years, I created test cases until every requirement was accounted for with a test case, or two. That covers it all, right? I accounted for each requirement in the test cases after all. Well it might, if you’re not too worried about what crazy things the user will try. It might if the system works perfectly all the time. But we don’t live in a perfect world, do we? We can’t expect those to be reality.
Getting the proper ROI
Then there’s the issue of return on investment (ROI). We can continue to test for weeks and weeks, but slowly, the rate of bug discovery will taper off. However, statistically, there are still bugs. Studies show that even with significant effort, companies can discover 85% of defects within a piece of software. Only until you adopt practices often found in organizations at Comparability Maturity Model (CMM) Level 5 can you hope to push that to 95%. Going beyond that is almost unheard of.
Caper Jones wrote in 2008, “In examining data from about 13,000 software projects over a period of 40 years, only two projects had zero defect reports in the first year after release.” Even with thorough testing efforts, it requires significant resources to reveal a bug. Keep in mind that quantifiable requirements test coverage is not an end-all solution that will get you that 95-100% defect detection. Many times, these defects are not related to requirements or code at all. There is a lot of work that goes into that outside of requirements test coverage, and even the QA organization. What it will provide is an increased coverage leading to the discovery of more defects before they go to production. Combining this with a mature QA, Development, and product owner organizations, you can increase your quality and detection of bugs.
Gut check…or something else?
So how do we get away from the gut check? Spending hours upon hours writing test cases only to discover negative cases not previously considered can be an endless cycle. Testing cycle after testing cycle leads to diminishing returns and spinning wheels. In the end, does all of this account for 100% of the requirements? This is the crux of quantifiable requirements test coverage: Have we adequately tested everything in the requirements and how do you go about proving it?
Why not spend a fraction of the time developing a complete set of test cases using Behavior Based Testing (BBT). BBT gives you the quantifiable coverage of requirements by using simple Boolean logic with truth tables to generate a comprehensive list of permutations of test cases that ensure you have 100% requirements coverage of all positive and negative test cases. With BBT you can speak with confidence to your stakeholder, you identified, verified, and proved all scenarios for the requirements.
1. Jones, Capers. “Measuring Defect Potentials and Defect Removal Efficiency©.” (2008).
Image courtesy of cooldesign at FreeDigitalPhotos.net