In Software Development, What Should QA Test?

The SQA² Blog: Advice Center

User interfaces…databases…APIs…data warehouses…and the list goes on. Your software has so many layers and moving parts that you might wonder which aspects of your software your Quality Assurance (QA) team needs to test in order to ensure the end product works as desired. The answer is all of it—and more. The key to success is to start with the high-level view and then drill down into the details.

Everything starts with the business requirements

Does the following scenario sound familiar? The dev team creates an application that works flawlessly. QA tests everything, and everything works. But when the app is demonstrated for the key stakeholders, instead of “wow, that’s fabulous,” the response is, “that’s not what those functions are supposed to do.” What the dev team thought was a huge success was actually a depressing failure.

Ensuring that products work on a technical level is just one aspect of QA’s responsibilities. More importantly, QA exists to ensure that the customer receives a high-quality product that meets the business needs that the application was developed to address. In other words, QA tests to confirm that what the business wants is what is actually being delivered.

Unfortunately, misunderstandings often get in the way of making this happen. In fact, what we’ve seen is that the number one root cause of bugs that get out into production is communication error. In other words, the dev team misunderstands the business requirements. Business requests a particular feature or functionality. They want “X.” But what the dev team thinks they’re saying is that they want “Y,” and the app is developed to deliver “Y.”

This is why QA needs to understand exactly what the business requirements are, why each item was included as a requirement, and what the desired functionalities and end results are. Who is the customer? How and why will they use this application? At the high level, these are the things that QA is testing for.

What should QA test?

Once QA has a crystal-clear understanding of the business requirements, here is what they should test…

• Test the product itself – Does the application work as intended based on the business requirements? To answer this question, QA should test all layers of the product, including:
  
  • Technical components of the product – Does the user interface work properly? Are the API calls and requests between the client and service correct and working properly? Is the data being processed and stored correctly?
    Test each item separately, and then do integration testing to ensure the various components work together.
    
  • Features and functionality – Test each feature and functionality to ensure it works as Business intended it to work. Be sure to also look at regression issues, checking to see that with each new release the previous/existing features continue to work as expected.
    
  • Business intelligence (if included) – Test all of the components that allow the business to run reports and pull information about their environment, including the data warehouse.

• Test the product’s performance – Test all aspects of the product’s performance against the associated service level agreements (SLAs).For example, if the product is a website, is the site speed in compliance with the SLA? If it’s an API, is the response time for the various endpoints within the SLA? If you were to have a high number of users at once, would it crash the system? And so forth.

• Test the deployment – Does the product work in production? Does it pass a “smoke test” or “sanity test,” meaning a test to make sure that things don’t “go up in smoke” as soon as you start using the product in production? After you deploy into production, QA should run an end-to-end test to be sure everything integrates well and there are no glaring issues, such as “it’s now impossible to log in.”

• Test the development processes – Don’t fall into the trap of thinking that “this is how we’ve always done things, so our development process is fine.” It might be flawed. Remember, the number one root cause of bugs that get into production is communication issues. A good process will have checks in place to avoid this problem, because it is far better to prevent issues up front than to find and correct them later.If your development process itself is not working as well as possible, QA needs to identify the problem areas and then test any changes that are made. Does the revised process go more smoothly, with better communication and fewer issues?

• Do ad hoc testing – Even if everyone thoroughly understands the business requirements and all of the business logic is 100% correct, things can still go wrong. After all of the critical business requirement function and performance issues have been tested, it’s a good idea for QA to also do some “ad hoc” testing.What happens if users don’t behave the way you expect them to? For example, what happens if you double click this button? If you navigate to here, then hit the back button and then hit the forward button, what happens? Quite often testing these type of things turns up more bugs.

Why should QA test so many different things?

At SQA² we advocate for testing everything. Every aspect of your efforts should have some level of QA involvement. Without a robust QA program to catch the bugs and ensure that what the Business wants is what’s being delivered, you’re at risk for a whole host of problems. These include…

• Wasted time and money – You don’t want your dev team to put a great deal of effort into developing something that Business didn’t actually want. This is both very expensive and extremely demoralizing for everyone involved.

• Poor decision making – If your business intelligence systems are delivering incorrect information, your organization will be making important business decisions based on bad data.
  
• Lower revenue and profits – If you’re developing an external application, it won’t take long for word to get out that your product is buggy and should be avoided. Of course, a buggy product will not make your Customer Service Department happy, either.
  
• Decreased effectiveness of your internal organization – If you’re developing an internal application, and that application does not work well or does not actually meet Business requirements, employees will waste time trying to get around the issues.
  
• Loss of trust/confidence – If things are regularly delivered incorrectly, the Business team in charge of the project will lose confidence in the dev team’s ability to deliver on what they want. Once that trust and confidence is lost it can be next to impossible to get anything done.
  

Conclusion

Software is usually created to solve a problem of some sort. If the product fails to solve this problem, either because of bugs in the technology or misunderstandings regarding the desired end results, then the entire effort was for naught.

QA exists to ensure the quality of the product and deliverables for the customer. This is why we say that QA should essentially test everything. Robust QA testing organizations and processes ensure that the product works on the technical level and delivers the functionalities and capabilities that meet business needs.