The SQA2 Blog: IT News, Software Quality
On July 19, 2024, the U.S. witnessed a significant disruption due to a major outage of CrowdStrike’s cybersecurity software. The effects were far-reaching—air travel was heavily delayed, hospitals faced operational challenges, and businesses struggled with essential functions. The root cause of this outage has been traced back to a bug introduced in a software update, which caused widespread crashes across millions of Windows devices. The corporate software impact was severe, affecting one in four Fortune 500 companies and resulting in losses exceeding $5 billion.
Leadership’s Decision and Its Implications
Initial reports suggest that the outage resulted from CrowdStrike leadership bypassing crucial quality checks. This decision, though a significant misstep, is not the sole fault of the leadership. Instead, it appears to be a symptom of a deeper issue within the company’s quality assurance (QA) processes.
The question arises: why did leadership skip these essential checks?
The Root Cause: QA’s Diminished Value
“In analyzing this situation, we believe leadership’s decision to bypass QA checks likely stemmed from their perception that the QA function did not justify its cost,” stated Tim Harrison, President at SQA². “This perception is crucial to understanding why CrowdStrike made the controversial decision to lay off approximately 200 staff members, including many from the QA team, last year.
“In our experience as QA experts, we’ve observed that when QA teams are not able to demonstrate clear value, they become vulnerable to cutbacks. For CrowdStrike, this was likely the case that the QA team struggled to adapt to the fast-evolving demands of their software development lifecycle (SDLC). The inability to provide tangible value made the QA team an easy target for downsizing.”
What the QA Team Could Have Done Differently
To prevent such situations and demonstrate their value, the QA team could have employed several strategic approaches:
Shift from Gatekeepers to Team Integrators
“QA must understand that they are not gatekeepers of production. Rather, QA should evolve from being seen as a bottleneck to becoming a core part of the development team working to mitigate risk and prevent defects. This means integrating QA early in the design and requirements phases rather than as an afterthought,” said Harrison. By collaborating closely with development teams from the start, QA can help design test cases in parallel with coding, ensuring issues are identified and addressed sooner.
Embrace Collaboration
Building strong collaborative relationships with development teams is essential. QA should work side by side with developers to ensure quality checks are seamless and effective. “This collaboration should go beyond mere communication; it should involve joint problem-solving and continuous feedback,” said Harrison.
Mitigate Risks Proactively
Effective QA involves identifying and addressing potential risks before they escalate into major issues. This requires testing early and frequently, enabling rapid detection of problems. Regular updates and open lines of communication ensure that all team members are aware of risks and can take appropriate action.
Adopt Flexible QA Cycles
QA processes need to be adaptable to changing project requirements. This means avoiding rigid testing cycles and instead employing flexible methods to respond to varying demands. Automated testing suites can be instrumental here, allowing for thorough testing without the need for extensive manual intervention.
“Build methods that allow more thorough testing quickly and continuously such as building a comprehensive automation suite against a regression test plan. This allows testing to occur quickly, catching major issues,” says Harrison.
Promote Comprehensive Testing
While speed is important, thorough testing is crucial for ensuring software reliability. QA teams should advocate for appropriate testing cycles that balance the need for speed with the necessity of quality. Automated testing can help maintain thoroughness even under tight deadlines.
“Comprehensive testing ensures higher software quality and reliability, building trust with stakeholders. Advocating for appropriate testing cycles ensures that the team can identify issues while not being unnecessarily rushed. Again, implementing an automated test suite enables more thorough testing when time is of the essence,” stated Harrison.
Implement Shift-Left Practices
Involving QA in the early stages of the development process can significantly reduce the risk of defects. QA should be part of design and requirements discussions, offering feedback that helps shape testable features and precise requirements.
“You can do this by including QA in design meetings, allowing for feedback and questions about the design and testability of the feature. QA should be writing test cases early and reviewing them with product and development. These processes, along with many more, ensure that you shift the focus as far left as you can to minimize costs,” said Harrison.
Focus on Bug Prevention
Preventing bugs before they occur is more cost-effective than merely detecting them later. By implementing the practices mentioned above and maintaining a proactive approach, QA can help create more stable and reliable software, reducing the overall defect rate. “Prevention is the name of the game and everything QA can do to prevent bugs saves on costs more than just detecting bugs,” said Harrison.
The Path Forward
CrowdStrike’s recent outage highlights the critical importance of an agile, integrated QA process. Companies can avoid similar pitfalls by shifting QA from a gatekeeping role to an integral part of the development team, fostering collaboration, and adopting flexible and proactive practices. Ensuring that QA provides tangible, long-term value is key to maintaining its role within the SDLC and preventing costly disruptions in the future.
For organizations seeking to enhance their QA processes and safeguard against such failures, engaging with experienced QA partners like SQA² can offer invaluable support. At SQA², we’re committed to providing high-quality, cost-effective QA solutions that drive innovation and reliability in every project.
Contact SQA² and find out how we can ensure seamless acquisition and integration of the latest technology for your company.
