Continuous testing
Continuous testing was originally proposed as a way of reducing waiting time for feedback to developers by introducing development environment-triggered tests as well as more traditional developer/tester-triggered tests.
Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.
For Continuous testing, the scope of testing extends from validating bottom-up requirements or user stories to assessing the system requirements associated with overarching business goals.
Adoption drivers
In the 2010s, software has become a key business differentiator. As a result, organizations now expect software development teams to deliver more, and more innovative, software within shorter delivery cycles. To meet these demands, teams have turned to lean approaches, such as Agile, DevOps, and Continuous Delivery, to try to speed up the systems development life cycle. After accelerating other aspects of the delivery pipeline, teams typically find that their testing process is preventing them from achieving the expected benefits of their SDLC acceleration initiative. Testing and the overall quality process remain problematic for several key reasons.- Traditional testing processes are too slow. Iteration length has changed from months to weeks or days with the rising popularity of Agile, DevOps, and Continuous Delivery. Traditional methods of testing, which rely heavily on manual testing and automated GUI tests that require frequent updating, cannot keep pace. At this point, organizations tend to recognize the need to extend their test automation efforts.
- Even after more automation is added to the existing test process, managers still lack adequate insight into the level of risk associated with an application at any given point in time. Understanding these risks is critical for making the rapid go/no go decisions involved in Continuous Delivery processes. If tests are developed without an understanding of what the business considers to be an acceptable level of risk, it is possible to have a release candidate that passes all the available tests, but which the business leaders would not consider to be ready for release. For the test results to accurately indicate whether each release candidate meets business expectations, the approach to designing tests must be based on the business's tolerance for risks related to security, performance, reliability, and compliance. In addition to having unit tests that check code at a very granular bottom-up level, there is a need for a broader suite of tests to provide a top-down assessment of the release candidate's business risk.
- Even if testing is automated and tests effectively measure the level of business risk, teams without a coordinated end-to-end quality process tend to have trouble satisfying the business expectations within today's compressed delivery cycles. Trying to remove risks at the end of each iteration has been shown to be significantly slower and more resource-intensive than building quality into the product through defect prevention strategies such as development testing.
Goals and benefits
The goal of continuous testing is to provide fast and continuous feedback regarding the level of business risk in the latest build or release candidate. This information can then be used to determine if the software is ready to progress through the delivery pipeline at any given time.Since testing begins early and is executed continuously, application risks are exposed soon after they are introduced. Development teams can then prevent those problems from progressing to the next stage of the SDLC. This reduces the time and effort that need to be spent finding and fixing defects. As a result, it is possible to increase the speed and frequency at which quality software is delivered, as well as decrease technical debt.
Moreover, when software quality efforts and testing are aligned with business expectations, test execution produces a prioritized list of actionable tasks. This helps teams focus their efforts on the quality tasks that will have the greatest impact, based on their organization's goals and priorities.
Additionally, when teams are continuously executing a broad set of continuous tests throughout the SDLC, they amass metrics regarding the quality of the process as well as the state of the software. The resulting metrics can be used to re-examine and optimize the process itself, including the effectiveness of those tests. This information can be used to establish a feedback loop that helps teams incrementally improve the process. Frequent measurement, tight feedback loops, and continuous improvement are key principles of DevOps.
Scope of testing
Continuous testing includes the validation of both functional requirements and non-functional requirements.For testing functional requirements, Continuous Testing often involves unit tests, API testing, integration testing, and system testing. For testing non-functional requirements, it involves practices such as static code analysis, security testing, performance testing, etc. Tests should be designed to provide the earliest possible detection of the risks that are most critical for the business or organization that is releasing the software.
Teams often find that in order to ensure that test suite can run continuously and effectively assesses the level of risk, it's necessary to shift focus from GUI testing to API testing because 1) APIs GUI tests require considerable rework to keep pace with the frequent changes typical of accelerated release processes; tests at the API layer are less brittle and easier to maintain.
Tests are executed during or alongside continuous integration—at least daily. For teams practicing continuous delivery, tests are commonly executed many times a day, every time that the application is updated in to the version control system.
Ideally, all tests are executed across all non-production test environments. To ensure accuracy and consistency, testing should be performed in the most complete, production-like environment possible. Strategies for increasing test environment stability include virtualization software service virtualization, and test data management.
Common practices
- Testing should be a collaboration of Development, QA, and Operations—aligned with the priorities of the line of business—within a coordinated, end-to-end quality process.
- Tests should be logically-componentized, incremental, and repeatable; results must be deterministic and meaningful.
- All tests need to be run at some point in the build pipeline, but not all tests need be run all the time.
- Eliminate test data and environment constraints so that tests can run constantly and consistently in production-like environments.
- To minimize false positives, minimize test maintenance, and more effectively validate use cases across modern systems with multitier architectures, teams should emphasize API testing over GUI testing.
Challenges/roadblocks
One reason teams avoid continuous testing is that their infrastructure is not scalable enough to continuously execute the test suite. This problem can be addressed by focusing the tests on the business's priorities, splitting the test base, and parallelizing the testing with application release automation tools.
Continuous testing vs automated testing
The goal of Continuous Testing is to apply "extreme automation" to stable, production-like test environments. Automation is essential for Continuous Testing. But automated testing is not the same as Continuous Testing.Automated testing involves automated, CI-driven execution of whatever set of tests the team has accumulated. Moving from automated testing to continuous testing involves executing a set of tests that is specifically designed to assess the business risks associated with a release candidate, and to regularly execute these tests in the context of stable, production-like test environments. Some differences between automated and continuous testing:
- With automated testing, a test failure may indicate anything from a critical issue to a violation of a trivial naming standard. With continuous testing, a test failure always indicates a critical business risk.
- With continuous testing, a test failure is addressed via a clear workflow for prioritizing defects vs. business risks and addressing the most critical ones first.
- With continuous testing, each time a risk is identified, there is a process for exposing all similar defects that might already have been introduced, as well as preventing this same problem from recurring in the future.
Predecessors
Continuous testing tools
In 2016, both Forrester Research and Gartner made Continuous Testing a primary consideration in their annual evaluations of test automation tools. Gartner have published an updated version of the research in 2019.Gartner evaluated 10 tools that met their criteria for enterprise-grade test automation tools. The evaluation involved inquiries with Gartner clients, surveys of tool users, vendor responses to Gartner questions, vendor product demonstrations. Gartner required tools to support native Windows desktop application testing and Android or iOS testing support as well as support 3 of the following: responsive web applications, mobile applications, package applications, API/web services. The results of the 2019 Magic Quadrant research are:
- Leaders: Tricentis, SmartBear Software, Eggplant
- Challengers: Micro Focus, IBM
- Visionaries: Broadcom, Parasoft
- Niche players: Worksoft, Ranorex, froglogic
- Leaders: Parasoft, IBM, Tricentis, Hewlett Packard Enterprise
- Strong performers: Microsoft, Micro Focus, SmartBear Software, TestPlant
- Contenders: Conformiq, Original Software, LogiGear