Continuous Testing in DevOps

Agile Development Challenges with Testing

Software has shifted from a role of supporting the business to being a major source of innovation. Software has invaded all areas of operation for a business. This means there is an expectation of very rapid iterations of applications with shortened development and delivery times. The rise of agile development is in response to these pressures. Agile development allows development teams to be more agile and responsive to feedback with much shorter cycle times for deployment of software. However, this increased pace of development has meant more pressure on Ops and QA teams to release more software reliably.

The DevOps movement has focused on breaking down barriers between Ops and Development and having feedback and information flow easily between the teams. DevOps relies heavily on automation to make sure that software is released reliably and teams have the tooling to respond effectively to outages.

Test teams face a trifecta of pressures based around:

• Increasing complexity of software that is being tested
• Cost of maintaining dedicated testing environments that cover all supported platforms
• Shortened delivery schedules impact time available for testing

The good news is these issues can be addressed. Testing early, testing often and leaning on automation for infrastructure as part of the DevOps process can help resolve these problems.

What is Continuous Testing?

Continuous testing is the process of executing automated software tests as part of a Delivery pipeline to obtain immediate feedback of business risk associated with a software release. This means that the scope of testing is increased from just validating bottom up requirements and user stories to assessing system requirements associated with business goals.

DevOps and Continuous Testing

The adoption of DevOps has been driven in part by increasing software complexity and shortened delivery times. DevOps teams can leverage Continuous testing to ensure quality and help balance speed and quality.

This kind of testing leverages automation, the cloud, and service virtualization to eliminate testing bottlenecks. Leveraging Infrastructure as a code allows the simplification of the creation of complex test environments that can be easily deployed, shared and updated. These capabilities reduce the cost of provisioning and shorten test cycle times by allowing earlier integration testing.

Continuous testing along with automated release and deployment can:

• Drive down costs – Replacing traditional dedicated testing environments with on-demand environments in the cloud drives down the cost of testing
• Reduce risk – Testing earlier and more often helps find and resolve defects earlier in the development cycle.
• Shorten Cycle time – Having on-demand testing environments reduces dependency on the availability of environments and helps reduce the cycle time for testing

Automated Testing is not Continuous Testing

Making a distinction between test automation and continuous testing may seem like a semantic exercise, but the gap between automating functional tests and executing a continuous testing process is substantial.

The main differences between automated testing and Continuous testing can be grouped into 3 main categories:

• Risk – Continuous testing is built on the premise of estimating the amount of risk each software release will entail. With the compressed timelines for deployment, there is very little time for test design, maintenance and especially test execution. With Continuous testing, test cases need to be built with business risks in mind to allow proper assessment of risks.
• Breadth – Just knowing that a unit test failed or a UI test passed doesn’t tell you whether the overall user experience is impacted by recent application changes. To protect the end user experience, you need tests that are broad enough to detect when an application change inadvertently impacts functionality which users have come to rely on.
• Time – These days speed is of the essence when it comes to delivering software, and has become a key competitive differentiator. With Agile, DevOps and Continuous delivery, testing needs to happen in parallel with development. This means there are several impacts for the test teams:
• Testing is now integral to development and not tacked on at the end
• Tests need to run as soon as functionality is developed
• Tests must be fast to avoid creating bottlenecks
• Testing environments need to be easily created, torn down and updated.

Teams relying solely on test automation hit critical roadblocks when they move to a faster-paced development and release cadence. These include:

• Inability to create and execute realistic tests fast or frequently enough
• High test maintenance work because of fragile tests and large number of application changes
• No insight into how risky each release is to the business

Cloud – The Testing Equalizer

Traditionally software test teams try to replicate production environments. There is a great deal of effort and expense needed to procure and maintain these environments. Also, due to the complexity of software under test, the test environment will invariably drift in configuration from the production environment.

To address these concerns test teams, need better testing environments. The cloud is a much-needed enabler here. What the cloud enables for testing teams is to have a dynamic, highly agile infrastructure that can change as quickly as their requirements change. The cloud also takes away the cost of procuring and then maintaining these production clone test environments because complex environments can be specified in code and spun up on demand and spun down when not needed.

Service Virtualization + Continuous Testing…Test Early Test Often! The costs of setting up integration testing environments for complex software with all dependent services challenge an organization’s ability to conduct integration tests. Coupled with that, the pressures of faster delivery and the dependence on third party services complicates integration testing for test teams. An effective means of getting around these problems is Service Virtualization. When testing teams use service virtualization, testing is shifted left and testing teams can find problems earlier in the cycle when the cost to fix them is much lower. Testing teams can also author tests before dependent components are available to enable behavior driven testing.

This allows testing teams to:

• Create tests earlier
• Reuse tests and services
• Create tests and data that can be triggered during builds

Service virtualization can also simplify the creation of large workloads to enable performance testing and find bottlenecks. This allows test teams to have more confidence in the scalability of the application under test and its behaviour under varying loads.

Conclusion

The demand for software to be delivered faster, and for software and applications to do more resulting in increasing complexity of developed software forces organizations not having time to thoroughly validate and run every test.

But when continuous testing is implemented as an integral part of a successful DevOps strategy, the addition of early integrated automated and continuous ongoing testing can address the tradeoff between complete testing and speed. Opcito offers expertise around testing automation, cloud based on-demand infrastructure and service virtualization that ease the process of testing complex integration software applications.