QA Outsourcing with CI/CD: How to Choose a Reliable Vendor?

Photo by Lukas from Pexels

Shifting to the DevOps culture and adopting a CI/CD: Continuous Integration & Delivery Explained pipeline, enterprises may overlook the importance of QA. At some point, they find out that they cannot upgrade their QA processes’ maturity and QA engineers’ competencies quickly enough to keep up with the fast-paced CI/CD cycle. In fact, according to a Vanson Bourne survey, in the US, 73% of 200 enterprises IT decision-makers have adopted DevOps, but 57% of them experienced bottlenecks during software testing, which affected how quickly their software was released. To overcome these difficulties, companies may choose to turn to QA outsourcing. 

In this article, we will highlight the specifics of QA and testing processes in projects with CI/CD pipelines and dwell on the competencies a QA outsourcing provider should possess to drive value in such projects.

The specifics of QA and testing processes in projects with CI/CD pipelines

Businesses resort to CI/CD to release high-quality software frequently. Attaining this goal requires QA and testing processes to adjust to the fast-paced delivery cycle in the following ways:

• Rather than being an isolated task carried out when coding is over, testing becomes an integral part of the development process. CI/CD presupposes having ‘quality gates’ throughout the software delivery pipeline, and the code is required to pass through the gate in order to proceed to the next stage of the development cycle. The code should pass through at least four checks, comprised of automated integration, system, regression, and user acceptance testing. 

• The rate of test automation increases. Automated tests are executed to perform repetitive, parallel checks of expected software behavior. Contrary to popular belief, a CI/CD pipeline does not require the automation of all test cases. Test cases that are executed infrequently or change with every delivery cycle are not cost-effective to automate. Some testing types, such as exploratory and usability testing, are always performed manually.  

• With a continuous delivery pipeline, automated test cases are triggered and executed automatically whenever a recently completed code is pushed to the main repository. The possibility to trigger and execute test cases automatically requires such pre-testing activities as provisioning of test environments and test data to become automated as well. The task is performed with CI/CD tools, e.g., Jenkins, GitLab CI/CD, GoCD.

• Testing begins early in the delivery cycle. In projects with CI/CD pipeline, the share of API testing increases, which allows testing activities to start before an application’s UI is ready.

The competencies to search for in a QA outsourcing provider

The above characteristics (an integral role of testing in the development cycle, the automation of pre- and post-testing tasks, an increased rate of test automation, and a larger share of API testing versus UI testing) determine the set of skills a competent QA outsourcing provider should possess to be able to drive value in the project with CI/CD pipeline. So choosing a QA outsourcing vendor for your project, along with such factors as general domain knowledge, elaborate SLA, comprehensive KPIs, and a sound approach to quality management software make sure that a vendor has expertise in the following areas:

• The ability to build and sustain effective collaboration with the project participants.

Unlike one-time projects, projects with CI/CD pipelines presuppose long-term collaboration and require joined effort from all project participants. Thus, a QA outsourcing provider should be able to sustain friendly, result-oriented communication with the development, product planning, and operations teams, as well as in-house testers or TCoE. A QA outsourcing vendor should provide dedicated points of contact who will communicate with the client, which may include:

• Testing team lead for daily collaboration.
• QA manager for risk management, test planning and scheduling.
• Account manager for resolving escalated issues and discussing contract changes.
• CTO or CEO for strategic alignment and defining long-term priorities.

• The ability to automate testing processes and align them with the CI/CD cycle.

A QA outsourcing vendor should be able to set up the pipeline for continuous testing and tune its orchestration engine in such a way that there is no manual intervention in:

• Virtualizing test environments.
• Provisioning, managing, and updating test data.
• Triggering and executing automated test cases. 
• Executing such post-testing tasks as archiving test data assets, creating test reports, etc.

Carrying out these tasks requires a QA vendor to have a good understanding of such orchestration tools as Jenkins, GitLab CI/CD, GoCD. 

• Expertise in test automation.

A QA vendor should have experience in automating all types of testing applicable to a given project employing relevant test automation tools, e.g., Selenium, Appium, Unified Functional Test (UFT), Cucumber, TestComplete. In the projects with the CI/CD pipelines, integration, functional, regression, and performance testing are usually automated. Of equal importance is the vendor’s ability to ensure adequate test coverage and decide which test cases need to be automated.

• The ability to start testing activities as early in the software delivery cycle as possible.

As the goal of CI/CD is to provide the development team with feedback on the quality of software as early as possible, it is important that a QA outsourcing vendor can incorporate automated tests at the start of a software delivery lifecycle, for instance, validate an application’s business logic via automated API tests, without waiting for an app’s UI to be ready. 

To sum it up

Projects with the CI/CD pipelines have certain specifics in terms of how the QA and testing processes are organized. Choosing a QA outsourcing provider for a CI/CD-driven project, it is important to look beyond such factors as domain expertise, a detailed SLA, and comprehensive KPIs. Make sure that a QA outsourcing vendor can integrate QA and testing processes into the earliest stages of a software delivery lifecycle, is competent enough to automate test cases for various testing types, and is able to sustain result-oriented collaboration with the project