Manual quality gates in CI/CD pipelines

The ultimate goal of microservices development is to achieve daily releases to production. The only way to accomplish this is to completely eliminate all manual steps and automate the process of delivering individual microservices. This can be done using a series of quality gates. The gates must guarantee the quality of the microservice itself and ensure that new versions of the microservice don’t not break the system. If any problems are detected, the delivery process must rollback the changes, leaving the system in a working state.

The process of automated delivery is called the “CI/CD Pipeline”.

‍

The simple CI/CD pipeline that is shown in the picture above consists of 4 stages:

• Development. Developers write code and tests and push their results into a code repository. Before pushing the code, developers must run unit and integration tests.
• Build. After the push, the process is initiated on a build server. It pulls the code, builds it, and runs unit and integration tests at component and container levels, to ensure the quality of the microservice.
• Stage deployment. The tested microservice is deployed into a stage environment, in which a fully working system is running. Once deployed, a series of system-level tests is executed. Even though we’ve already tested the microservice and know that it is fully functional, we don’t know whether or not it will play well with the rest of the system’s microservices. The purpose of this quality gate is to ensure that the system won’t be compromised by the new version of the microservice. If everything goes well, then the new version of the microservice is promoted to production.
• Production deployment. After having successfully completed Stage deployment, we attempt to deploy the microservice into production. Once deployed, we run all system-level tests once more to double check the quality of the system as a whole. If everything is fine - we leave the new version of the microservice in production.

If you have reliable tests with sufficient coverage, then you can probably trust them enough to be able to run the entire CI/CD pipeline in a fully automated fashion. But what if you don’t have automated tests? Or you simply can’t trust the well-being of your system and, potentially, the success of your entire company to some mere tests?

Interesting enough, most of my enterprise clients fall right into that category. Their systems are big, and their automated tests are quite spotty. Some unit tests may exist, but system-level tests are usually way behind. If you are also finding yourself in such a situation, how should you modify your CI/CD pipeline to throw-in some traditional manual testing?

‍

When manual testing is a must, a new QA environment and an associated QA deployment stage should be added between the Stage and Production deployment stages. It is not a good idea to perform manual testing in the Stage environment, because the environment itself can be quite chaotic, and you don’t know what exactly you’re actually testing. Any minute developers can push new changes, which can break already tested parts of the system. Testing in production is also not a good idea, because the whole point of testing is to deliver a quality product to customers.

Here’s how to execute manual verification in a CI/CD pipeline:

• Create a baseline (snapshot) of the system. If you use docker, tagging every deployed container with a baseline version number (usually X.Y.Z-rcN) is a good way to create a system baseline. Otherwise, you should come up with a working method to create reactive system baselines.
• Deploy the baseline to the QA environment and perform manual testing.
• If testing fails - drop the baseline, return to the Stage environment, create a new baseline, move it to the QA environment once again, and run the test anew. Repeat the process until the baseline reaches sufficient quality.
• When the baseline seems to be working as needed, it can be “promoted to production”. This process may require tagging the system with a new baseline version (excluding the “rc” part), creating some release notes, and announcing that the new release is ready.
• In the end, the “promoted” system baseline can be deployed in production

To read more about reactive baseline management, please have a look at the post that dedicated to this topic.

Happy microservicing,

Sergey Seroukhov, Enterprise Innovation Consulting

‍