The CDS Workflow is a key feature of the OVH CI/CD Platform. This structural choice adds an additional concept above CI/CD pipelines and jobs, and after more than three years of intensive use, is definitely an essential feature.
Before delving into a full explanation of CDS workflows, let’s review some of the key concepts behind pipelines and jobs. Those concepts are drawn from the reference book, 8 Principles of Continuous Delivery
The basic element: “The job”
A job is composed of steps, which will be run sequentially. A job is executed in a dedicated workspace (i.e. filesystem). A new workspace is assigned for each new run of a job.
A standard build job looks like this:
You can use “built-in” actions, such as checkoutApplication, script, jUnit, artifact upload/download.
- The checkoutApplication action clones your Git repository
- The Script action executes your build command as “make build”
- The artifactUpload action uploads previously-built binaries
- The jUnit action parses a given Junit-formatted XML file to extract its test results
A pipeline: How to orchestrate your jobs with stages
With CDS, a pipeline is not a job flow. A pipeline is a sequence of stages, each of which contains one or more jobs.
A Stage is a set of jobs that will be run in parallel. Stages are executed sequentially, if the previous stage is successful.
Let’s take a real-life use case: the pipeline that built CDS. This pipeline has four stages:
- The “Build Minimal” stage is launched for all Git branches. The main goal of this stage is to compile the Linux version of CDS binaries.
- The “Build other os/arch” stage is only launched on the master branch. This stage compiles all binaries supported by the os/arch: linux, openbsd, freebsd, darwin, windows – 386, amd64 and arm.
- The “Package” stage is launched for all Git branches. This stage prepares the docker image and Debian package.
- Finally, the “Publish” stage is launched, whatever the Git branch.
Most tasks are executed in parallel, whenever possible. This results in very fast feedback, so we will quickly know if the compilation is OK or not.
CDS Workflows: How to orchestrate your pipelines
The workflow concept is a key feature, and widely considered a native, manageable and feature-rich entity in CDS. A CDS Workflow allows you to chain pipelines with manual or automatic gates, using conditional branching. A workflow can be stored as code, designed on CDS UI, or both, depending on what best suits you.
Let’s take an example. One workflow for building and deploying three micro-services:
- Build each micro-service
- Deploy them in preproduction
- Run integration tests on preproduction environment
- Deploy them in production, then re-run integration tests in production
For the building part, there is only one pipeline to manage, which is used three times in the workflow with a different application/environment context each time. This is called the “pipeline context”.
Any conditional branching against the workflow (e.g.“automatic deployment on the staging environment, only if the current Git branch is master”) can be executed through a “run conditional” set on the pipeline.
Let’s look at a a real use case. This is the workflow that builds, tests and deploys CDS in production at OVH (yes, CDS builds and deploys itself!):
- For each Git commit, the workflow is triggered
- The UI is packaged, all binaries are prepared, and the docker images are built. The “UT” job launches the unit tests. The job “IT” job installs CDS in an ephemeral environment and launches the integration tests on it. Part 2 is automatically triggered on all Git commits.
- Part 3 deploys CDS on our preproduction environment, then launches the integration tests on it. It is started automatically when the current branch is the master branch.
- Last but not least, part 4 deploys CDS on our production environment.
If there is a failure on a pipeline, it may look like this:
The same kind of workflow is used for building and deploying the Prescience Project (https://labs.ovh.com/machine-learning-platform):
But of course, you’re not limited to the most complex tasks with CDS Workflows! These two examples demonstrate the fact that workflows allow to build and deploy a coherent set of micro-services. If you have simpler needs, your workflows are, of course, simpler.
Pipeline reusability allows you to easily maintain the technical parts of the build, tests and deployments, even if you have hundreds of applications. If hundreds of applications share the same kind of workflows, you can leverage the maintainability of the workflow templates. We will talk more about this in a future post.
Much more than “Pipeline as Code”… “Workflow as Code”
There is no compromise with CDS. Some users prefer to draw the workflows by the web UI, others prefer to write yaml code. CDS lets you do both!
There are two ways to store workflows: either in the CDS database or on your Git repository with your source code. We call this “Workflow as Code”.
This makes it possible to have a workflow on a given branch, and then develop it on another branch. CDS will instantiate the workflow on the fly, based on the yaml code present on the current branch.
CDS is OVH open-source software, and can be found on https://github.com/ovh/cds, with documentation on https://ovh.github.io/cds.
Previous Posts:
- CDS Introduction: https://www.ovh.com/fr/blog/how-does-ovh-manage-the-ci-cd-at-scale/
- DataBuzzWord Podcast (French): https://www.ovh.com/fr/blog/understanding-ci-cd-for-big-data-and-machine-learning/
