ECHR Open Data - Build and Deployment Process for an open source data project

Introduction¶

The European Court of Human Rights Open Project (ECHR-OD) aims at providing data about the European Court of Human Rights. To reach this objective, the project follows few standards to ensure the highest possible data quality. In particular,

1. no data is manipulated by hand,
2. the whole creation process is available and open-source such that the consummable data can be recreated from scratch,
3. each generated piece of data must be versionned and intermediate file must be publicly available.

This is particularily important because on top of the usual pitfals common to all open-source projects, data projects can also suffer from poor data quality. The large amount of data makes it barely impossible to check for every chunk of data, and a lot of malformed data might go undiscovered, even with proper QA tests.

In this article, I present the general DevOps strategy adopted for the ECHR-OD project. Bare in mind that this project is managed only by myself and on my free time, and with very limited resources. Therefore, many aspects presented here are engineered taking into account these constraints. In other words, many parts would not scale for first class industrial projects.

Following the above-mentioned principles, we defined the following constraints on the release process:

1. The build and deployment must happen automatically on a regular basis,
2. The build and deployment must be fully automated,
3. The only point of failure must be connectivity or hardware (i.e. not the code, QA must be enforced properly, which is outside the scope of this entry),
4. A build failure should not be released but also not impact future build attempt nor production.

High Level Picture¶

The projects is composed of two main components:

• The process of collecting and processing the data from scratch,
• The explorer, a web component that serves three purposes:
  1. the main frontend for the project, including downloading the latest data,
  2. the API to programmatically retrieve the data,
  3. the explorer itself to visualize and explore the data.

Both have a separated git repository on GitHub. Both components are containerized for scalability, convenience and reproducibility.

The container process generates a build build_name in a folder build. This folder is mounted into the container explorer which loads the latest build at startup.

The release (build and deployment) process is is illustrated by the following sequence diagram:

The three steps can be described as follows:

1. Every month, the Cyclic Database Release workflow is automatically triggered by GitHub.
   • Build the image with the latest source
   • Trigger test and coverage
   • Build a small database with only 100 documents
   • Push the image to docker.io
   • Trigger ECHR-OD Workflow “Deploy”
2. The ECHR-OD Workflow “Database” is then started
3. The endpoint /api/build/update of the explorer API is called which trigger the update and the container reboot

Deployment and Update¶

Deploy Workflow¶

The workflow “Deploy” consists of a single action that does the following:

1. Open a SSH connection to a given server,
2. Clone a repository in a given location if it does not exist,
3. Checkout the new code from origin,
4. Create a tmux session and attach,
5. Trigger the workflow specified in the parameters of the Deploy workflow, including the final endpoint call specified (in our case /api/v1/build/update),
6. Detach the process

The source code for the action “Deploy” can be found here.

Build History, Build API and Build upgrade¶

For the explorer container to know that there is a new build, we introduced the build history and build info. Whenever a build is finished, a short summary is appended to the build build/.build_history and a descriptive file build_info.yml is added to build/<build_name>.

The explorer provides a simple API, mostly for internal usage, to manage build status:

• Latest build status for badge /api/v1/build/status:
  {"schemaVersion":1,"label":"Database Update","message":"2020/12/19 23:04:48","color":"green"}
  

• Current build status /api/v1/build/current:
  {"build_time":"2020/12/19 23:04:48"}

• Check if a new build is available /api/v1/build/new_build_available:
  [false,null] or [true,<build_name>]

and the most important:

• Update the current build /api/v1/build/update:
  1. Check if there is no running build and if a new build is available
  2. Create a backup for the current build in case it needs to be rolled back
  3. Copy and replace the current build by the new build
  4. Restart the explorer container

The last operation is slightly technical, because we need to restart the container from the container. But ultimately, the container is restarted and the new SQLite database is loaded such that the explorer can serve the updated data.

Conclusion¶

As we have seen, the main challenges faced by ECHR-OD are the lack of resources and the time constraint on GitHub Workflows. Initially, GitHub Workflows are made for pure software DevOps, and therefore, a time limit of 72h might be reasonable. However, for data science projects with a need to (pre)process large amount of data on a regular basis, this limit is easy to reach. We get around this limit by using our own server and a flexible and modular workflow system, inspired by GitHub and that we will describe in another entry.

Another engineering challenge was the properly synchronized the process container with the explorer container such that the explorer container is aware of the existence a new build and can restart itself.

Finally, we ensure that the production data is not disturbed by a failing build, which should happen only in case of problem with the runner (most likely connectivity or hardware issue).