Rainmaker: Local Linux Development Containers for the Discerning Drupal Developer
Rainmaker is a local development environment for developers. It makes use of Linux Containers to rapidly allow you to clone your whole development environment - including the codebase, files and database.
In simple terms, this lets you have one local development environment which can be branched to create other containers for development of specific features. A bit like Gitflow - but for everything, not just the codebase.
If this sparks your interest then please bear with me whilst I take you on a journey on how we got here.
The Problem
As a Drupal developer building and maintaining Drupal sites my time is broken up between:
- Investigating and resolving bugs/defects with the current live/production version of a site
- Applying security fixes and recommended module updates
- Building new features
At Deeson we use Gitflow and utilise:
- “hotfix” branches for fixing bugs, applying security updates and recommended modules updates
- “feature” branches for site additions and new site features
We do all of our development locally creating Git tags and releasing the tagged Git revision to the production site.
Drupal 7 is notorious for putting configuration into the database that backs the site so we utilise the Drupal Features contrib module to get configuration of content types, fields, display modes, etc out of the database and into code where it can be version controlled.
In spite all of the above it is still not completely possible to prevent the Drupal database containing artefacts from other iterations of the Drupal site as you switched between your branches in Git.
This is not ideal as those artefacts can prevent the site from working correctly with the current branch that you are working with.
Real example
Consider this scenario. We are working on a new feature in a Gitflow feature branch. We have a content type named “news” with an existing field of “title” and we are adding a field named “byline”.
We used Drupal Features to get the configuration for “byline” into our “news” feature module. During the course of our development we have to make a change to the field settings for “title” in the production site.
We create a new Gitflow “hotfix” branch and check it out. If we were to list the state of our Drupal Features we would find that the “news” feature is showing as overridden. Why? Because the site database contains configuration for the “byline” field which is attached to the “news” content type and those settings are not in the “news” feature in our hotfix branch.
So, what can we do to avoid those artefacts from getting in the way of our site development?
We could dump the state of the database to disk before switching to another branch and after switching branch restore that branch’s database. Drupal databases can grow large and this can be a slow process and not very efficient for small changes we might need to make. It is also prone to human error.
OK, so why not create a database for each branch of our site?
Whilst there is an initial setup cost per branch, this is more efficient than the previous approach.
However, we would need to keep changing the database name in the Drupal settings or come up with some clever logic allowing Drupal to select the correct database based on branch. Once again this is still prone to human error.
The above approach does not solve the problem of preventing artefacts in our Solr indices. So, maybe we create separate Solr cores per site branch to avoid this? While that could work we are now growing the initial branch setup and maintenance costs.
Instead, I will simply create a new installation of the Drupal site, with its own database, Solr core. I will get a copy of the production database, maybe even the sites files, and reindex the site content into Solr.
Now we are on the right track. But the setup cost of each new branch is now massive to the point we would have to ask ourselves whether what we are going to work on warrants the setup costs or could we just live with the risk of artefacts and avoid all of this?
The Nirvana of Containerised Development
At Deeson we have been trying to solve the challenges above now for a few months and we believe we have found a strategy that is free from the kinds of human error that has been discussed above.
Imagine the scenario where we have a sort of Virtual Machine for each of the branches of a Drupal site we are working on.
Each VM runs its own database, Solr core, has its own public/private filesystem. If we had a copy of the production version of a site running in a VM, we could simply clone that VM, but the cloning would need to be quick and efficient.
Enter Linux Containers (or LXC). Linux Containers are like lightweight Virtual Machines running inside of Linux. If you place the filesystem of each container on a filesystem like BTRFS then it is possible for a Linux Container to be cloned in seconds. If we could network those containers so that our host machine could reach them and if we could introduce some DNS resolution so each container could have a unique name, then we could potentially have found our optimal solution.
Well, we have been working really hard at Deeson to make this happen and we are pleased to introduce to you the development environment we are using - Rainmaker.
This is being used in anger now but still in development. If you want to try out the latest Alpha then follow the links below - otherwise stay tuned for more updates!