Configuration Flags: A Love Story

[article]
Summary:

By implementing configuration flags as part of the initial stages of development, engineers imbue all new features with the capability to leverage system-level strategies, such as multivariate testing, beta testing, and “emergency toggles.” In this article, Noah Sussman describes some battle-tested strategies to implement and leverage configuration flags in production.

Imagine sending code to production that does not execute—at least not in a way visible to the customer. When the time comes, the company changes a text file (called a configuration file), which does not require a new build or production push; that also makes rollback easier. Many companies, from big ones like Google and Amazon to smaller ones like Flickr and Etsy, credit dark releases with speeding time to market and reducing risk. Configuration flag (config flag) architecture also contributes to the capability to perform large-scale beta testing; for example, it contributes to the capability to perform multivariate testing at scale.

The problem is not the what, but the how; the actual implementation of config flags may well be the least-understood part of the continuous delivery stack. I’m not aware of a place to go to get what people need—a little training and advice on a straightforward way to build a config flag architecture in the simplest way that can reasonably be expected to work.

In this article, I’d like to impart that training with some battle-tested strategies to implement and leverage config flags in production.

Branch in Code, Not in Source Control
As Paul Hammond has explained, branching in source control (including Git) has limitations that make it an inappropriate strategy for managing production deployments.

To quote Theo Schlossnagle:

In the online world, a software product drives a service to which users have access. There is, most often, a single copy of the actual software product in use. There is one consumer of the software: you. The users are consumers of the service built atop the software.

In the online world, a software product drives a service to which users have access. There is, most often, a single copy of the actual software product in use. There is one consumer of the software: you. The users are consumers of the service built atop the software.

This is a key concept and I found it counter-intuitive the first few (hundred) times I heard it. So I’ll reiterate: the only deployed “copy” of my website is the one running in my data center. Many people may consume my product. But that product is a service produced by the single copy of my software that is running on the host(s) in my data center(s).

Once I had gotten used to viewing the problem this way, it is became obvious that I could make changes to the product without affecting the quality of its service.

The Repo Doesn’t Have to Mirror the Org Chart
With config flags in place it’s possible for different groups to work on various projects at varying speeds, all while releasing weekly (or daily) to production.

Because each feature’s config flags are separate, it’s possible to deploy a feature to production while many other teams are arbitrarily working on other, much larger features in the same codebase.

Keep in mind, any code wrapped in a config flag is ready to go to production at any time. If it isn’t ready, the config flag is turned off and the result is a no-op. If the config flag is turned on but the code misbehaves, the config flag can be toggled off by changing a single line in the config file.

User Comments

2 comments
Timothy Western's picture
Timothy Western

Deploying code dark is an excellent idea from a production code standpoint. It can get a bit tricky though, when you are trying to write black box automation around it at the same time. If your test code may live in the same repository or code base, then you may be able to leverage the same flags to ignore tests that are dark, or maybe run tests to verify its dark, vs run tests that verify functionality when its live.

Using flags like this, enables you to have slightly different flag settings for your internal development environments for testing and building out the new feature, but keep them dark until you are ready to turn them on.

The other issue you may run into is that you'll get a lot more mileage out of more recent products with these flags than legacy software. It can take a bit of time to instrument things in a way so as not to affect much older code. One way around this is to find ways to divide the responsibility for that code from the newer feature code in such a way so that you can migrate as much of the newer code that may have less chance of impacting your sites key performance needs first, before tackling the more difficult legacy problems.

October 31, 2013 - 8:09am
Noah Sussman's picture

> If your test code may live in the same repository or code base, then you may be able to leverage the same flags to ignore tests that are dark, or maybe run tests to verify its dark, vs run tests that verify functionality when its live.

I would strongly recommend keeping tests and code in the same repository.

> The other issue you may run into is that you'll get a lot more mileage out of more recent products with these flags than legacy software.

Well yes, understanding and working with legacy systems is more expensive and risky in general than writing a new system from scratch. That aside, the "strangler" (a.k.a) "parallel rollout" pattern is a tried-and-true method of refactoring a legacy system to use config flags.

http://martinfowler.com/bliki/StranglerApplication.html

November 7, 2013 - 10:28am

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.