In his Behaviorally Speaking series, Bob Aiello discusses hands-on software configuration management best practices within the context of organizational and group behavior.
Corporations, government agencies, and other institutions need to embrace industry best practices that have been proven to help develop and implement reliable systems. One of the most important considerations is the need for a secure, trusted application base. This article will help you get started delivering systems that can be verified and supported while continuously being updated as needed.
The recent series of high-profile glitches has led many industry analysts to question the reliability of our financial infrastructure. The New York City 911 emergency call system and the newly launched healthcare.gov website have all had serious issues that impacted service reliability. These systems need to be fixed, and, therefore, reliable deployment is a must-have. The good news is that we really do know how to build reliable systems. Corporations, government agencies, and other institutions need to embrace industry best practices that have been proven to help develop and implement reliable systems. One of the most important considerations is the need for a secure, trusted application base. This article will help you get started delivering systems that can be verified and supported while continuously being updated as needed.
A secure trusted base needs to start with an operating system that is properly configured and verifiable. Deploying applications to an untrusted platform is obviously an unacceptable risk. But applications themselves also need to be built, packaged, and deployed in a way that is fully verifiable. The first place to start is baselining source code in a reliable version control system that has the capability to reliably track the history of all changes. A common source of errors is missing or ill-defined requirements, so it's fundamental to maintain traceability of requirements to changesets. Baselines provide for the management of multiple variants in the code and, more importantly, the ability to reliably support milestone releases without having to resort to heroic efforts to find and fix your code base.
The application build itself must be fully automated with each configuration item (CI) built with an embedded immutable version ID that facilitates the physical configuration audit; this is essential for verifying that the correct code can be verified as having been successfully being deployed. The CIs should themselves be packaged into a container, which is then cryptographically signed to verify the identity of the source and also verify that the container has not been compromised. The only way to get this right is to build the package in a secure way and deploy it in an equally secure manner. Deming was right when he pointed out that quality must be built in from the beginning, and this is a fundamental aspect of the application build used to deploy a secure trusted base.
The deployed application must be baselined in the runtime environment and constantly verified to be free of unauthorized changes. This approach fundamentally provides a runtime environment that is verifiable and actually facilitates the automated application deployment pipeline.
The secure trusted base must be frequently updated to provide new functionality and also retiring obsolete or unwanted code. Creating the automation to reliable build, package, and deploy code on a constant basis is essential to implementing a secure trusted base. While you cannot test quality in a system, building systems that are fully verifiable is very important. The deployment pipeline itself must be designed to be verifiable on a constant basis.
Complex systems today also have many interfaces that must be understood and verified. This may involve checking the feed of market data or simply ensuring that correct ports have been provisioned and remain available for inter-process communication. In fact, monitoring the application environment is another important aspect of ensuring the secure trusted base. This means that you do not want unnecessary ports opened, which would result in a security risk, and you also do not want application issues to occur due to a port being closed that is required by the application system.
In the secure trusted base, we automate the build of all code components and embed identifiers into the code to make it possible to audit and verify the configuration. We also provide a reliable packaged container that cannot be compromised and we deploy in a way that is fully verifiable. Most importantly, we ensure that unauthorized changes can be detected and then remediated if they do occur. This may sound like a lot of effort and it actually does take a lot of work. The DevOps approach enables the development of these automated procedures from the very beginning of the application lifecycle which is really the only viable approach. More importantly the DevOps approach to creating an automated deployment pipeline enables you to rapidly build, package and deploy applications.
My career has often been focused on joining a team that is having problems releasing its code. The first thing that I do is push for more frequent releases and, whenever possible, smaller in scope. We automate each step and build in code to test and verify the deployment pipeline itself. The codebase that can be quickly and reliably deployed is inherently more secure. Ideally, you want to start from the very beginning of the software and system development effort. More often than not, I have had to jump onto a moving train. Taking an iterative approach to scripting and automated the application build, package and deployment will help you create a secure and reliable trusted application base!
[1] http://www.finra.org/Industry/Regulation/Guidance/TargetedExaminationLetters/P298161
[2] Deming, W. Edwards (1986). Out of the Crisis. MIT Press
