Knight Capital Group is a corporation that uses complex automated trading systems to provide order execution for 19,000 equities in addition to trading services for options, currencies, and bonds. On August 1, 2012, Knight Capital suffered a disastrous electronic trading glitch. This bug was introduced during a software upgrade that occurred the night before the incident, involving the installation of the NYSE's new Retail Liquidity Program.
Knight CEO Tom Joyce stated that the glitch was a large, although simple, breakdown caused by an issue with trading technology. Knight is doing an internal investigation to identify what truly happened and to identify how they can enhance the system to prevent this type of incident from occurring again. The trading glitch resulted in a catastrophic $440 million dollar loss and prompted inquiries from regulators including the Securities and Exchange Commission. Publicly traded firms are required to adhere to federal regulatory requirements as described in the ISACA Cobit framework including managing changes.
Upgrading a trading environment can very difficult, potentially causing an outage resulting in downtime for the entire trading system. Every minute that a trading system is unavailable is costly because the market moves quickly, eliminating opportunities to trade at a favorable price. The Knight Capital incident was even more serious because it resulted in undesirable trades highlighting that the system lacked a switch to halt the flood of rapid trades, resulting in a catastrophic—possibly fatal—loss for the firm. DevOps is emerging as a set of principles and functions that can help streamline the entire build, package, and deployment of software. DevOps prevents mistakes by helping development and operations to work more closely together, identifying and eliminating risk early in the software and systems lifecycle. Many organizations learn that automating the entire build, package, and deployment process can indeed reduce risk and result in more reliable production deployments. Better communication between development and operations results in systems that are built and deployed with the needs of runtime operations in mind, which might have helped operations react more swiftly in this incident. It may take some time to sort out exactly what occurred in the Knight Capital incident, but what is clear is that complex software upgrades need to be performed in accordance not only with all regulatory guidelines but also making use of industry best practices—including DevOps!