Painful. I cannot think of another way to describe the feeling that the technicians, managers, and other people directly or indirectly involved must have experienced when they learned that Knight Capital had lost $460 million in a matter of minutes due to a software problem, an event that nearly drove the company into bankruptcy.
And how was such a debacle possible? Due to an update that did not reach all the computer systems, leaving some systems running on old code. But let's start from the beginning.
To allow its buyers to participate in the Retail Liquidity Program (RLP) of the New York Stock Exchange, Knight Capital made numerous changes to its computer systems. Among these changes was the development of new code for SMARS, a high-speed algorithmic router that sent orders to the market. One of the main functions of SMARS was to receive orders from other components of Knight's trading platform ("parent" orders) and, based on available liquidity, send one or more representative orders ("children") to external sites for execution.
The new RLP code for SMARS was intended to replace functionality known as Power Peg, designed to increase or decrease the price of shares so that the behavior of algorithms in controlled environments could be properly verified.
Despite the fact that it would no longer be used after the update, the Power Peg functionality had to remain present since it could be needed during deployment. For this reason, an existing selector (flag) was recoded to allow it to be activated. Since this functionality was going to become obsolete, Knight would eventually remove it by setting this selector to the appropriate value.
In late July 2012, Knight deployed its new RLP code progressively on a limited number of SMARS servers over several days. However, during this deployment one of Knight's technicians forgot to copy the new code onto one of the SMARS servers. Knight had no review processes for these deployments, so no one noticed that one of the servers had not been updated as it should have been.
On August 1 of the same year, Knight received orders from its brokers to begin participating in the RLP. The seven servers with the new code processed the orders correctly, but there was one that still had the old code. As a result, the unupdated server began endlessly sending child orders for every parent order received to certain trading venues for execution, without taking into account the number of executions already completed (the Power Peg functionality no longer existed in the SMARS system).
Since the SMARS system was not aware of the orders already executed by that server with old code, Knight executed 4 million operations in 154 stocks for more than 397 million shares in approximately 45 minutes (the continuous buying/selling devalued the shares), resulting in estimated losses of $460 million.
By the time the technicians realized the error, it was too late. Despite this, Knight Capital managed to raise an investment of $400 million (including from Jefferies Group) to deal with the financial blow.
A catastrophic error, caused by poor review and update deployment procedures in computer systems.