NASDAQ Crash: a black op?

[Jan Klimkowski]

The official version:

Nasdaq crash triggers fear of data meltdown

Digital infrastructure exceeding limits of human control, industry experts warn

Juliette Garside
The Guardian, Friday 23 August 2013 20.09 BST
Jump to comments (507)

Nasdaq is not the only victim of system crashes: Microsoft and the New York Times website both suffered failures this year. Photograph: Spencer Platt/Getty Images

A series of system crashes affecting Google, Amazon, Apple and Microsoft in the past fortnight has brought warnings that governments, banks and big business are over-reliant on computer networks that have become too complex.

The alarm was sounded by industry experts in the aftermath of a three-hour network shutdown that paralysed the operation of the Nasdaq stock market in New York on Thursday, on what should have been a quiet day of routine share trading on the exchange.

Jaron Lanier, the author and inventor of the concept of virtual reality, warned that digital infrastructure was moving beyond human control. He said: "When you try to achieve great scale with automation and the automation exceeds the boundaries of human oversight, there is going to be failure. That goes for governments, for consumer companies, for Google, or a big insurance company. It is infuriating because it is driven by unreasonable greed. In many cases, the systems that tend to fail, fail because of an attempt to make them run automatically with a minimal amount of human oversight."

The Nasdaq collapse was caused by a communication failure between its platform for processing quotes and trades and that of another party reportedly the New York Stock Exchange. So serious was the fallout that it resulted in a third fewer shares being traded in the US on that day.

"These outages are absolutely going to continue," said Neil MacDonald, a fellow at technology research firm Gartner. "There has been an explosion in data across all types of enterprises. The complexity of the systems created to support big data is beyond the understanding of a single person and they also fail in ways that are beyond the comprehension of a single person."

From high volume securities trading to the explosion in social media and the online consumption of entertainment, the amount of data being carried globally over the private networks, such as stock exchanges, and the public internet is placing unprecedented strain on websites and on the networks that connect them.

By 2017, an amount of data equivalent to all the films ever produced will be transmitted over the internet in a three-minute period, according to Cisco, a manufacturer of communications equipment.

Internet traffic today per person is measured in gigabytes, with six gigabytes of information exchanged per human per year. In 2017, that number will have risen to 16. By then, global data will be counted in zettabytes roughly one trillion gigabytes.

High frequency trading by computers built to automate buying and selling high volumes of shares by hedge funds and banks has triggered and magnified the impact of IT failures on stockmarkets. In May 2010, $862bn (£553bn) was erased from the value of US shares in 20 minutes when one company triggered a cascade of selling.

"You get under the covers and high frequency trading algorithms are beyond understanding," said MacDonald. "Sub-millisecond trades taking place, tens of thousands per second, and when that fails it fails spectacularly. That is what you are seeing manifested in Nasdaq."

This month's spate of outages came to international attention with the two-hour failure of the New York Times website on 14 August, during which it resorted to publishing articles on its Facebook page. While a malicious attack was initially suspected, the problem was caused simply by a scheduled system maintenance.

On the same day, Microsoft customers began to report email failures. The outage was traced to problems with the Exchange ActiveSync service which serves email to many of the world's smartphones. When Exchange hit a glitch, the sheer volume of phones trying to connect triggered a ripple effect that took three days to control.

On 16 August, many of Google's websites, from email to YouTube to its core search engine, suffered a rare four-minute global meltdown. The episode, the cause of which Google has not explained publicly, served to illustrate the sheer volume of traffic its servers process. During its outage, one monitor put the drop in global internet traffic at 40%.

Three days later, on 19 August, Amazon's North American retail site went down for about 49 minutes, with visitors greeted with the word "Oops". No explanation was given, but one estimate by Forbes put the cost to Amazon at nearly $2m in lost sales.

On 22 August, Apple's iCloud suffered a blackout that affected a small number of its customers but lasted 11 hours. Storing the collections of photos, music, documents and address books that would once have been kept on shelves at home, iCloud now has 300 million users.

"The volume of data overall is absolutely exploding," says Rachel Dines, senior analyst at Forrester. "This week has been especially bad for downtime. Because we are now so dependent on these high profile services we notice them more. The impacts for the companies are huge from both lost revenue but also more importantly reputation damage."

James Acres, whose company Netcraft monitors outages at data storage companies, says digital businesses are racing, not always successfully, to built the infrastructure needed to cope with the data that many consumers are gradually transferring into the cloud from the hard drives of their laptops or their collections of CDRoms.

"More and more people are putting their data in the cloud," says Acres, "and to deal with this services are changing their back end to cope, and because it's all quite new they are experiencing some difficulties."As well as selling books and music, Amazon is the largest provider of public digital storage space worldwide, and this side of the business was hit by an outage in 2012 despite upgrades designed to make its servers less likely to collapse.

"The outage at Amazon last year was traced back to some of the processes and technologies they had put in place to make it more resilient," said MacDonald. "It is almost like an auto-immune disease, where the systems they created to make it more resilient actually spread the failure more rapidly."

Lanier, whose Who Owns The Future? details the concentration of power among organisations with the largest computers, said outages would increase until human oversight was improved. "We don't yet have a design for society that can run this technology well. We haven't figured out what the right human roles should be."

An informed alternative suggestion.

Foul play by one or more of the High Frequency Trade players, (residing at a major Wall Street investment bank), who got on the wrong side of a massive trade?

The article below is written by Karl Denninger of Market Ticker who is highly knowledgable both on trading and on the innards of IT infrastructure.

Nasdaq BlackOut: The Next Day


Oh do come on.

The Nasdaq is now claiming that NYSE Arca caused the Nasdaq "full stop" yesterday.

I don't buy it.

For those who were not paying attention (or got overwhelmed by the jargon) yesterday the Nasdaq unexpected went "dark" for close to three hours.

Let's dissect this.

Some time before all of it became publicly known there was a problem I saw a number of option quotes inexplicably widen out to a ridiculous degree. I had no idea what was going on and at the time it just a few names; that sort of thing can happen due to rumors of news in a given stock, so standing alone that didn't raise much in the way of alarms.

Then the quotes slowed to a trickle.

And then the halt, first only for "Tape C" names (Nasdaq) on their exchange and then at their request across all venues.

We still don't know exactly what happened or why. Nasdaq this morning claims there was some sort of disruption coming from NYSE Arca, one of the other trading venues, that somehow managed to cause their systems to malfunction.

I have no idea if that's a legitimate explanation or not.

What I do know is that there have been plenty of odd things happening of late.

And I also know that yesterday, before the Nasdaq went dark, there were multiple instances of extremely high quote volume and the option spreads on a number of names went nuts.

As you're aware if you've been following a number of sources of market news, including myself and Nanex, there has long been a "tactic" of playing "feel the market up" among the HFT boys who issue thousands or even tens of thousands of quotes for each trade that actually executes, which leads one to believe that the law is being systematically broken.

What law is that? The Securities Act which says that any issuance of a bid or offer for any purpose other than to legitimately transact in the security in question is unlawful.

One would expect that the exploitation of any dislocation one may cause by putting such an order into the system would be per-se proof of intent.

Well, one would expect that if we lived in a land where the Rule of Law meant something.

But we don't live in such a world.

Instead, we live in a world where anyone who can manage to exploit something -- effectively ripping someone else off -- it's just fine so long as you're rich, powerful or well-connected -- and it helps if you're all of the above and can in addition threaten to bring financial Armageddon on the world if you are not left alone.

I don't know if we'll ever get to the bottom of exactly what caused the dislocation yesterday.

But this much I do know -- there is no real marketplace in the securities business until and unless the HFT games end, and anyone who puts quotes on the wire without intent to execute is prosecuted for attempted manipulation.

The simple technological fix for this is to require that all quotes be valid until executed or for 2 seconds, whichever is less, and then additionally require that for each open quote the firm have the documented ability to clear the trade they propose to contract for by placing the quote.

That would be (1) trivially easy to implement and (2) stop all the games -- all at once.

That there is no intent to actually present a fair market to the public is proved by the refusal to promulgate such a requirement.

[Magda Hassan]

Yeah, I wondered about this the other day too. Maybe building up to the big electromagnetic pulse...? Clear the books.