Panopticon of global surveillance

[Magda Hassan]

The NSA's power largely comes from its myth of being super-smart and hyper-competent. So much for that.

Spying Blind

The National Security Agency has an intelligence problem: It won't admit how dumb it is.

BY SHANE HARRIS | AUGUST 16, 2013

The Obama administration's claim that the NSA is not spying on Americans rests on a fundamental assertion: That the intelligence agency is so good at distinguishing between innocent people and evildoers, and is so tightly overseen by Congress and the courts, that it doesn't routinely collect the communications of Americans en masse.


We now know that's not true. And we shouldn't be surprised. The question is, why won't the NSA admit it?
On Thursday night, the Washington Post released a classified audit of NSA's intelligence-gathering systems, showing they are beset by human error, fooled by moving targets, and rely on so many different servers and databases that NSA employees can't keep tabs on all of them.
It had been previously reported that the NSA had unintentionally collected the communications of Americans, in violation of court orders, as it swept up electronic signals in foreign countries. But officials had sought to portray those mistakes as limited, swiftly corrected, and not affecting that many people.
Wrong again.
One of the reasons that the NSA has been able to gather so much power is that the agency has built a reputation over the years for super-smarts and hyper-competence. The NSA's analysts weren't just the brainiest guys in the room, the myth went; they were the brightest bulbs in the building. The NSA's hackers could penetrate any network. Their mathematicians could unravel any equation. Their cryptologists could crack any cipher. That reputation has survived blown assignments and billion-dollar boondoggles. Whether it can outlast these latest revelations is an open question.


The Post found that the NSA "has broken privacy rules or overstepped its legal authorities thousands of times each year since Congress granted the agency broad new powers in 2008..." That's the year when NSA's global surveillance system went into hyperdrive. The agency was granted unprecedented authority to monitor communications without individual warrants and to surveil whole categories of people and communications.
Most of the violations affecting Americans' information were the result what the agency calls "incidental collection." So how many Americans were caught up in the NSA's surveillance nets as they were dragged across supposedly foreign targets? The exact number is unclear. But the short answer is: lots and lots of them.
In one instance, a programming glitch collected a "large number" of calls from Washington, D.C, instead of the intended targets in Egypt, according to the audit. Somehow, the area code 202 (for Washington) was keyed instead of 20 (the country code for Egypt.) The NSA's supposedly discriminating surveillance architecture was undone by a typo.
The audit reveals a recurring problem with human error in the day-to-day operations of global surveillance and shows what a messy and imprecise business it can be. In the first quarter of 2012, 123 incidents of non-compliance with the rules, or 63 percent of those examined, were attributed to human or operator error. These included typographical errors, inaccurate or overbroad search queries, and what the report calls "inaccurate or insufficient research information and/or workload issues."
Analysts needed more "complete and consistent" information about their targets to avoid errors, the audit found. This suggests that while the NSA's collection systems are dipping into data streams, the analysts aren't always equipped to determine who is and isn't a legitimate target.
The NSA's systems also have problems knowing when a target is on the move, and possibly has entered the United States. (When he does, different regulations come into play about how the surveillance is authorized and what can be monitored without approval from the court.)
As recently as 2012, NSA was not always able to know when targets using a mobile phone had crossed a U.S. border. These so-called "roamers" accounted for the largest number of technological errors in the violations that were examined.
A problem discovered last year, which appears in the report under the heading "Significant Incidents of Non-Compliance," helps illustrate how NSA is collecting so much information that it can actually lose track of it and store it in places where it shouldn't be.
In February 2012, the NSA found 3,032 "files containing call detail records" on a server. A call detail record, or CDR, is analogous to a phone bill. It shows whom was called, when, and for how long. This is metadata, like what's collected today on all phone calls in the United States.

http://www.foreignpolicy.com/articles/20...telligence

[Magda Hassan]

No safety in whistle-blower laws.

How to trap a whistleblower

Tell them that going through "proper channels" will provide meaningful redress to their concerns, not injure them

By Jesselyn Radack Barack Obama talks with Jay Leno during the taping of his appearance on "The Tonight Show with Jay Leno" on Aug. 6, 2013. (Credit: AP/Jacquelyn Martin)

Last week, President Obama misled the public when he told a comedian Jay Leno that protected legal channels exist that Edward Snowden could have used to challenge government misconduct:

I can tell you that there are ways, if you think that the government is abusing a program, of coming forward. In fact, I, through Executive Order, signed whistleblower protection for intelligence officers or people who are involved in the intelligence industry.

This message is false. And the President repeated it at his press conference a few days later. Obama is referring to Presidential Policy Directive #19. If the President had bothered to read his own Executive Order, he would have known that it was not implemented at all when Snowden blew the whistle on the National Security Agency. Further, it fails to provide protected legal channels to contractor positions such as Snowden's.
Thomas Drake, a former senior executive at the NSA, is living proof of how insidious the "channels" argument is. Shortly after 9/11, he complained about NSA programs that were embryonic versions of what Snowden is now revealing. He complained that NSA foreign collection programs were being turned inward on Americans. One of those programs, Stellar Wind, stripped off data anonymization features, auditing trails, and other privacy protections that were available in a cheaper, effective, and non-intrusive program called ThinThread.
Drake complained to his boss, to the NSA Inspector General, andwith three other retired NSA colleagues and a former House intelligence stafferto the Department of Defense Inspector General. The Inspector General substantiated their claims, but immediately classified its report to keep it out of public view. (Most portions are now unclassified and never had to be.)
Drake then served as a material witness in two key 9/11 investigations. He told Congress about multiple secret domestic surveillance programs, including Stellar Wind, and critical indications and warning intelligence about al Qaeda and associated movements pre- and post-9/11, which NSA did not share.
After the infamous New York Times article that revealed NSA's warrantless wiretapping of Americans, the Justice Department launched an investigationnot of the U.S.'s vast lawbreaking, but for those who exposed the secret surveillance. That multi-million dollar investigation spanned five years, required five full-time prosecutors and 25 FBI agents.

---

---

Although Drake and his colleagues were not sources, they were made targets of this federal criminal "leak investigation." The IG that had promised them protection and confidentiality sold them out to the Justice Department, and Congress failed to protect its witness, Mr. Drake. This was just one element of ruinous retaliation that went on for years and in some respects has not yet ended. Reprisals included getting fired, having security clearances pulled, armed raids on their' homes, shattered relationships with friends and family, and depletion of retirement accounts and second mortgages to pay attorneys' fees.
Drake became the Obama administration's first Espionage Act prosecution of someone accused of mishandling classified information. This draconian experiment soon escalated into a full-blown war on whistleblowers. The Drake case collapsed in spectacular fashion when the judge found that the information Drake possessed was completely unclassified, and had only been marked otherwise after it was seized from his home.
Around the same time, Senators Ron Wyden (D-OR) and Mark Udall (D-CO) claimed that the Justice Department had a secret, twisted interpretation of the Patriot Act that enabled domestic surveillance activities. Wyden said famously: "When the American people find out how their government had secretly interpreted the Patriot Act, they will be stunned and they will be angry."
Despite being treated unmercifully by the government, that fall, Drake and two of the same NSA colleagues with whom he blew the whistle, briefed Senators Wyden and Udall on the secret interpretation of 215. One of them, William Binney, had written the algorithm that was bastardized for use in a number of the secret surveillance programs. The result: Nothing.
Not only did going through proper channels provide no meaningful redress to the five whistleblowers' complaints, it gravely injured them. Drake lost his job, security clearance, and income stream, while simultaneously incurring half a million dollars in legal debt. And that was just during the investigatory phase. By the time of his indictment, he was declared indigent. Today, he works as a wage-grade employee at an Apple computer store, a far cry from his six-figure job at NSA.
Drake's story puts the lie to the notion that internal channels serve as anything other than a trap for unwitting whistleblowers. What is so revealing is that if Snowden had gone through internal channels, the outcome would have been worse: the United States would have charged him with espionage and he'd be in jail for, in essence, spying on his own country on behalf of the public. It should not require martyrdom for a free citizen to challenge government abuses of power. It should not require choosing one's conscience over one's career, citizenship, or freedom.

[Jan Klimkowski]

Whistleblowers are truthtellers.

Which is why the military-multinational-intelligence complex will always brand them traitors and subversives.

[Magda Hassan]

The NSA Searches Ten Times as Much of the Internet as It Said It Does

AP

Philip Bump 2,778 Views 2:51 PM ET
The National Security Agency assured Americans last week that it only surveils a tiny percentage of the web data it collects. But it turns out the NSA screwed up the math, and that percentage was off by an order of magnitude.
That error is in a document released by the agency on the heels of the president's speech earlier this month announcing measures to review NSA surveillance. We described the math at stake last week, but the pertinent section is this:

Unfortunately, if you do the math in suggested by that paragraph, you don't get that tiny percentage, 0.00004 percent, or 4 parts per 10 million. It's actually 0.0004 percent, with one fewer zero or 10 times as much as the NSA suggested. It's ten dimes on the basketball court, not one. (See the math at the bottom of this post.)
That's significant largely because of the weight the NSA puts on its percentages. In a New York Times article last Friday, the agency used similar tiny numbers to respond to The Washington Post's blockbuster report indicating that it had repeatedly violated Americans' privacy. (We spotted this via Mother Jones' Kevin Drum.)

The official, John DeLong, the N.S.A. director of compliance, said that the number of mistakes by the agency was extremely low compared with its overall activities. The report showed about 100 errors by analysts in making queries of databases of already-collected communications data; by comparison, he said, the agency performs about 20 million such queries each month.

Twenty million queries, as Drum points out, is a lot of daily queries (or, if you prefer, database searches). It's about 666,000, in fact, in a 31-day month. That's about seven queries every second. (How the NSA defines "query" in this context isn't clear.) In the context of the amount of data the NSA processes, it's also significant. Each day, using the 0.025 percent of 1.6 percent figure above, the government reviews about 7.304 terabytes of data. If you're curious, the ratio of data reviewed to number of queries is about 12.2 megabytes meaning that the government sets aside 12 megabytes for every query it runs.
Between the second quarter of 2011 and the first of 2012, the NSA committed about 7.5 privacy violations each day. Which was the NSA's point: of the 20 million queries a month, only a tiny, tiny percentage violate Americans' privacy. But a tiny percentage of a big number gives you seven privacy violations every 24 hours.
The NSA's incorrect .00004 percent figure was picked up by a variety of outlets at CNN and the Daily Mail, for example. Ten times a very, very small number is still a very, very small number, but it's a small number that represents 10 times as much surveillance as the NSA originally indicated.
Update, 5:00 p.m.: Vanee' Vines of the NSA responded to our question about the calculation over email:

Our figure is valid; the classified information that goes into the number is more complicated than what's in your calculation.

We asked for further clarification of the discrepancy between the numbers. Vines replied:

Our overall number is valid. I'm not sure why you're calling this a "discrepancy" when the number in the white paper is valid.

The math

[TABLE]
[TR]
What Percent Quantity [/TR]
[TR]
[TD] Daily internet traffic[/TD]
[TD] [/TD]
[TD] 1,826 petabytes, or
1,826,000 terabytes[/TD]
[/TR]
[TR]
[TD] Amount the NSA "touches"
(.016 * 1826000)[/TD]
[TD] 1.6%[/TD]
[TD] 29,216 terabytes[/TD]
[/TR]
[TR]
[TD] Amount selected for review
(.00025 * 29216)[/TD]
[TD] .025%[/TD]
[TD] 7.304 terabytes[/TD]
[/TR]
[TR]
[TD] Review amount as percentage of daily
(7.304 / 1826000)[/TD]
[TD] .0004%[/TD]
[TD] [/TD]
[/TR]
[/TABLE]
http://www.theatlanticwire.com/politics/...ath/68490/

[Magda Hassan]

Google and the NSA: Who's holding the shit-bag' now?

by Julian Assange
August 24th, 2013


Image latimesblogs.latimes.com

It has been revealed today, thanks to Edward Snowden, that Google and other US tech companies received millions of dollars from the NSA for their compliance with the PRISM mass surveillance system.
So just how close is Google to the US securitocracy? Back in 2011 I had a meeting with Eric Schmidt, the then Chairman of Google, who came out to see me with three other people while I was under house arrest. You might suppose that coming to see me was gesture that he and the other big boys at Google were secretly on our side: that they support what we at WikiLeaks are struggling for: justice, government transparency, and privacy for individuals. But that would be a false supposition. Their agenda was much more complex, and as we found out, was inextricable from that of the US State Department. The full transcript of our meeting is available online through the WikiLeaks website.
The pretext for their visit was that Schmidt was then researching a new book, a banal tome which has since come out as The New Digital Age. My less than enthusiastic review of this book was published in the New York Times in late May of this year. On the back of that book are a series of pre-publication endorsements: Henry Kissinger, Bill Clinton, Madeleine Albright, Michael Hayden (former head of the CIA and NSA) and Tony Blair. Inside the book Henry Kissinger appears once again, this time given pride of place in the acknowledgements.
Schmidt's book is not about communicating with the public. He is worth $6.1 billion and does not need to sell books. Rather, this book is a mechanism by which Google seeks to project itself into Washington. It shows Washington that Google can be its partner, its geopolitical visionary, who will help Washington see further about America's interests. And by tying itself to the US state, Google thereby cements its own security, at the expense of all competitors.
Two months after my meeting with Eric Schmidt, WikiLeaks had a legal reason to call Hilary Clinton and to document that we were calling her. It's interesting that if you call the front desk of the State Department and ask for Hillary Clinton, you can actually get pretty close, and we've become quite good at this. Anyone who has seen Doctor Strangelove may remember the fantastic scene when Peter Sellers calls the White House from a payphone on the army base and is put on hold as his call gradually moves through the levels. Well WikiLeaks journalist Sarah Harrison, pretending to be my PA, put through our call to the State Department, and like Peter Sellers we started moving through the levels, and eventually we got up to Hillary Clinton's senior legal advisor, who said that we would be called back.
Shortly afterwards another one of our people, WikiLeaks' ambassador Joseph Farrell, received a call back, not from the State Department, but from Lisa Shields, the then girlfriend of Eric Schmidt, who does not formally work for the US State Department. So let's reprise this situation: The Chairman of Google's girlfriend was being used as a back channel for Hillary Clinton. This is illustrative. It shows that at this level of US society, as in other corporate states, it is all musical chairs.
That visit from Google while I was under house arrest was, as it turns out, an unofficial visit from the State Department. Just consider the people who accompanied Schmidt on that visit: his girlfriend Lisa Shields, Vice President for Communications at the CFR; Scott Malcolmson, former senior State Department advisor; and Jared Cohen, advisor to both Hillary Clinton and Condoleezza Rice, a kind of Generation Y Kissinger figure a noisy Quiet American as the author Graham Greene might have put it.
Google started out as part of Californian graduate student culture around San Francisco's Bay Area. But as Google grew it encountered the big bad world. It encountered barriers to its expansion in the form of complex political networks and foreign regulations. So it started doing what big bad American companies do, from Coca Cola to Northrop Grumman. It started leaning heavily on the State Department for support, and by doing so it entered into the Washington DC system. A recently released statistic shows that Google now spends even more money than Lockheed Martin on paid lobbyists in Washington.
Jared Cohen was the co-writer of Eric Schmidt's book, and his role as the bridge between Google and the State Department speaks volumes about how the US securitocracy works. Cohen used to work directly for the State Department and was a close advisor to both Condolezza Rice and Hillary Clinton. But since 2010 he has been Director of Google Ideas, its in-house think/do' tank.
Documents published last year by WikiLeaks obtained from the US intelligence contractor Stratfor, show that in 2011 Jared Cohen, then (as he is now) Director of Google Ideas, was off running secret missions to the edge of Iran in Azerbaijan. In these internal emails, Fred Burton, Stratfor's Vice President for Intelligence and a former senior State Department official, describes Google as follows:
"Google is getting WH [White House] and State Dept support and air cover. In reality they are doing things the CIA cannot do…[Cohen] is going to get himself kidnapped or killed. Might be the best thing to happen to expose Google's covert role in foaming up-risings, to be blunt. The US Gov't can then disavow knowledge and Google is left holding the shit-bag"
In further internal communication, Burton subsequently clarifies his sources on Cohen's activities as Marty Lev, Google's director of security and safety and.. Eric Schmidt.
WikiLeaks cables also reveal that previously Cohen, when working for the State Department, was in Afghanistan trying to convince the four major Afghan mobile phone companies to move their antennas onto US military bases. In Lebanon he covertly worked to establish, on behalf of the State Department, an anti-Hezbollah Shia think tank. And in London? He was offering Bollywood film executives funds to insert anti-extremist content into Bollywood films and promising to connect them to related networks in Hollywood. That is the Director of Google Ideas. Cohen is effectively Google's director of regime change. He is the State Department channeling Silicon Valley.
That Google was taking NSA money in exchange for handing over people's data comes as no surprise. When Google encountered the big bad world, Google itself got big and bad.
http://thestringer.com.au/google-and-the...t-bag-now/

[Magda Hassan]

The Pentagon as Silicon Valley's Incubator

Illustration by James C. Best, Jr/The New York Times

By SOMINI SENGUPTA

Published: August 22, 2013

SAN FRANCISCO In the ranks of technology incubator programs, there is AngelPad here in San Francisco and Y Combinator about 40 miles south in Mountain View. And then there is the Pentagon.


Jay Kaplan, left, and Mark Kuhr met at the National Security Agency and raised $1.5 million in seed money to start Synack.



In the last year, former Department of Defense and intelligence agency operatives have headed to Silicon Valley to create technology start-ups specializing in tools aimed at thwarting online threats. Frequent reports of cyberattacks have expanded the demand for security tools, in both the public and private sectors, and venture capital money has followed. In 2012, more than $1 billion in venture financing poured into security start-ups, more than double the amount in 2010, according to the National Venture Capital Association.
For years, the Pentagon has knocked on Silicon Valley's door in search of programmers to work on its spying technologies. But these days, it's the Pentagon that is being scouted for expertise. Entrepreneurs and venture capitalists are finding it valuable to have an insider's perspective on the national security apparatus when trying to find or prevent computer vulnerabilities or mine large troves of data.
"They have unique insights because they've been on the front line," said Matthew Howard, a former intelligence analyst in the Navy and now a managing partner at Norwest Venture Partners, referring to former military and intelligence operatives who have hatched start-ups. He has invested in several such companies. "Now they've got commercial desires. The lines are blurring."
One of the start-ups is Synack, which promises to vet an army of hackers to hunt for security vulnerabilities in the computer systems of government agencies and private companies. The company's co-founders, Jay Kaplan and Mark Kuhr, met in Fort Meade, Md., in the counterterrorism division of the National Security Agency. They left the agency in February after four years there, and later decamped to Silicon Valley. Within weeks, they had raised $1.5 million in seed money; they are now working with their first customers and pitching their experience in the spy agency.
"Doing things on a classified level really opens your eyes," Mr. Kaplan said. "The government is doing a lot of interesting things they don't disclose. You have a unique perspective on what the adversary is doing and the state of computer security at a whole other level."
Morta Security, another of the start-ups, was founded by Raj Shah, a former F-16 fighter pilot for the Air Force in Iraq. He described himself as "a policy adviser" to the N.S.A. before moving to Silicon Valley to establish the company this year with two former analysts. Morta's work is in such "stealth mode," in valley parlance, that the company has said nothing about what it is working on. Nor would Mr. Shah describe fully what his two co-founders were doing at the agency before they formed the company.
"There are very sophisticated threats that are able to steal data from corporations and government," is all Mr. Shah would say. "Our guys' background they just have a deeper understanding of that problem."
Though Silicon Valley sees itself as an industry far removed from the Beltway, the two power centers have had a longstanding symbiotic relationship. And some say the cozy personal connections of ex-intelligence operatives to the military could invite abuse, like the divulging of private information to former colleagues in the agencies.
"They have enormous opportunities to cash in on their Washington experience, sometimes in ways that fund further innovation and other times in ways that might be very troubling to many people," said Marc Rotenberg, executive director at the Electronic Privacy Information Center in Washington. "Both sides like to maintain a myth of distant relations. The ties have been in place for a long time."
The ties are more than personal; the National Security Agency is among the few organizations in the world, along with companies like Facebook and Google, with a cadre of engineers trained in mining big data.
By working at the N.S.A., "you get to be on the bleeding edge, not just the cutting edge of what's possible," said Oren Falkowitz, who left the agency last year to start Sqrrl, a big data analytics company based on technology developed at the agency. Mr. Falkowitz has since left Sqrrl, which is in Boston, and is considering moving to Northern California to start working with a big data company.
Last year, Sumit Agarwal left his post as a deputy assistant secretary of defense to join Shape Security, a Mountain View company that offers what it calls "military grade" security solutions against botnets, groups of infected computers used for attacks.

Shape Security's chief executive is Derek Smith, a former Pentagon consultant whose last company, Oakley Networks, which specialized in detecting insider threats, was sold to Raytheon, the military contractor, in 2007. Since its inception in 2011, Shape Security has raised $26 million in venture financing.




Computer security experts are leaving other parts of government for start-ups, too. Sameer Bhalotra, who worked on cybersecurity issues at the White House, was recruited by a Redwood City-based security company called Impermium. And Shawn Henry, a former computer security specialist from the F.B.I., left his job in government last year to help establish CrowdStrike, a computer security firm.
In Israel, government security workers have long found a career path in moving to start-ups, said Peter Wagner, a partner at a recently opened venture firm, Wing Venture Partners, in Menlo Park. Many Israeli entrepreneurs come out of the Israeli military and intelligence services, he pointed out.
"It's not surprising that some of the same type of experience is finding its way into entrepreneurial endeavors here in the U.S.," Mr. Wagner said.
The idea for Synack came to its founders, Mr. Kuhr, 29, and Mr. Kaplan, 27, when they were working side by side at the N.S.A.'s computer network operations division; within the agency, that includes figuring out how to attack or exploit data gathered from a computer network. Nights and weekends, they hatched their business plan. They proposed to assemble an army of vetted bounty hunters from around the world to find security bugs. Their product is a variation of the so-called bug bounty programs run by large companies, like Facebook and Microsoft, that in effect invite security researchers to try to crack vulnerabilities in their systems and reward them if they do.
Part of their pitch to potential customers is that they will vet the bounty hunters before setting them loose. They hope to sign up government agencies as customers, along with private firms, especially in the software services sector.
"We are able to provide security experts previously inaccessible to companies," Mr. Kaplan added.
Both men's college educations were paid for by N.S.A. scholarships Mr. Kaplan at George Washington University, Mr. Kuhr at West Point Military Academy and then at Auburn University. With that came an obligation to work at the agency, which they did, each for four years.
"We really liked our jobs there," Mr. Kuhr said.
Then they headed west, drawn by the same dream of riches that draws so many other people here.

http://www.nytimes.com/2013/08/23/techno...ss&emc=rss

[Peter Lemkin]

The latest disclosures from whistleblower Edward Snowden have provided more details on how the National Security Agency has spied on the United Nations and countries worldwide. Citing Snowden's leaks, the German magazine Der Spiegel reports the NSA decoded the U.N.'s internal video conferencing system to eavesdrop last year. One NSA official wrote: "the data traffic gives us internal video teleconferences of the United Nations. (yay!)" The spying on U.N. communications would violate the U.N.'s Espionage Act. Overall, an NSA operation called "Special Collection Services" bugged or monitored some 80 embassies and consulates around the world, including the European Union offices near the U.N.'s New York headquarters as well as the International Atomic Energy Agency. According to Der Speigel, the NSA's embassy spy program "has little or nothing to do with warding off terrorists."

[Magda Hassan]

The latest disclosures from whistleblower Edward Snowden have provided more details on how the National Security Agency has spied on the United Nations and countries worldwide. Citing Snowden's leaks, the German magazine Der Spiegel reports the NSA decoded the U.N.'s internal video conferencing system to eavesdrop last year. One NSA official wrote: "the data traffic gives us internal video teleconferences of the United Nations. (yay!)" The spying on U.N. communications would violate the U.N.'s Espionage Act. Overall, an NSA operation called "Special Collection Services" bugged or monitored some 80 embassies and consulates around the world, including the European Union offices near the U.N.'s New York headquarters as well as the International Atomic Energy Agency. According to Der Speigel, the NSA's embassy spy program "has little or nothing to do with warding off terrorists."

Yes, not such a surprise that they spy on them, and every one else for that matter, but the decryption is the bigger worry.

[Peter Lemkin]

US drug agency partners with AT&T for access to 'vast database' of call records

Hemisphere project, revealed by NYT, has AT&T employees sit alongside drug units to aid access to data in exchange for payment
James Ball

AT&T database includes every phone call which passes through the carrier's infrastructure not just those made by company's own customers. Photograph: AFP

US law enforcement officers working on anti-drugs operations have had access to a vast database of call records dating back to 1987, supplied by the phone company AT&T, the New York Times has revealed.
The project, known as Hemisphere, gives federal and local officers working on drug cases access to a database of phone metadata populated by more than four billion new call records each day.
Unlike the controversial call record accesses obtained by the NSA, the data is stored by AT&T, not the government, but officials can access individual's phone records within an hour of an administrative subpoena.
AT&T receives payment from the government in order to sit its employees alongside drug units to aid with access to the data.
The AT&T database includes every phone call which passes through the carrier's infrastructure, not just those made by AT&T customers.
Details of the program which was marked as law enforcement sensitive, but not classified were released in a series of slides to an activist, Drew Hendricks, in response to freedom of information requests, and then passed to reporters at the New York Times.
Officials were instructed to take elaborate steps to ensure the secrecy of the Hemisphere program, a task described as a "formidable challenge" in the slide deck, which detailed the steps agencies had taken to "try and keep the program under the radar".
The instructions added that the system should be used to generate leads towards new material, with call records obtained through standard subpoenas then used to provide evidence. The "protecting the program" section concluded that "all requestors are instructed to never refer to Hemisphere in any official document".
A key purpose of the Hemisphere database appears to be tracking "burner" phones used by those in the drug trade, and popularized in the long-running drama The Wire. Slides published in the Times reveal details on how Hemisphere traces "dropped" phones and "additional" phones used by law enforcement targets.
A Justice Department spokesman said in a statement given to the New York Times that "subpoenaing drug dealers' phone records is a bread-and-butter tactic in the course of criminal investigations."
He added the program "simply streamlines the process of serving the subpoena to the phone company so law enforcement can quickly keep up with drug dealers when they switch phone numbers to try to avoid detection" a similar explanation to those given by intelligence officials of the NSA's PRISM program, which grants the NSA access to information held on the servers of some of the largest internet companies.
The DoJ spokesman refused to disclose the cost of the program, telling the Times the figure was not immediately available.
Jameel Jaffer of the ACLU told the Times the program raised "profound privacy concerns".
"I'd speculate that one reason for the secrecy of the program is that it would be very hard to justify it to the public or the courts," he said.
Separately, the Washington Post published fresh details from the US National Intelligence Budget request for 2013, released by the former contractor Edward Snowden.
The document details a multi-million dollar program to prevent "insider threats" from intelligence officers, with plans to launch more than 4,000 investigations into unusual staff activity at the agencies, including downloading large numbers of documents or accessing material which they would not need in the normal course of their duties.
Such steps were in motion before Snowden disclosed any material to journalistic organisations, or the publication of such details in the Guardian and Washington Post. The NSA has yet to establish what material was taken by Snowden, according to press reports.
An NSA spokeswoman told the Post contractors were not included among the 4,000 planned investigations of security clearances. More than 500,000 US contractors hold top-secret clearances.

[Peter Lemkin]

Drug Agents Use Vast Phone Trove, Eclipsing N.S.A.'s

By SCOTT SHANE and COLIN MOYNIHAN

Published: September 1, 2013

For at least six years, law enforcement officials working on a counternarcotics program have had routine access, using subpoenas, to an enormous AT&T database that contains the records of decades of Americans' phone calls parallel to but covering a far longer time than the National Security Agency's hotly disputed collection of phone call logs.

A New York training site for the High Intensity Drug Trafficking Area program, which includes federal and local investigators. AT&T employees are embedded in the program in three states.

DOCUMENT: Synopsis of the Hemisphere Project

The Hemisphere Project, a partnership between federal and local drug officials and AT&T that has not previously been reported, involves an extremely close association between the government and the telecommunications giant.
The government pays AT&T to place its employees in drug-fighting units around the country. Those employees sit alongside Drug Enforcement Administration agents and local detectives and supply them with the phone data from as far back as 1987.
The project comes to light at a time of vigorous public debate over the proper limits on government surveillance and on the relationship between government agencies and communications companies. It offers the most significant look to date at the use of such large-scale data for law enforcement, rather than for national security.
The scale and longevity of the data storage appears to be unmatched by other government programs, including the N.S.A.'s gathering of phone call logs under the Patriot Act. The N.S.A. stores the data for nearly all calls in the United States, including phone numbers and time and duration of calls, for five years.
Hemisphere covers every call that passes through an AT&T switch not just those made by AT&T customers and includes calls dating back 26 years, according to Hemisphere training slides bearing the logo of the White House Office of National Drug Control Policy. Some four billion call records are added to the database every day, the slides say; technical specialists say a single call may generate more than one record. Unlike the N.S.A. data, the Hemisphere data includes information on the locations of callers.
The slides were given to The New York Times by Drew Hendricks, a peace activist in Port Hadlock, Wash. He said he had received the PowerPoint presentation, which is unclassified but marked "Law enforcement sensitive," in response to a series of public information requests to West Coast police agencies.
The program was started in 2007, according to the slides, and has been carried out in great secrecy.
"All requestors are instructed to never refer to Hemisphere in any official document," one slide says. A search of the Nexis database found no reference to the program in news reports or Congressional hearings.
The Obama administration acknowledged the extraordinary scale of the Hemisphere database and the unusual embedding of AT&T employees in government drug units in three states.
But they said the project, which has proved especially useful in finding criminals who discard cellphones frequently to thwart government tracking, employed routine investigative procedures used in criminal cases for decades and posed no novel privacy issues.
Crucially, they said, the phone data is stored by AT&T, and not by the government as in the N.S.A. program. It is queried for phone numbers of interest mainly using what are called "administrative subpoenas," those issued not by a grand jury or a judge but by a federal agency, in this case the D.E.A.
Brian Fallon, a Justice Department spokesman, said in a statement that "subpoenaing drug dealers' phone records is a bread-and-butter tactic in the course of criminal investigations."
Mr. Fallon said that "the records are maintained at all times by the phone company, not the government," and that Hemisphere "simply streamlines the process of serving the subpoena to the phone company so law enforcement can quickly keep up with drug dealers when they switch phone numbers to try to avoid detection."
He said that the program was paid for by the D.E.A. and the White House drug policy office but that the cost was not immediately available.
Officials said four AT&T employees are now working in what is called the High Intensity Drug Trafficking Area program, which brings together D.E.A. and local investigators two in the program's Atlanta office and one each in Houston and Los Angeles.
Daniel C. Richman, a law professor at Columbia, said he sympathized with the government's argument that it needs such voluminous data to catch criminals in the era of disposable cellphones.
"Is this a massive change in the way the government operates? No," said Mr. Richman, who worked as a federal drug prosecutor in Manhattan in the early 1990s. "Actually you could say that it's a desperate effort by the government to catch up."
But Mr. Richman said the program at least touched on an unresolved Fourth Amendment question: whether mere government possession of huge amounts of private data, rather than its actual use, may trespass on the amendment's requirement that searches be "reasonable." Even though the data resides with AT&T, the deep interest and involvement of the government in its storage may raise constitutional issues, he said.
Jameel Jaffer, deputy legal director of the American Civil Liberties Union, said the 27-slide PowerPoint presentation, evidently updated this year to train AT&T employees for the program, "certainly raises profound privacy concerns."
"I'd speculate that one reason for the secrecy of the program is that it would be very hard to justify it to the public or the courts," he said.
Mr. Jaffer said that while the database remained in AT&T's possession, "the integration of government agents into the process means there are serious Fourth Amendment concerns."
Mr. Hendricks filed the public records requests while assisting other activists who have filed a federal lawsuit saying that a civilian intelligence analyst at an Army base near Tacoma infiltrated and spied on antiwar groups. (Federal officials confirmed that the slides are authentic.)
Mark A. Siegel, a spokesman for AT&T, declined to answer more than a dozen detailed questions, including ones about what percentage of phone calls made in the United States were covered by Hemisphere, the size of the Hemisphere database, whether the AT&T employees working on Hemisphere had security clearances and whether the company has conducted any legal review of the program
"While we cannot comment on any particular matter, we, like all other companies, must respond to valid subpoenas issued by law enforcement," Mr. Siegel wrote in an e-mail.
Representatives from Verizon, Sprint and T-Mobile all declined to comment on Sunday in response to questions about whether their companies were aware of Hemisphere or participated in that program or similar ones. A federal law enforcement official said that the Hemisphere Project was "singular" and that he knew of no comparable program involving other phone companies.
The PowerPoint slides outline several "success stories" highlighting the program's achievements and showing that it is used in investigating a range of crimes, not just drug violations. The slides emphasize the program's value in tracing suspects who use replacement phones, sometimes called "burner" phones, who switch phone numbers or who are otherwise difficult to locate or identify.
In March 2013, for instance, Hemisphere found the new phone number and location of a man who impersonated a general at a San Diego Navy base and then ran over a Navy intelligence agent. A month earlier the program helped catch a South Carolina woman who had made a series of bomb threats.
And in Seattle in 2011, the document says, Hemisphere tracked drug dealers who were rotating prepaid phones, leading to the seizure of 136 kilos of cocaine and $2.2 million.