Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
My Kind of Search Engine! Finds Declassified Documents
#1

A Search Engine To Find Secrets The Government Doesn't Want You To Know

By Co.Exist / January 1st, 2014


[Image: 3022023-inline-screen-shot-2013-11-20-at...200&crop=1]






Go ahead, use the search engine. America's public servants can't very well object to citizens spying on them if they have nothing to hide. Besides, President G.W. Bush made it abundantly clear that this country will not abide terrorism. As the Washington Post reported on August 25, 2010, "America has long been an exporter of terrorism, according to a secret CIA analysis released Wednesday by the Web site WikiLeaks. …" Fight terrorism by using the search engine. Do your patriotic duty root out the terrorists operating at the highest levels of the US government. Nest week, I'll see if I can't scare up software for tapping their cell phones and laptops … AC
Screenshot: "The Sphere of Influence" is an interactive visualization of State Department cables, mapping where government secrets travel around the globe and helping to flag anomalies.

On Google, you can find specialized tabs that only search all sorts of specific results, like news articles, images, patents, and scientific studies. The Declassification Engine is an ambitious project that aims to help citizens search for and uncover one very particular sort of result: U.S. government secrets.

The project, based out of Columbia University and launched about a year ago, uses advanced computer science methods in big data, machine learning, and natural language processing to scale up what some scholars have been struggling to do by hand for a decade: Document the rise of government secrecy, learn more about what the government isn't releasing, and uncover new patterns and information in the millions of documents that do get declassified but contain heavy redactions.
Without more accountability, the historians, statisticians, computer scientists, and lawyers involved in the Declassification Engine project fear that our past will be "shredded in secrecy."
"People have always complained about official secrecy, but over time there's been measurable growth in the government secrets that are created," says Columbia University historian Matthew Connelly, the co-leader of the project. "The whole system is breaking down."

Today, to actually review the classified documents being created every year would demand the full-time work of every single federal and state government employee in the country, the project's site says. The National Declassification Center has a current and rather hopeless backlog of 370 million pages of documents, and the government spends $10 billion protecting its secrets. It allocates only $50 million to declassification work.

[Image: 3022023-inline-screen-shot-2013-11-20-at-35519-pm.png]
The Redaction Archive is another tool that is turning up matches of redacted and unredacted documents side-by-side to uncover what's beneath the black marker.The tools that the Declassification Engine have created thus far offer a glimpse into why the overabundance of secrecy hurts American democracy.
Their first mission was to gather as many declassified documents as possible into one database. The National Archives and other government troves are one source of information. But researchers have also collected others, such as scanned and full-text documents from private database companies like ProQuest and Gale Cengage Learning. Researchers involved in the project are now working with the Internet Archive to analyze the millions of PDFs that group has scraped from government sites since 1996. It also hopes to incorporate the results of FOIA requests, which are housed in online reading rooms of government agencies.
From this work, programmers and scientists are now starting to create apps, tools, and visualizations that will help others do analysis and searches.
Often, for example, the government posts declassified documents in different places and at different timeswhich means the redactions can differ. The Redaction Archive is turning up matches of redacted and unredacted documents side-by-side to uncover where they differ. This reveals the unknown text (like one sentence redacted from a 1969 memo from Henry Kissinger to Nixon which read: "In Israel, preasures [sic] will increase to deploy strategic missiles and nuclear weapons."), and it will also help academics to do large-scale pattern analysis of the "logic" of redaction. A "Redactometer" under development will measure the number of redactions in documents being released, to try to provide a degree of accountability to the process.
"The Sphere of Influence" is a massive visualization of the State Department's early electronic diplomatic cables from 1973 to 1976 (the government hasn't yet released subsequent years). It seeks spatial patterns in the million declassified cables and also at the metadata, such as the "to" and "from" fields and certain topic words, from still-classified cables.
Secrets are the coin of the realm of government. It's what people trade to get what they need.One area of development is an accurate model that will try to predict which embassy a cable came from based on the language and topics used. "What's interesting is the 2%when it can't accurately classify a cable. It means the cable is off-topic. What's interesting about these off-topic cables is they tend to be classified a secret," Connolly says. "Someone who is dealing with a million cables and they don't even know where to beginit's a way to start."
Already, there have been new insights from these tools. The research found that cables with the word "Boulder" in the subject or file name were 130 times more likely to be withheld from the public's eye. Connolly found out this term referred to "Operation Boulder," a little known program that existed in the 1970s that subjected Arabic visa applicants to an FBI investigation and that the Bush administration went to greater lengths to keep secret after 9/11. After learning about Operation Boulder through the Declassification Engine project, historians have delved in deeper and produced an exhibitabout the secret program.
The Declassification Engine project, which is really just getting started, aims to help both researchers, journalists, and citizens, but also hopes to help the government itself prioritize and speed-up its declassification backlog. An advisory board has been set up to make sure the "big data" processing proceeds with caution and doesn't tread too far into the realm of secrets that, for valid reasons, should be kept as such.
"Secrets are the coin of the realm of government. It's what people trade to get what they need, whether that's access or other information. Because there's not a penalty to over classification, you get this built-in inflationary pressure where the currency of classification gets debased," says Connelly. Releasing this pressure is clearly one motivation for the Bradley Mannings and Edward Snowdens of the world, he notes. But the Declassification Engine and other forms of more measured accountability could work, too.
http://www.fastcoexist.com/3022023/a-sea...ou-to-know
"Let me issue and control a nation's money and I care not who writes the laws. - Mayer Rothschild
"Civil disobedience is not our problem. Our problem is civil obedience! People are obedient in the face of poverty, starvation, stupidity, war, and cruelty. Our problem is that grand thieves are running the country. That's our problem!" - Howard Zinn
"If there is no struggle there is no progress. Power concedes nothing without a demand. It never did and never will" - Frederick Douglass
Reply
#2
This project is only beginning...but could turn up some interesting things.....e.g. sorry about the order of these...:p


Attached Files
.jpg   Diem Assassination.jpg (Size: 14.2 KB / Downloads: 5)
.jpg   McNamara on Vietnam Coup.jpg (Size: 18.17 KB / Downloads: 5)
.jpg   Diem.jpg (Size: 13.27 KB / Downloads: 3)
.jpg   McLamara.jpg (Size: 17.39 KB / Downloads: 2)
"Let me issue and control a nation's money and I care not who writes the laws. - Mayer Rothschild
"Civil disobedience is not our problem. Our problem is civil obedience! People are obedient in the face of poverty, starvation, stupidity, war, and cruelty. Our problem is that grand thieves are running the country. That's our problem!" - Howard Zinn
"If there is no struggle there is no progress. Power concedes nothing without a demand. It never did and never will" - Frederick Douglass
Reply
#3
An interesting find.
The shadow is a moral problem that challenges the whole ego-personality, for no one can become conscious of the shadow without considerable moral effort. To become conscious of it involves recognizing the dark aspects of the personality as present and real. This act is the essential condition for any kind of self-knowledge.
Carl Jung - Aion (1951). CW 9, Part II: P.14
Reply
#4
David Guyatt Wrote:An interesting find.

Now, if they could be convinced to use their software on the JFK Assassination documents...and other deep political events.....
"Let me issue and control a nation's money and I care not who writes the laws. - Mayer Rothschild
"Civil disobedience is not our problem. Our problem is civil obedience! People are obedient in the face of poverty, starvation, stupidity, war, and cruelty. Our problem is that grand thieves are running the country. That's our problem!" - Howard Zinn
"If there is no struggle there is no progress. Power concedes nothing without a demand. It never did and never will" - Frederick Douglass
Reply
#5

Brief:


[Image: Declassify_Infographic_Draft_4_Resized.png%22]The enormous growth in the number of official documents - many of them withheld from scholars and journalists even decades later - has raised serious concerns about whether traditional research methods are adequate for ensuring government accountability. But the millions of documents that have been released, often in digital form, also create opportunities to useNatural Language Processing (NLP) and statistical/machine learning to explore the historical record in very new ways.
Historians, journalists, legal scholars, statisticians, and computer scientists are joining together to determine whether novel statistical/machine learning methodologies can accelerate the declassification process, or at least help illuminate the broad patterns of official secrecy. Challenges we will consider include:

[LIST=|INDENT=1]
[*=2]Attributing authorship to anonymous documents
[*=2]Characterizing attributes of redacted text
[*=2]Modeling spatial and temporal patterns of diplomatic communications
[/LIST]
The featured projects indicate some of the preliminary work we have done. More fully-developed versions will be made available to the public as they become ready. The long-range goal is to create a cloud-based virtual archive. It would aggregate the digitized documents now scattered across dozens of different repositories, offer a place for scholars and journalists to upload their own archival finds, and provide a range of visualization and attribution tools to advance research on the history, and future, of world politics.
"Let me issue and control a nation's money and I care not who writes the laws. - Mayer Rothschild
"Civil disobedience is not our problem. Our problem is civil obedience! People are obedient in the face of poverty, starvation, stupidity, war, and cruelty. Our problem is that grand thieves are running the country. That's our problem!" - Howard Zinn
"If there is no struggle there is no progress. Power concedes nothing without a demand. It never did and never will" - Frederick Douglass
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  The best search engines for deep politics research David Guyatt 7 19,940 23-01-2016, 08:41 AM
Last Post: David Guyatt
  PRISM/NSA/Etc. -Free Search Engine?.....maybe...they seem to be trying. Peter Lemkin 4 15,593 22-05-2014, 01:21 AM
Last Post: Magda Hassan
  Wikileaks and clones. Other ways and places to leak documents, films etc Magda Hassan 18 29,214 04-01-2013, 03:40 PM
Last Post: Magda Hassan
  JFK Assassination Documents;; Collections from the Investigations Bernice Moore 0 3,776 08-03-2012, 03:08 PM
Last Post: Bernice Moore
  Search Digital Archives JFK Library. Bernice Moore 0 3,486 08-03-2012, 05:55 AM
Last Post: Bernice Moore
  Berlin Crisis CIA Newly Declassified Documents and Free CIA Multimedia DVD-ROM Available Bernice Moore 0 3,608 08-11-2011, 05:31 AM
Last Post: Bernice Moore
  Procedure for getting documents declassified from US State Dept. Magda Hassan 0 4,115 09-10-2011, 05:23 AM
Last Post: Magda Hassan
  Office of Historical Documents Bernice Moore 0 4,298 16-07-2011, 02:55 PM
Last Post: Bernice Moore
  57 documents at cryptome Ed Jewett 2 4,640 04-06-2011, 04:13 AM
Last Post: Ed Jewett
  Douglas Valentine Vietnam Phoenix Program research documents available Magda Hassan 0 4,945 02-12-2010, 11:26 PM
Last Post: Magda Hassan

Forum Jump:


Users browsing this thread: 1 Guest(s)