The FBI Takes Seven Years to Develop a Top Secret FriendFeed

I was paging through a pretty well written article over at Ars Technica regarding the bumbling results of a seven year long project at the FBI to create what’s being dubbed an “everything bucket” to warehouse the data they collect on an ongoing basis.

The description will make you cringe, but go ahead and read it so the rest of what I say will make sense to you:

Earlier this week, the EFF published a new report detailing the FBI's Investigative Data Warehouse, which appears to be something like a combination of Google and a university's slightly out-of-date custom card catalog with a front-end written for Windows 2000 that uses cartoon icons that some work-study student made in Microsoft Paint. I guess I'm supposed to fear the IDW as an invasion of privacy, and indeed I do, but given the report's description of it and my experiences with the internal-facing software products of large, sprawling, unaccountable bureaucracies, I mostly just fear for our collective safety.

The idea behind the system, which the FBI has been working on since at least 2002, is that the Bureau can dump all of its information in there so that it can be easily searched and shared. IDW contains more documents than the library of congress—a stew of TIFFs with OCRed text, multiple Oracle databases, news streamed in from the Internet, reports and records in various in-house data formats, watch lists, telephone data, and an alphabet soup of smaller databases and records repositories—all accessible as one sprawling system that processes batch jobs, runs queries, and issues alerts. In short, the IDW is an "everything bucket" for the FBI.

Tell me if you caught the same thing I caught (it shouldn’t be hard, since I used italics and bold). This IDW thing is basically a lifestreaming services for the whole country?

The data collected is supposedly more plentiful than the library of congress, and contains all manner of inane data, sometimes substantive and relevant discussion, and news streamed in from the Internet.

Sounds a bit like FriendFeed or the Facebook news feed (only, y’know, searchable).

Took them seven years to achieve that? Really?

I do sort of want to take a peek at the system, now, based on that revelation. Based on what the EFF has said, though, that ain’t gonna happen.

(Incidentally, I can almost guarantee that while it is probably an inferior product to FriendFeed, the color scheme is almost certainly superior).