A little LOTR is a dangerous thing

[nancylebov]

His company, named for the all-seeing stones in the Lord of the Rings series, sells software that fuses vast amounts of disparate data together and then allows it to be searched for connections. Using Palantir, intelligence analysts can look for links using phone and tax records....

As any reasonably careful reader of LOTR knows, the information you get from a palantir isn't especially reliable. There's a real chance you have much less context than you think you have, so when I hear that definition of the company, I'm going "Ow-ow-ow!".

When I hear this:

Intelligence officials have been hoping for some time that vacuuming up vast amounts of information and putting it into a computer would uncover some sort of discernable terrorist pattern. The technique, known as data mining, is controversial because information on the innocent, as well as potential terrorists, ends up in the same database. Now it is increasingly unclear whether data mining will ever really work because terrorists don't appear to have predictive patterns.

"We don't even have enough of a data set to get a good pattern of 'What does a terrorist look like?' " says Fred Cate of Indiana University's Center for Applied Cybersecurity Research. "And terrorists, of course, are constantly changing their patterns because, quite simply, they don't want to get caught."

That's why they use one-time cell phone numbers and drop-box addresses.

"There had been, over the past seven years, this sense that if you collect more and more data and put it into a powerful enough computer, shake it and bake it the right way you'll come up with the unknowns" — terrorists who aren't yet on law enforcement's radar screens — says Jim Dempsey, the executive director of the Center for Democracy and Technology, a privacy group in San Francisco.

"I think, and other people who are more technically adept than I think, that's really a fool's errand."

I feel a little better. If you're wondering why the quotes are out of order, it's because I was running the radio in the background and heard the story more than once.

Then we get to this:

Intelligence information basically comes in two forms — structured and unstructured data. Structured data have fields, like a spreadsheet. It is relatively easy to search. Unstructured data, which is the form most intelligence information comes in, are like notes in a reporter's notebook. There is a tip here, a phone call there. Palantir can search both kinds of data simultaneously. That's one of the reasons why the FBI, CIA and New York Police Department, among others, have all recently started using the software..

and I'm not feeling better any more.

At least it's possible that the credit card companies know they're making stuff up as an excuse to cancel cards.

Comments

[sinboy.livejournal.com]

I hope the Tolkien estate sues them.

[mama-hogswatch.livejournal.com]

Handmaid's tale, wheeee!!!!!!!!!!

[metahacker.livejournal.com]

They lost me at the word "search". That's not what you want to do with unstructured data. If you search, you'll find what you're searching for, not what you need to find.

Grrr.

[whswhs.livejournal.com]

Funny thing, The Wire showed that process repeatedly, with the drug dealers figuring out that they were being tapped and coming up with new codes and new ways of sending messages. You'd think the intelligence industry would be as smart as television scriptwriters.

[nancylebov.livejournal.com]

Not when something wrong and complicated feels like you're doing something useful and leads to lucrative contracts.

[avram]

Keep in mind that, no matter how smart the smartest bad guys are, there are probably some dumb one in the mix as well. There's probably some strategic value in forcing them to keep coming up with new ways around your surveillance techniques in the hope that a few of them will slip up.

[sodyera.livejournal.com]

The intelligence community is actually building its own version of one million monkeys. Once these computers compile enough data on terrorist patters they will then collectively generate a Shakespeare play. Some programmer will read it, re-brand it as his own then shop it around to Hollywood, where it'll appear as a remake of Conspiracy Theories or Enemy of the State. Wait, isn't there another Bourne movie coming out soon? Paging Mr. Damon!