Dangerous Government Computer Applications Part II

Special Agent Jaime Zapata

On the afternoon of February 15, 2011,Jaime Zapata, a 32-year-old special agent with US Immigration and Customs Enforcement, was shot to death by members of a drug cartel as he drove along a four-lane highway in Mexico. Zapata’s partner, 38-year-old Victor Avila, who survived the attack, later said that as many as 15 gunmen opened fire with automatic rifles, even though Zapata had identified himself as a diplomat and the armored SUV in which the pair was riding bore license plates identifying it as an official vehicle.

Back in Washington, senior administration officials wanted revenge for what they saw as the deliberate killing of a federal agent and the attempted murder of another. Michele Leonhart, head of the Drug Enforcement Administration and a veteran of the drug wars, asked, “What can we do to make an immediate impact against the cartels, to send a message?” recalls Derek Maltz, the special agent in charge of the DEA’s Special Operations Division. “We decided to do a law-enforcement strike,” Maltz says.
To conduct what became known as Operation Fallen Hero, investigators turned to a little-known Silicon Valley software company called Palantir Technologies. Palantir’s expertise is in finding connections among people, places, and events in large repositories of electronic data. Federal agents had amassed a trove of reporting on the drug cartels, their members, their funding mechanisms and smuggling routes. They had dossiers, reports from informants, surveillance images, intercepted electronic communications, footage from drone aircraft. But investigators lacked a way to assemble and share all that intelligence with one another and to quickly find leads buried in mountains of information.

Investigators with Zapata’s agency bought Palantir’s software, plugged it into their databases, and used it to track down members of the cartel. The results were astonishing.

Palantir helped identify connections among key individuals and organizations. Officials later reported that this kind of painstaking detective work—reading reports, piecing together clues, drawing links among people—would have taken months without technological assistance. With the help of Palantir, large amounts of data from disparate sources were analyzed within days.

Law-enforcement officers across the United States, Mexico, and South America confiscated 467 kilograms of cocaine, 64 pounds of methamphetamine, and 282 weapons and arrested 676 people—including the cartel member suspected of killing Zapata.

Officials were so impressed with Palantir’s software that seven months later they bought licenses for 1,150 investigators and analysts across the country. The total price, including training, was $7.5 million a year. The government chose not to seek a bid from some of Palantir’s competitors because, officials said, analysts had already tried three products and each “failed to provide the necessary comprehensive solution on missions where our agents risk life and limb.”

As far as Washington was concerned, only Palantir would do.

Such an endorsement would be remarkable if it were unique. But over the past three years, Palantir, whose Washington office in Tysons Corner is just six miles from the CIA’s headquarters, has become a darling of the US law-enforcement and national-security establishment. Other agencies now use Palantir for some variation on the challenge that bedeviled analysts in Operation Fallen Hero—how to organize and catalog intimidating amounts of data and then find meaningful insights that humans alone usually can’t.

Palantir has sold its software to the CIA, the military’s Special Command, and the Marine Corps, which use it to help track down terrorists. The FBI, the Defense Intelligence Agency, the National Counterterrorism Center, and the Department of Homeland Security are customers. The director of the National Security Agency has said Palantir’s software could help the agency “see” into cyberspace to defend against hackers and spies attempting to breach government computer networks. The board that’s set up to oversee federal stimulus spending uses the software to spot fraud. The Los Angeles Police Department uses Palantir. So does the New York Police Department, whose intelligence-and-counterterrorism unit rivals the sophistication of the FBI and the CIA.

Palantir’s success in Washington—the likes of which is unheard of for a Silicon Valley start-up—suggests a question: If the US intelligence community had been using the company’s software before 9/11, could it have stopped the attacks?

(Watchman note: the 9/11 plot was uncovered by a small team of U.S. Army intelligence counter terrorism analysts  using a counter terrorism program called Able Danger.)

The answer is a qualified yes. Put aside all the obstacles to preemption that a computer program couldn’t have removed: There were legal restrictions on how the FBI and CIA worked cases together, regulations that limited investigators’ access to information on American citizens, and a set of bureaucratic incentives that favored territoriality over cooperation. But if lawmakers and policymakers overcame those hurdles and if they were no longer an impediment to intra governmental harmony, then yes, a system designed to locate and then share important clues could give analysts a strong chance of spotting the nascent signals of the 9/11 plot. This may have then led investigators in the direction of some of its architects, whom they could arrest or otherwise take out of commission. If you connect the dots this way, Palantir might have helped save the day.

It’s understandable why such a barrier-busting technology wasn’t being used on September 10, 2001. But the lack of information-sharing that led to the attacks was almost immediately obvious. So why didn’t the government get to work right away on a technological solution to the problem?

There are a few answers to that, starting with this: The government did try to solve the problem, but it failed.

In January 2002, the Defense Advanced Research Projects Agency (DARPA), the military brain trust that invented the Internet and stealth technology, launched a research initiative to develop counterterrorism technologies. It was called Total Information Awareness (TIA), its goals were twofold: First, use advanced data-searching-and-integrating software to help the government get a better handle on the massive databases it had already set up but hadn’t used to their full potential. Second, use the same technology to search private data stores that might also hold valuable clues, such as airline reservation systems, credit-card records, and phone and e-mail logs.

The animating hypothesis behind TIA was that technology could help analysts identify potential terrorists amid a sea of mostly innocent people. This was an idea for which the technology didn’t exist. But such a powerful system would also pose a threat to civil liberties, so TIA would encrypt information about American citizens and place legal controls on analysts using the system.

The man who ended up in charge of the effort, retired admiral John Poindexter, had been thinking about the problem since the early 1980s, when he was on the staff of the White House National Security Council. He had been working for nearly two decades under the presumption that computers could assist counterterrorism analysts.

TIA was shuttered in 2003 amid a national debate over its goals and its director’s past—Poindexter was an architect of the Iran-Contra affair and prosecuted for his involvement. But his resignation didn’t diminish the government’s appetite for his ideas. So it should come as no surprise that Palantir’s founders sought Poindexter’s counsel.

One day not long after Palantir was launched in 2004, Poindexter received a call from his friend Richard Perle, ex-chair of the Defense Policy Board, asking if he’d come to Perle’s house for a meeting with two entrepreneurs he knew from Palo Alto—Alex Karp, Palantir’s CEO, and his fellow cofounder Peter Thiel, a billionaire venture capitalist who’d helped start PayPal and was an early investor in Facebook.

Karp, a self-described progressive, knew Thiel, a prominent libertarian, from their days at Stanford Law School. After 9/11, Karp had reconnected with Thiel, who had the idea that Silicon Valley should do something to improve national security and secure civil liberties. They and three other Palantir cofounders wanted to use PayPal’s fraud-detection technology as the model for a new counterterrorism software. Karp wasn’t a natural choice for CEO—although he had spent most of his career helping other people start companies, he hadn’t run them. But the partners revered him for his intellect and business savvy.

The parallels between moving money and fighting al-Qaeda may not be immediately obvious. PayPal, founded in 1998, became famous for letting people electronically pay for goods and services they bought online. But that’s not why it became a successful business. First, engineers had to figure out a way to keep criminals from stealing PayPal customers’ money.

Credit-card companies and banks had written algorithms to detect patterns of fraud and theft, but the algorithms became useless as soon as criminals figured out what triggered an alarm and then adjusted their tactics. PayPal wanted to pit fraud experts against the scammers—an electronic version of cops and robbers. Engineers designed a software program that let human fraud experts—many were former law-enforcement officers—quickly sift through transaction data and start asking questions. Did a transaction occur between two parties in the US? That was probably safe. Two parties in Russia? A bit more suspicious. Did someone in the United States transfer money to someone in Russia? A red flag.

By giving the analysts a way to look into the transaction network and map out connections among the suspected criminals, they could draw conclusions based on their years of expertise. That approach made PayPal the world’s most trusted system for online payments. In 2002, the company was acquired by eBay for $1.5 billion.

“The bread and butter of PayPal was to look at a transaction and to know if it came from a bad IP address,” says Bob McGrew, Palantir’s director of engineering, referring to a computer’s distinct address on the Internet. That same approach became the heart of Palantir, which Karp says is an “attribution” software, meaning it’s used to find the people behind pieces of data.

Palantir wanted to beat terrorists the way PayPal beat the Russian mafia. And that approach bore some striking similarities to the one Poindexter took with TIA.

Thiel, who’s something of a policy dilettante, also thought the idea could make a lot of money. He has said that Palantir will be the next Facebook; a top executive at JPMorgan Chase says it’s poised to do for information locked inside organizations what Google did for information on the Web. Palantir is considered one of the most valuable start-ups in America, with an estimated market capitalization as high as $2.5 billion.

“I told them I thought they had an interesting idea,” Poindexter says. He looked at an early mockup of the product and thought it had “an absolutely beautiful user interface.”

With little training, Palantir gives users the ability to turn text documents into charts and graphs and to layer intelligence reports onto maps or into interactive timelines.

Karp says Poindexter was one of many experts whom Palantir’s founders consulted in their early days. Many of them opened doors that usually remain closed to small companies with no experience in Washington.

In a short time, Palantir has assembled a legion of advocates from the most influential strata of government. Karp counts former CIA director George Tenet as a friend as well as Tenet’s employer, Herb Allen, who runs the enigmatic investment bank Allen & Co., a Palantir investor. Michael Leiter, former National Counterterrorism Center director, is a senior counselor to Palantir. Another top adviser, Bryan Cunningham, was a CIA intelligence officer and a senior staffer to former national-security adviser Condoleezza Rice.
Palantir has hired powerful lobbyists, including onetime senator John Breaux and ex–Senate majority leader Trent Lott, both of whom now work for Patton Boggs. And Palantir’s advocates in Congress include representatives Buck McKeon and Adam Smith, respectively the chairman and ranking member of the House Armed Services Committee, as well as Norm Dicks, the ranking member of the House Appropriations Committee, all of whom have urged the Defense Department to consider buying more of Palantir’s software.

“They provide a product that actually does what they say it can do, and at the price and on the schedule they promise,” Leiter says. “Regrettably, that is often rare among many government contractors.” This fact also helps explain how a start-up launched by five Washington outsiders found such remarkable success here. And it provides another explanation for why the government didn’t build something like Palantir immediately after 9/11. The fact is, the way Washington works made a Palantir-like success practically impossible. So Palantir decided not to play by Washington’s rules.

Big companies have dominated defense contracting for more than 60 years, but Palantir is an insurgent. Its founders don’t obey the hierarchical order of the defense-industrial complex, which they think has failed the government and ripped off taxpayers.

They have a point. The files of inspectors general overflow with accounts of how big-name companies over-promised and under-delivered on multibillion-dollar technology “modernization” projects meant to cure the intelligence failures that led to the 9/11 attacks. The FBI and the National Security Agency, both of which wasted billions on custom-built computer systems, offer some of the most dramatic examples of high-priced failure. But the IRS, the Interior Department, and the Pentagon, among others, can claim a place in that undistinguished company.

During a speech in London, where geographic and cultural distance may have put Karp in a more candid mood, he gave a scathing assessment of how big defense contractors—his competition—traditionally do business in Washington.

“People charge you every time you want something fixed, every time you want to build something, before it’s built, after it’s built,” he told the London crowd. “You’re paying by the hour. You’re paying 20 people. You think you’ve hired five.”

Palantir wanted to upend the system. But before Palantir could get to the point that it could offend Washington’s sensibilities, it had to get out of Silicon Valley.

Prospective investors were skeptical. Venture-capital firms were looking for the next Facebook or Google, but most presumed it would come in the form of a consumer technology, probably a new social-media Web site. No one wanted to back an expensive software platform for large organizations. The Palantir founders met with several venture-capital firms, and all turned them down. “We believed it would work,” Karp says. “No one else did.”

Potential investors were also leery of Washington. “The government was unpopular in Silicon Valley,” Karp says.

“Everyone told us we should stay away from Washington because it was corrupt and we didn’t know how to play golf with senators,” says Joe Lonsdale, another Palantir cofounder and former editor-in-chief of the Stanford Review, the libertarian publication Thiel founded as an undergraduate. He recalls that one prospective backer ridiculed Karp for having a PhD in philosophy, not a technical field. Says Lonsdale: “It was kind of a horrible experience.”

According to Karp, at the end of one failed pitch meeting, an investor who’d just turned him down said there was a group he should talk to “that does this kind of thing.” The group was In-Q-Tel, the venture-capital arm of the CIA, which was set up in 1999 to bypass the cumbersome government procurement process and to fund technologies that might be useful to the spy agencies.

In-Q-Tel is a gatekeeper between small technology companies, which usually know Washington only by its location on a map, and the secretive regions of the intelligence community. “If you’re crazy enough to want to work with government, they’re the right people to talk to,” Stephen Cohen, one of the cofounders, told an audience at the Startup School conference held at Stanford in October. Palantir got a meeting with In-Q-Tel’s CEO, Gilman Louie, a former computer-game designer.

Cohen was 22 at the time, but had been writing software code since he was a teenager. He spent the next eight weeks with another cofounder hammering out a version of Palantir they could take to the meeting. They worked—and slept—in an office Thiel had used when he founded PayPal.

“Like anytime you’re putting together a prototype really fast,” Cohen said at the conference, “you’ve got to focus a lot on using the fireworks to cover up for the broad strokes on the back end, the pieces that are going to get there eventually.”

Louie was impressed. After the demonstration finished, he told Cohen, “If you guys aren’t able to help us with counterterrorism, you have a real future in video games.”

In-Q-Tel invested a relatively insignificant amount of money—reportedly about $2 million, a small chunk of the nearly $40 million that Karp says Palantir’s investors spent before the company saw its first dollar in revenue. But more crucially than the money itself, In-Q-Tel put Palantir’s founders in the room with frontline intelligence analysts, the people they hoped would one day use their product. The analysts gave Palantir the software equivalent of a test drive. “They’d say, ‘I love that, I hate that,’ ” Karp says.

“We would watch to see how they interacted with it,” Cohen said. “We wanted to know not just what the analysts asked for but really what was happening between the lines.”

During his conference talk, Cohen recalled that at one meeting, his government host introduced him as Mr. Two Weeks. “He can give you anything you want in two weeks,” the host told the assembled analysts. “So ask away.”

The founders spent the next three years flying to Washington, taking notes, and then returning to Palo Alto to tweak the software. Cohen said he was getting “most of my calories from Red Bull.”

Karp estimates that he and Cohen had more than 300 meetings with likely users, people far down in the government’s hierarchy. The Silicon Valley techies found themselves deep into an all-new culture: They encountered some people who introduced themselves only by their first names and refused to say where they worked in the government.

While shuttling between the coasts, Palantir’s founders discovered that intelligence analysts wanted a way to search their own databases and to know what their colleagues in other agencies had available. But just as important, agencies needed to restrict access, so that only those with the proper security clearances could, for instance, look at the video of a drone attack or read a classified interrogation summary. Palantir developed a method for indexing information so the system would match up a particular data point with the user’s security clearances. If he didn’t have the authority to read it, the information was off-limits. This technique had the added benefit of creating an audit trail of what the Palantir users were reading, whether they’d handled the information properly, and whether they’d modified it in any way.

It’s difficult to overstate the importance of this security regime. Absent such nuanced controls, down to the level of a single person or one nugget of intelligence, the kind of collaboration necessary to prevent terrorist attacks won’t happen. An audit trail like this also lets analysts check their own prior judgments to see if there was a flaw in their logic.

Before it landed a customer, Palantir was given a rare audition with the agency that knew better than any other the dangers of misguided analysis. According to a government official familiar with the episode, the CIA allowed Palantir to set up its software in the agency’s counterterrorism center, the hub of its global campaign to track down terrorists. The official was astounded that a little-known company from Silicon Valley was allowed to place its equipment on a network that pulses with some of the most highly classified intelligence the government collects. The CIA let Palantir use some of that intelligence to show off its software, the official says, an extraordinary departure from normal security protocols.

Palantir didn’t disappoint. The official says the company worked for several months without pay and convinced the CIA that its technology could do what it claimed. Yet heading into mid-2008, Palantir still hadn’t won a government contract or earned a dollar of revenue. The small number of investors the company had managed to attract were getting impatient. Karp had already delayed the release of the software by a year because he felt it wasn’t ready. “I’m not motivated by money,” Karp says. “It’s not what gets me out of bed in the morning.”

Patience paid off. At yet another analyst meeting, Cohen showed a group of more senior government officials what enhancements the engineering team had made. Out of the corner of his eye, Cohen said, he saw two stoic men in gray flannel suits, what he called “the cliché of government guys,” turn to each other and, without speaking a word, give each other high-fives.

Back in Palo Alto, Palantir moved into a 7,000-square-foot office. The founders were about to sign their first contract with a government agency. Karp won’t give the name, and while sources’ accounts conflict, it was likely the CIA or a Defense Department group set up to fight improvised explosive devices and bomb makers.

Before closing the deal, Palantir’s new government customers wanted to see the office. Cohen said he went to Ikea and bought as many desks as employees could assemble in 24 hours. Realizing they’d just filled up a big office with empty desks, they went to an electronics store and bought computers. All that was missing now were the employees—the founders still had hired only a handful of full-time staff.

Cohen, who said that the only time he’s awake early in the morning is when he’s been up all night, invited the government officials to come to the office at 9 am. When they looked around and asked about all the unoccupied desks, “we explained the obvious,” Cohen said. “Silicon Valley isn’t awake at 9 am.”

Palantir began recruiting top students from Stanford and other elite computer-science schools. It offered the typical tech-employee perks, including free dry cleaning and three meals a day. On the company intranet, Karp sent out motivational videos that instructed employees how to talk about Palantir with customers. Employees nicknamed it KarpTube.

“The office was like a fraternity for very smart people,” says Tim Su, who worked for two years as a software engineer. “We ate together, played video games together, and spent a lot of time at work together. Every Friday night, we’d hang out at the office and party together.”

The cultures of Washington and Palo Alto found enough common ground that Palantir’s single contract turned into a second and then a third.

“We came to market with an anti-Washington strategy,” Karp says. Traditionally, a company wanting to break into the government market would start at the top of an organization, trying to win friends and influence senior decision-makers who have the official say on what to buy. But Palantir’s early business was based on word of mouth from those early test users—Karp calls it “a rumor mill of people who’d worked with the product”—who called friends in other agencies and urged them to buy it.

Rod Rhines, a former Navy SEAL and CIA officer, recalls that a colleague in the military phoned to tell him, “Palantir is a game-changer for us.” Rhines now helps manage Palantir’s business with the Defense Department.

“They got in at the tactical level,” says a former military intelligence officer who now works for a large defense contractor and has used Palantir’s software. Forces on the frontline got hooked on the product and then demanded that their bosses buy it. They also told intelligence organizations high up the chain to send them intelligence reports in a format compatible with Palantir. These early battlefield adopters were essentially on the same rung of the hierarchy as the intelligence analysts who gave Palantir its early tests.

As the word spread, Palantir began making money. In fiscal year 2008, it booked nearly $1.2 million in sales through government contracts called schedules, which let agencies buy goods and services without having to go through a lengthy contracting process. The next fiscal year, Palantir’s schedule sales ballooned to $5.8 million. The year after that, they hit $7.4 million. These figures don’t account for sales using contracts other than the schedule.

Beyond government, Palantir’s business now includes some of the country’s most prominent banks and financial institutions, such as JP Morgan Chase and the hedge-fund manager Bridgewater. Palantir is now moving into health care, helping to spot fraud and inefficient spending. In 2011, including all sales to government and private-sector clients, Palantir is estimated to have earned just over $250 million.

Palantir’s success was built on a foundation in Washington, and its bottom-up insurgent maneuver left larger defense contractors bewildered—even offended.

The former military intelligence officer says that when Palantir employees came to his office to discuss how they might work together, they showed up in jeans and matching black track jackets embroidered with the Palantir logo. “I was unimpressed,” he says.

Palantir’s early success may have gone to its employees’ heads. “Their assumption was that they would sell their product to us and that they would have control over training,” the former officer says. “I was like, ‘Who do you think you are?’ All they wanted to do was to charge us money, an exorbitant amount, to teach our customers how to use their tool.”

The Palantir employees also seemed to presume that selling their software to a government agency would give them special privileges to help run that agency’s computer network. “That was completely naive,” he says. “Any government agency already has systems administrators who’ve been running their networks for years. It was like Palantir was selling a doorknob, and to make it work they wanted the government to let them build the house around it.”

Palantir’s corporate ethos sometimes feels more connected to fantasy than reality. In public remarks, several employees have said their job is to assist the people “who are out saving the Shire.” It’s an allusion to J.R.R. Tolkien’s The Lord of the Rings saga. The Shire is the home of the Hobbits, who band together with their elf, dwarf, and human compatriots to save the world from the armies of Sauron, the master of evil. The company is permeated with Tolkien references.

Its Palo Alto office is known as the Shire, and the Tysons Corner branch is Rivendell, which is the home of the elves. The company’s name itself is from Tolkien—a palantir is a magical stone that lets its holder see across great distances.  (Watchman, wow that makes Palantir sound satanic.) That the stone is also used by Sauron to conduct surveillance as he wages war against the known world isn’t mentioned in any of Palantir’s corporate literature, although a figurine of the evil wizard Saruman, Sauron’s chief flunky in the Lord of the Ring movies, sits on a windowsill in the Rivendell lobby.

These are apt allusions, because just like Tolkien’s palantir, the ends to which the Palantir software is used depends on who’s manipulating it. And that includes not just Palantir’s clients but its own employees, some of whom have embraced their increasingly powerful status as an arm of the real surveillance state.

In the fall of 2010, Palantir employees partnered with an ex–Navy intelligence analyst named Aaron Barr, the new CEO of HBGary Federal, which specialized in identifying computer viruses, on an ill-conceived project that carried the promise of big money—but also a lot of risk.

Palantir and HBGary Federal teamed up with a third intelligence contractor, Berico Technologies, to provide information on groups and individuals deemed hostile to the US Chamber of Commerce. (Watchman comment, ah the drift to illegality.) The law firm Hunton & Williams first approached Palantir about the work, which was to include reconnaissance of various Web sites and social media in order to build dossiers on the chamber’s opponents. Operating under the name Team Themis, the companies would set up an analysis cell to provide the law firm with intelligence about “adversarial entities and networks of interest,” according to a proposal the team drew up. Palantir would “serve as the foundation for all of the data collection, integration, analysis, and production efforts.”

For its work, Palantir asked to be paid $1.1 million. Anticipating that its client might balk at such a price, Matthew Steckman, a Palantir employee in the Rivendell Washington office, wrote an e-mail to his teammates urging them to emphasize, “We are the best money can buy! Dam it feels good to be a gangsta.”

A few days later, the law firm asked Team Themis whether it could offer a proposal for another job, this time targeting the anti-secrecy group WikiLeaks, which was threatening to release internal records from Bank of America. As Steckman explained to his team, the bank wanted to sue WikiLeaks and enjoin it from releasing the information.

The Justice Department, which had been looking for a way to prosecute WikiLeaks’ founder, Julian Assange, called Bank of America’s attorneys and told them to get in touch with Hunton & Williams. “Apparently, if they can show that WikiLeaks is hosting data in certain countries, it will make prosecution easier,” Steckman wrote.

Barr said that Themis should target WikiLeaks’ “global following and volunteer staff” as well as people donating money to the group. “Also need to get people to understand that if they support the organization we will come after them. Transaction records are easily identifiable.” He said they should submit fake documents to WikiLeaks and try to foment distrust among different camps of supporters. Barr also suggested they target “people like Glenn Greenwald,” the progressive blogger who was a vocal WikiLeaks supporter. And he wanted to launch “cyberattacks” on a server WikiLeaks used in Sweden in order to “get data” about people who were anonymously submitting information.

Team Themis never got the chance to carry out its campaign of espionage and propaganda. In February 2011, an article appeared in the Financial Times quoting Barr, who bragged that he’d been able to penetrate the inner ranks of another hacker-activist group, Anonymous. The group retaliated by breaking into Barr’s e-mail account and publishing years’ worth of his correspondence, which included the Team Themis proposals.

For Palantir, a company founded on the idea that technology should protect personal freedoms, it was a humiliating revelation. Palantir risked looking like a cyber mercenary. That image ran counter to the core values of the company, yet it was one of those values that allowed the Team Themis work to be considered in the first place.

Karp was apparently unaware of what his subordinates had been doing. Palantir has what Karp calls a “flat hierarchy”: Employees are encouraged to act like entrepreneurs and not to seek approval for every decision they make. Karp says this structure is essential to Palantir’s success: “No company in the Bay Area is disruptive with multiple layers of hierarchy.” That concept, of course, is anathema to Washington.

“It’s not that we give people a long leash at Palantir,” David Worn, a former intelligence analyst Karp hired to open the Tysons Corner office, explained to a technology blog in 2010. “There are no leashes here at Palantir.”

Palantir placed Steckman on leave pending a review of his action. Barr resigned from HBGary Federal. Karp ended all contacts with HBGary and issued a statement apologizing to “progressive organizations . . . for any involvement that we may have had in these matters.”

Karp says Palantir hired the law firm Boies, Schiller & Flexner to investigate the company’s role in Team Themis. It recommended that Palantir keep Steckman as an employee, which Karp says he did.

Some in the progressive circles that Karp identifies with now view the company as a sinister force that needs to be checked. What’s to stop a government intelligence agency from turning off Palantir’s privacy-protection features and using the software for illicit purposes? Karp insists that the controls are “very hard to circumvent” and that it would take a “world-class software team” to do it.

He doesn’t cite an example, but it’s hard to resist the thought that one agency Palantir employees claim hold it in high regard—the National Security Agency—would have both the skill and the motivation to modify Palantir for its own purposes. The agency employs the largest and most skilled cadre of software experts in the government. And for more than four years after the 9/11 attacks, it conducted a secret campaign of electronic surveillance against US citizens that bypassed federal courts. NSA also took over many of John Poindexter’s Total Information Awareness programs after they were officially shut down, but it rejected one: building privacy-enhancing technology into computer software. In congressional testimony, NSA’s director, General Keith Alexander, said the agency has examined Palantir’s software and that it could be useful for cybersecurity.

Over time, the cultural distinctions between employees in Palo Alto and Tysons Corner have become more pronounced, in turn affecting the company. Palantir is neither of the Valley nor entirely of the Beltway. It’s a kind of techno-military hybrid. In Tysons, there are predictable trappings of a start-up—a pool table, a fleet of Razor scooters, an amply stocked kitchen—but there are also employees wearing desert boots and customers in camouflage fatigues. The denizens of Rivendell are more likely to have come from the Korengal Valley than from Silicon Valley.

Palantir, like so many government contractors, has installed a Washington-style revolving door. It recruits employees from among the ranks of its customers. Most of its “embedded analysts,” employees who work on high-priority national-security threats, are former users from the military and intelligence community. A job description on Palantir’s Web site describes the ideal embedded analyst this way: “Although you loathe the bureaucracy, you have a deeply held belief that a revolution in intelligence affairs is not only possible, it is imminent. Help us craft that revolution.”

It’s hard to escape the suspicion that Palantir has created more of a cult than a culture. Karp—whom employees call Dr. Karp—insists they’ve built “a real culture that’s not based on money.” Palantir caps all salaries at $127,000, which is what Karp earns. Employees are compensated with bonuses and equity stakes, but most of the engineers could make much bigger salaries if they defected for Facebook or Google. If someone goes to work for Palantir, it’s probably because he or she believes in Palantir and its mission. As naive or unsettling as that mindset may seem, it’s aligned with a credo of public service found more often in the government than in its contractors. Karp is inclined to keep things that way, and despite Palantir’s obvious trajectory toward an initial public offering of stock in the near future, he says, “I don’t want an IPO. The minute you have it, people wake up and ask, ‘How rich am I?’ ”

Perhaps sensing the damage to its reputation from the Team Themis project, Palantir has resorted to more traditional tactics to strengthen its position in Washington. In 2011, it spent around $300,000 on wall-to-wall ads in Metro stations, including L’Enfant Plaza and Pentagon. This was likely the first introduction thousands of Washingtonians had to the shadowy company. Palantir employees gave more than $92,000 in campaign donations in the most recent election cycle, a record for the company.

Palantir has also stepped up its lobbying efforts. Last year, when the company was trying to gain entrée to an important Army program in Afghanistan, at least a half dozen members of Congress intervened and tried to strong-arm the military into giving Palantir a chance. The Army resisted and gave a contract to its chief competitor instead. But the fact that Palantir could so effectively persuade lawmakers shows it has learned to play hardball in Washington.

Today, some current and former government officials say Palantir’s star has dimmed in the intelligence community. They complain that the software has a hard time analyzing extremely large databases and that it takes a lot of time on the front end to arrange information in a format Palantir can use.

Still, Palantir has built what it claims, and despite its shortcomings, the technology has made significant contributions to solving some of the country’s most important national-security challenges. “The contradiction we wanted to remove was between civil liberties and fighting terrorism,” Karp said at a recent Palantir conference. “Do we really want to live in a world where everyone sees everything without any kind of permissions? Solving this problem . . . that’s a really cool idea.”

Security and liberty are competitors now. That’s not a natural condition; it’s a product of our time, of the decisions that we have all made—or failed to make—over the past decade. Could a piece of software allay that uneasy tension? Perhaps. But as any good student of Tolkien knows, whether a palantir is used for good or for evil depends on who’s holding the stone.

This article appears in the February 2012 issue of The Washingtonian.