Voters in America, and increasingly elsewhere too, are being ever more precisely targeted

When Tom Pitfield talks about the campaign of Justin Trudeau, who was recently elected Canada’s prime minister, he gets animated. Mr Trudeau’s Liberal Party could not afford a lot of television time and spent much of its advertising budget on social media. That proved an inspired choice. “We would create an ad, see how people reacted to it on Facebook, tweak the content and test it again. On some days we would produce more than 50 different ads,” explains Mr Pitfield, who was in charge of the campaign’s digital side. This rapid feedback, he says, allowed his team to offer much more flexible and targeted messages than the competition.

Although the trend is obscured by Donald Trump’s tweets and his other antics, Facebook will also play a big role in America’s presidential contest this year. The tools that the world’s biggest social network offers to campaigners are getting better all the time. Last year it provided a way to upload lists of people to its site so they could be sent targeted messages. Now it is offering a further service that allows campaigners to reach Facebook users who “like” and share a lot of political content.

Even Barack Obama’s first presidential campaign in 2008 was widely hailed as “data-driven”. But it was only in 2012 that his team systematically used digital technology to deal with every campaign’s biggest challenge: how to make the best use of a limited budget to reach the right voters. In the past, geography had served as a proxy target: if a precinct was considered Democratic, for instance, it would get a lot of attention from Democratic campaigners. But in recent years it has become possible to target voters individually, thanks to the availability of ever more data as well as ever cheaper computing power and better methods to mine them.

To find out where to concentrate its resources, the Obama campaign used polls and other data to generate a statistical model of the attributes potential Obama supporters had in common. “When volunteers knocked on doors in 2008, four out of ten people they met backed Obama. In 2012 the ratio was nine out of ten,” says Dan Wagner, who led the president’s data-science team during his second campaign.

Hidden persuaders

From a windowless office in Chicago that became known as the “data cave”, Mr Wagner and his colleagues also pioneered a number of other methods of persuasion in that election. They tested the subject lines of fundraising e-mails (“I will be outspent” raised $2.6m; “Do this for Michelle” only about $700,000). They found out whether a group of voters they wanted to target watched certain cable shows, which allowed them to use television advertising more cost-effectively. Beyond mobilising their own voters, they also tried to identify others who might be persuaded to change their mind (with limited success).

Such novel approaches helped scupper the campaign of Mitt Romney, the Republican candidate last time round. At the time the Democrats were widely expected to retain a lasting edge in data, not least because they find it easier than the Republicans to attract highly trained data scientists. But that turned out to be wishful thinking. “Most of what we did in 2012 is now a commodity,” says Mr Wagner, who went on to found Civis Analytics, a startup which offers data-management and analytics services to left-leaning groups.

The starting-point for all this information-gathering was the controversial Florida recount after the presidential election of 2000, which became necessary partly because of incomplete voter rolls. To avoid a repeat, Congress in 2002 passed the Help America Vote Act (HAVA), which required states to maintain a “single, uniform, official, centralised, interactive computerised statewide voter registration list”. This was a bigger deal than it appeared at first sight. Along with improvements in database technologies, HAVA made it possible for the first time for political parties to compile an up-to-date list of all voters in the country. In his book, “Hacking the Electorate”, Eitan Hersh of Yale University argues that this laid the groundwork for individual targeting. Campaigners were able to identify voters easily and link them to other public information.

The legislation also kicked off what might be called “database politics”. Democrats were the pioneers: in 2006 party officials set up a company called Catalist which today offers one of the most comprehensive databases on Americans of voting age, covering more than 240m people. Apart from the official register of voters, it also includes other public records and information from commercial data brokers. Each entry contains hundreds of pieces of information, from race to the probability of owning an SUV.

Catalist is best understood as a “data co-operative” for Democratic campaigns, trade unions and other left-wing organisations, says Laura Quinn, its chief executive. For a fee, it gives clients access to its common data pool. They can combine it with their own information and benefit from the firm’s analytics expertise, which mainly comes in the form of statistical “scores”. These numbers predict, for instance, how likely someone is to vote Democrat and go to the polls (see chart).

But the Democratic Party did not want to rely on an outside database. When Mr Obama became president, it decided to create its own. Called VoteBuilder, it also relies on the principle of sharing data. During the primaries all competing Democratic campaigns can use VoteBuilder and combine it with data they gather on the trail. Much of that additional information is kept separate, but campaigns share basic items such as name changes or deaths. Once a nominee has been chosen for the general election, a lot of these data become part of the common pool.

The Republicans’ central database, Voter Vault, has commanded less co-operation and has often been neglected between campaigns. Warring factions have insisted on producing their own databases, often working with commercial vendors. After Mr Romney lost to Mr Obama in 2012, Charles and David Koch, billionaire brothers with a passion for conservative causes, invested millions in i360, a for-profit firm that competes with Voter Vault. On both sides, access to all these databases can be a highly political issue. Insurgent Democratic candidates in state and local races frequently complain that they are being excluded; they need to show a minimum of co-operation to get the data.

Thanks to all this data-gathering, campaigners now seem to “know you better than you know yourself”, as CNN, an American cable news channel, once put it. But the reality is different, explains Mr Hersh in his book. A lot of the available data, particularly the commercial sort, are of little value in helping campaigners decide which voters to target. The best guide is the basic demographic information taken from public records, such as gender, age, voting history and party affiliation.

According to Mr Hersh, this explains why most lawmakers are in favour of allowing easy access to public records. In 2012 a legislator in Utah proposed giving voters the option of limiting access to their date of birth, but the idea was quashed by the leadership of both parties. Campaigners also file lots of requests based on the Freedom of Information Act and state statutes that govern public access to administrative data. “Ironically, laws ostensibly passed to help private citizens track the government’s action turn out to be laws that help political campaigns track private citizens,” writes Mr Hersh. But his main complaint is the conflict of interest arising when parties control the sources of data which they themselves use extensively.

The big question is whether the use of such databases and the algorithms that sift through them change the outcome of elections. Recent estimates suggest that they can add between two and three percentage points to a candidate’s result. In a closely fought election that could be crucial, but in an emotionally charged race between Mr Trump and Mrs Clinton it may not play a decisive role. Then again, Mrs Clinton will probably end up relying on technology more than she has done so far. The Democrats, explains Matt Hindman of George Washington University, always have a harder time getting their supporters out to vote than the Republicans do. Since Mrs Clinton has not enjoyed universal enthusiasm for her candidacy, getting out the voters in November may be even more difficult than usual.

Besides, even if parties do not derive a lasting and decisive advantage from data and analytics, they will still be obliged to invest in technology to keep up with their competitors. And although social media give politicians direct access to their voters, as the Trump campaign shows, big parties still enjoy an advantage because they have the money to hire technical talent, pay for polls and buy advertising on Facebook.

Shrinking the public sphere

More broadly, some people worry about how all this number-crunching will affect democracy, in America and elsewhere. Mr Hersh does not see much of a problem, as long as data are used just to get people out to vote, rather than to try to make them change their mind. But Zeynep Tufekci of the University of North Carolina argues that targeting voters with ever more personalised messages will shrink the “public sphere”, which Jürgen Habermas, a German philosopher, once defined as the basis of democracy. “This form of big-data-enabled computational politics is a private one. At its core, it is opposed to the idea of a civic space functioning as a public, shared commons,” writes Ms Tufekci. And privacy is a growing concern. In December a database containing the records of 191m voters found its way onto the internet.

Rasmus Kleis Nielsen of Oxford University, who has written a book on political campaigns in America, thinks that such targeting will remain largely confined to that country. Nowhere else have party organisations access to so much money, data and technical talent. Moreover, America’s political system lends itself well to analytics because once voters get to the polling station they often have only two options.

Yet there are signs that some other countries are shifting in America’s direction. Apart from buying lots of ads on Facebook, Canada’s Liberal Party in last year’s election used the services of Civis Analytics, a firm spun off from the Obama campaign in 2012. In Britain, too, targeted ads on Facebook helped the Conservative Party win the general election last May. And when it comes to using social media to influence the political weather more generally, America is by no means alone.

This article was from The Economist and was legally licensed through the NewsCred publisher network

Read our exclusive interview with Harper Reed, the CTO of the Obama campaign in 2012