Yes, Russia abused Facebook, but did it work?

Russia’s propaganda operations during the 2016 US presidential election were broader than previously thought, according to two recently published studies. But they don’t provide proof the influence campaign was as effective as the Kremlin may have hoped. Both reports, based on data provided by social networks, combine a distrust of the companies’ disclosures and a naive trust in the metrics they tout.

Facebook, Instagram, Twitter and Google have handed over data on the activities of the Internet Research Agency, a Russian troll farm that’s the subject of an indictment obtained by special counsel Robert Mueller, to the US senate’s select committee on intelligence. Two teams, one from the University of Oxford’s Computational Propaganda Project and social network analysis firm Graphika, the other from disinformation protection outfit New Knowledge, have combed through this data. Both allege the social networks were selective in their disclosures when the Russian influence campaign first came to light.

The initial controversy focused on the ads the Russian outfit bought on Facebook for about US$100 000. But the data show the “organic” posting of divisive content — material pandering to right-wing audiences, or posts aimed at stopping minorities from voting for Hillary Clinton — formed a much bigger part of the Russian agency’s work than the ads.

The organic posts had the bigger reach, too, according to ComProp. The content was seen by 126 million users on Facebook and 20 million on Instagram, according to Facebook. New Knowledge says it suspects the latter number has been low-balled.

Surprising

It’s surprising that a large part of the activity was on Instagram, part of the Facebook empire almost completely overlooked in the controversy. The Russian group’s Instagram accounts generated 183 million likes and four million comments, according to New Knowledge. On Facebook, some 31 million users shared the content and almost 39 million liked it, ComProp said.

The numbers suggest Facebook successfully restricted the scope of the initial debate to the paid ads on its main site as it attempted to limit the damage. The company’s first response to the exposure of the Russian group’s campaign was merely to make its political ad-sales practices more transparent. A crackdown on “inauthentic accounts” came later, and, as New Knowledge points out, some accounts linked to the agency still remain active.

It’s always been difficult to measure the impact of propaganda in print, radio or television. Given that Donald Trump outspent Clinton on Facebook, despite spending half as much as she did on the entire campaign, it’s tempting to think that digital ads are far more effective than traditional ones. Certainly, the audience and engagement numbers unearthed by the studies look impressive.

That’s not to say, however, that Russia’s campaign was highly efficient. According to ComProp, the methods the Russian campaign employed were lifted not from traditional political campaigns, but from digital marketing. And that provides the best metric by which to gauge their success.

In April, Brett Gordon and Florian Zettelmeyer of Northwestern University and two Facebook employees, Neha Bhargava and Dan Chapsky, published a paper on the efficiency of 15 Facebook advertising campaigns. Their most striking finding was that traditional observational methods — which look at how many people were exposed to an ad and how many bought a product — tend to overestimate grossly the true effect of ads.

The main reason? Many of the people exposed to an ad or an organic-looking post about a product would have bought it anyway. It’s no different with candidates or causes.

The Gordon paper uses a Facebook-devised measure of ad campaign success called “lift”, which doesn’t just observe how the targeted group buys, but sets up a control group and figures out how much of the audience would have bought anyway. That way, it can establish causality between ads and purchases better than traditional observational methods.

‘Lift’

It can’t be applied to elections because there’s no way to track how people actually vote. But in the 15 commercial campaigns studied in the Gordon paper, “lift” — the additional business generated — varied between -3.5% and 153.2%; only eight of the campaigns generated “lift” statistically different from zero at the 5% confidence level.

This suggests that for all the new evidence in the latest studies, it’s still unclear whether the Russian influence campaign worked as intended. It was certainly expensive — the Internet Research Agency’s monthly budget of $1.25-million for the US is comparable with the Clinton campaign’s Facebook spend. But until candidates in a future election ditch traditional advertising methods and go all in on social, it will be impossible to show how well online propaganda works. These crude audience and engagement numbers mean little and should be trusted less.  — (c) 2018 Bloomberg LP