Follow up to: The effectiveness-alone strategy and evidence-based policy

A translation of the opinion piece can be found here.

I

Effective altruism is a great concept, but it's not trivial to sell. There are therefore good reasons to ally ourselves with other rationalist memes to increase the level of rationality and effectiveness in the world. One powerful such rationalist meme is "evidence-based policy", which is inspired by the "evidence-based medicine" movement. 

The exact meaning of evidence-based policy is somewhat disputed, but generally proponents of evidence-based policy demand that the standards on which policy is based should be raised. Many believe strongly in randomized control trials (RCTs) and in the "hierarchy of evidence", but there is not complete agreement on the strength of RCTs relative to other kinds of studies.

In the US and the UK, there are several organizations which work on evidence-based policy, such as the British What Works Network and the American Coalition for Evidence-Based Policy. Inspired by them, I took the initative to start a Swedish network for evidence-based policy at the start of this year. We are by now around 50 (depending on how you count) researchers, civil servants, journalists, consultants, students and other activists in the network. Only myself and a few others are EA members, so it's not an EA organization, but as I argued in my previous post, I do believe working on this nevertheless is an effectively altruistic cause.

One difference between us and What Works is that we aim to be a broad campaigning organization. We believe that policy not being evidence-based is not only due to a lack of knowledge, but also due to a lack of will, especially among politicians. Politicians often disregard expert advice (on what policies are the most effective to reach a given set of goals) which goes against their political prejudices.  Therefore we need to put pressure on politicians - not the least in the media - rather than just work behind the scenes as an expert organization.

II (Most linked replies below are in Swedish)

Our activities were fairly modest until last Sunday, when we wrote an opinion piece calling for evidence-based policy (English). The opinion piece was published in the most widely-read broadsheet, Dagens Nyheter, on DN Debatt - a sort of op-ed forum. DN Debatt has a special standing in Swedish politics. Everybody reads it and it's well-respected.

Hence we had expected a lot of attention, but the results still exceeded them. Ours was the second most shared DN Debatt-article in the month of May. We got seven replies in Dagens Nyheter, were strongly criticized in the other main broadsheet, Svenska Dagbladet (conservative), parodied in a popular public service (equivalent of BBC) podcast, and were also commented on in a number of smaller newspapers. The discussion on Twitter was pretty intense. Subsequently, we also published two replies to replies in Dagens Nyheter and Svenska Dagbladet.

It's hard to tell what the majority opinion on our piece was. Certainly, there was a lot of praise and a lot of Facebook likes, but also some fierce criticism. This was almost exclusively down to misunderstandings. I won't bog you down with all of the details here, but will rather summarize my general conclusions. They could be useful for anyone trying to write on evidence-based policy or related concepts in other countries. 

I should say that "evidence-based policy" isn't as entrenched a concept in Sweden as it is in the US and the UK, which probably played to our disadvantage. 

1) You need to be very clear over the means-ends-distinction. Evidence-based policy is about making the methods for reaching your political goals (happiness, equality, liberty, etc) more effective by the use of evidence. It is not about propagating any particular set of political goals. We tried to be clear about this, but partly failed for two reasons. Firstly, Dagens Nyheter set the headline, which was misleading. Second, we only clarified this distinction at the end. It should have been at the top.

2) There is a straw man conception of evidence-based policy, or expert-informed policy and political rationality more generally, akin to Julia Galef's "Straw Vulcan" conception of rationality. Perhaps this varies a bit from country to country, but in Sweden it's strong. Let's call it "Straw Soviet" for now (please come with suggestions!).

According to this conception, evidence-based policy means technocracy (of a dictatorial form, according to the more extreme interpretations), disregard of non-quantifiable values (cf "Straw Vulcan"), disregard of emotions, "Mad Scientist"-conception of society as a labratory, etc, etc. You need to everything you can to counter such interpretations. I certainly underestimated the power of this straw man meme. I should also say that the Straw Soviet is probably more vicious than the Straw Vulcan, who seems more innocent (perhaps this is partly down to Julia's plafyul presentation of it, though). 

For instance, Svenska Dagbladet's criticism was all about the "Straw Soviet". We were said to want to "design voter behaviour" (this was also partly due to the article having been signed by a few nudgers who call themselves "behavioural engineers" - a big trigger of the Straw Soviet). Here are some more quotes:

It is perhaps not the “enlightened despot” who is called for in the opinion piece, but rather Dr Despot. Today’s most frightening reading came from the recently formed “Network for Evidence-Based Policy” (Dagens Nyheter 1 June).

//

Since there probably are very few citizens who base their votes on research reports, free elections yields results which are not evidence-based. According to the argument in the opinion piece, that means that since we “see the world through partisan lenses”, the election results are as a rule problematic or directly harmful.

//

Now if the network were correct, true evidence-based policies would lead to a single proposal, a solution “free from ideology and populism”. That would in turn mean that all parties arrived at the same answer, and it is absolutely impossible why that – though ever so full of evidence – would be desirable.

A vibrant democracy is based on the existence of conflicts of opinion and value, intellectual diversity and the citizen’s right to freely express it. The complete and rational citizen is an anomaly, and based on the unpleasant idea that enlightened powers can raise, design, a new man.

Paradoxically, it is precisely highly ideological regimes which have attempted just that. The results have been devastating.

We got several other replies along these lines, though we also got a much more positive one from Dagens Nyheter itself. A large group of replies treated more technical and hum-drum issues concerning RCTs, practical policy-making, etc.

3) Connected to the Straw Vulcan and the Straw Soviet, there is a "Straw Naive Positivist Scientist" (again, suggestions for better terms are welcome), who thinks that knowledge is easily obtainable even in messy fields like economics, that it's easy to reach consensus if you just don't lead political misconceptions mislead you, that you always easily can infer policy-advice from research, etc. We got a lot of criticism which was based on the Straw Naive Positivist. Obviously, we don't hold any of those views.

4) People read very superficially. This is not only true of the man in the street, but also of many journalists, politicians, etc. At some level I know this, having myself written about research on this on my blog, but it's harder to make full use of that knowledge when you write.

Also lots of people don't use the principle of charity at all. Some of the replies - including one from a philosophy professor - were exceedingly uncharitable. Thus don't expect people to use the principle of charity - especially when emotional memes like the Straw Soviet are around.

When you fight such powerful memes, you need to be extremely clear. You need to say the things you really want to get across early, to repeat them, and to give examples. If at all possible, you should control the title, since that sets so much of the tone of the piece (give the publishers a juicy suggestion and they might buy it). Don't say too much, but focus on getting the central message across.

This is so different from writing an academic paper. Of course that's obvious, but it's one thing to get it on an intellectual level, quite another to really internalize it. If you could get a skilled public communicator on board, that would be very useful.

I also think it would be good to pre-test major articles (e.g. on Mechanical Turk) to get a clearer picture of whether the message gets across. If you don't want the content to leak beforehand, that might not be doable, though.

5) We were probably a bit too extreme regarding RCTs, which triggered the Straw Soviet and the Straw Naive Positivist (for epistemological and ethical reasons). It would have been more tactical to emphasize other stuff.

6) We would have come off as more concrete if we had based our opinion piece on a research report on the state of Swedish policy-making. It's great if you can do that, but I don't think it would have been rational for us (see below).

7) We should have stressed how big the movement on evidence-based policy is in the US and the UK. For instance, we could have mentioned that "Obama's 2016 budget calls for an emphasis on evidence-based approaches at all levels of government". Obama being popular and respected in Sweden, that would have done much to disarm the Straw Soviet. 

8) It was a mistake to mention legal means as a way of making politics more evidence-based, since it strongly triggers the Soviet meme. Even those who otherwise supported us criticized this suggestion.

III

In our replies, we focused on rectifying the misunderstandings, focusing on the claim that we are calling for "Dr Despot". These replies normally gets much less attention, and so it was with ours as well. However, the reception also was more unanimously positive, especially from academics and civil servants who know the field.

I don't regret writing this opinion piece at this early stage. Before I started writing it (I wrote the body of the text, and the others then made minor tweaks) there wasn't much activity in our network. Now, we have many more members, including more senior ones. Also, those who already were in the network grew much more enthusiastic after the publication. Thus all-in-all it's been a major success. Still, I think you can learn a lot from things we could have done better.

I'll write more later on how the network is developing more generally. Also I should add that I'm still digesting what I've learnt, so my conclusions aren't set in stone. Any comments are welcome.

[Edit: Added #7, #8]