Policy for Campaigns 03: How to make policy when you don’t have any evidence - Part 1.

It can easily feel like you don't have the evidence you need, particularly in smaller organisations

One of the most daunting and frustrating challenges you can experience when working in policy is not having the evidence you need. However, things are usually not as bad as they might first seem, and this article explores approaches you can take when you are low on evidence.

Why do you need evidence?

With the current state of the public discourse, it might be tempting to think that all you need to succeed these days is a few bold assertions and a heap of confidence. Isn’t evidence-based policy all a bit old-fashioned?

Sadly, while government ministers might be able to get away with that attitude, at least for a while, it doesn’t really work for a campaigning organisation. You will be bound by the traditional expectations that you should be able to back up what you’re saying. But more seriously, of course you need evidence in order to make policy effectively: it is essential for developing your understanding of whether you need to change something, and ensuring that, when you ask for that change, you’re asking for the right one (and there will be plenty in future articles on policy development and identifying priorities).

On the flip-side of this, if you don’t have evidence you potentially have some fundamental problems. You won’t be able to back up your claim that there is a problem to solve, and you won’t be able to develop a solution. This is a particular problem if the solution is likely to be in any way technical, although it could be slightly less difficult if you’re simply calling for a proposed action not to be taken. Then again, the latter is always a hard situation: if there’s a defined change in the offing and your only solution is “don’t” your campaign may well struggle: really you want to be offering alternatives. But again, the particular challenges of a “stop the bad thing” situation are a candidate for a future article. For now, let’s look more broadly at what it means not to have the evidence you need for your policy work.

What evidence would you ideally have?

If you feel you’re lacking in evidence, you may be lacking specifically in data: that is, numerical, quantitative evidence. This can be incredibly important: if you don’t know how many people or organisations are affected by the problem you’re working on, for instance, it can be hard to persuade anyone that it matters. Going further, if you can show the extent of a problem in numerical terms your case can be more compelling: is it costing households, or businesses, or a particular category of person, a certain amount each year? Can you therefore point to a financial or economic benefit from remedying the problem?

Ideally also you will have longitudinal data, showing how your issue has been changing over time. Has it been getting worse? If not, maybe you’ll be open to the criticism that t’was ever thus and we’ve always muddled through. As this implies, evidence can also be important to pre-empt or rebut criticisms either from decision makers who are reluctant to act, or from other campaigning groups who might have competing agendas, or at least competing priorities.

Ideally you will have a lot of detailed information about the people or organisations you are representing. How do they live their lives, run their businesses, or whatever it is? What do they spend their money on, and what do they struggle to do? This is not to say you’ll necessarily need all of that detail for the actual campaign, but that’s downstream of understanding your issue and developing your policy, where detailed evidence is highly useful.

You will also want qualitative evidence, not least case studies or real life examples. Ideally you would have a massive bank of these, with people in every possible permutation of the situation you’re talking about on-tap, ready to feed their experiences into your policy work, as well as attend meetings with ministers, speak to the media and do whatever else is helpful in supporting the campaign. However, the word “ideally” is doing a lot of work there: gathering case studies is resource-intensive, as is the business of maintaining the relationship with each of those individuals, and ensuring the information you hold is up to date. Only organisations that can dedicate meaningful resource to the task will have such a bank.

A question for readers: what other forms and types of evidence do you find useful? Are their categories or approaches that I’ve missed out here?

Why don’t you have any evidence?

You may sometimes find yourself getting queries from journalists, members of the public or even other organisations asking the most specific questions about the cause you work on and people affected by it, seemingly expecting that you’re sitting on vast banks of information just waiting to be used. You probably aren’t: it’s very unusual even for larger organisations to have truly comprehensive knowledge of the area they work on.

For the most part, evidence doesn’t exist unless you gather it, which usually means going out and looking for it in some way. You will usually be doing this for a particular reason, rather than on spec and for the sake of having it to hand on the off-chance that it’s ever needed. That said, you might be lucky and have strong evidence of your problem that is gathered as a matter of routine, such as sales data from member businesses in a trade association for example; but even then, you’ve got to get the members to pass it to you and, and then you’ve got to curate it so that it exists in a form that can be used for policy.

But more commonly, you might run up against one or more of a wide range of barriers. Evidence can cost money to gather, or more accurately can require resource: even a relatively basic online survey needs staff time to put together and then analyse; commissioning an external supplier for research obviously carries a cash cost. You may find that the issue you want to explore is hard to pin down: even explaining what you’re asking about might be complicated if it’s a technical issue; you might find it hard to identify or reach the population of individuals or organisations you need to speak to; turning raw data into the insight you need might require some sophisticated analytical techniques. Obviously, all of this is likely to be a greater problems for smaller organisations, but these are barriers that any policy professional or team can run up against.

So, there are plenty of barriers between you and the evidence you need. But are they insurmountable? Or even as numerous as they might first appear?

Do you really have no evidence in your organisation?

Firstly, you’re an organisation that campaigns in some form, so you must think there’s something to campaign for: a problem to be solved, or a better way of doing things to be sought. Presumably that hasn’t come out of thin air, so don’t overlook the basic reasons you have for thinking what you think: someone must have mentioned it, whether that’s calls to an advice line, the direct experiences of founder members (for a new organisation), or whatever. You may have some basic information about your members: if 90% of people in XYZ Situation depend on a particular grant; cutting the grant will make XYZ Situation unviable, won’t it? From there, all you need to do is show why that would be a Bad Thing / why maintaining XYZ situation is a Good Thing. Ensure this basic stuff is captured and compiled.

Equally, if you’re at the outset of a possible campaign and find the cupboard bare of evidence, it could be a useful check. Is the problem that you’re worried about definitely a problem, and can you demonstrate that? If it’s a matter of a looming problem that hasn’t happened yet, obviously you don’t want to be waiting for the consequences to affect people before you can present evidence and start campaigning: in this situation, the evidence you present will need to be much more about explaining the causation involved, and what the consequences will be for people or organisations in a given situation. For that, you need evidence about how the status quo works. Another useful check in this sort of scenario is to consider whether comparable changes have happened to another group, and what the consequences were for them: in this case, the main evidence you then need to present is about why the situations are comparable, and again that’s a matter first and foremost of talking about the status quo and how it works.

A variation of the “no evidence” problem is a “no budget” problem: you know what evidence you need, but you don’t have the resources to go out and get it. Don’t overlook your organisation’s existing activities: maybe it runs a member survey that is focused on satisfaction with services provided to members; can this be expanded to include policy-related questions? Don’t assume the answer is yes: there may be reasons why that’s not a suitable channel; but equally, don’t overlook it. Also, consider other interactions with your member or supporter base: do they communicate via social media, or online forums, or simply through regular contact with the organisation? Consider what you can glean from this correspondence: you could comb through it manually; if it seems suitable, you could introduce some policy-related questions to those channels; and you might consider developing some way to categorise any records created, for instance of calls to a helpline, so that you can easily search them in future.

Or consider a thornier variation: what if there is evidence, but it doesn’t support your position? For instance, maybe there is opinion polling of a representative sample of the population that paints a different picture from what you’ve been hearing from your supporters, but only have lower quality evidence for. Firstly, check whether they really are telling different stories. I recently dealt with some data from a sizeable but self-selecting sample that showed a largely negative view of remote healthcare appointments, only then to see some polling published that presented markedly more positive opinions on the subject. Drilling down however, the polling data showed more negative responses among older respondents – the same group towards which my self-selecting data was heavily skewed! The proportions didn’t match exactly (nor should they have, with such different methodologies), but where they overlapped the two sets of data were telling the same stories, despite superficially contrasting headlines.

However, it’s possible that larger scale evidence might jar with yours more comprehensively: indeed, it may suggest that your supporters are just weird. Depending on the issue, this might be OK: the people agitated enough to join a campaigning group may well be outliers (although some trade bodies look to have majority or even total coverage of the organisations they represent, so nasty surprises of this sort won’t be a concern). But the implication could be that the problem that has seemed so drastic is relatively minor, or just a matter of preference among a slightly awkward group of people. Then again, don’t leap to that assumption: a disparity between your limited evidence and more robust, comprehensive evidence might actually shed light on something that makes your group distinctive. This could in turn imply that there’s something about the status quo that works well for most people, but not your group: your challenge then will be to make the case that it merits the effort of sorting it out.

So, you’ve identified the limits of the evidence you hold. Part 2 of this article explores possible ways of doing some useful policy work despite its limitations.