A shield not a sword - reflecting on ‘The Elusive Craft of Evaluating Advocacy’ with Mark Schmitt and Steven Teles
Introduction
In 2011, the article ‘The Elusive Craft of Evaluating Advocacy’ by political scientists Mark Schmitt and Steven Teles, was published in the Stanford Social Innovation Review.
This influential article highlights the importance of adopting a learning-focused approach to evaluation in advocacy. It emphasises the limitations of quantifiable metrics and the need for evaluators to possess deep political knowledge and judgment.
It has been central to the way we think about evaluation at Firetail. We spoke to the authors about why they wrote it, their reflections on its impact, and how they think about evaluating advocacy today.
Revisiting “the Elusive Craft of Evaluating Advocacy”
The central argument of the article is that evaluation is a craft, and its true purpose is learning. In 2011, this perspective stood in opposition to a more mechanical, audit-driven view of evaluation that believes it is possible to “prove” the effectiveness or “impact” of an advocacy project with metrics.
Instead, the article encouraged an approach to evaluation which embraces the messy, real world, by recognising:
“Advocacy evaluation is a craft - an exercise in trained judgement - one in which tacit knowledge, skill, and networks are more useful than the application of an all-purpose methodology.”
The challenge is necessary because advocacy is about political change, and political change is too uncertain to “prove” things about a project that can be reduced to a number, or proven to the satisfaction of an audit.
Any desire to reduce success or failure in advocacy to quantitative measures is probably doomed to fail. The authors described how many of the evaluation approaches at the time were designed for evaluating service delivery, and were simply inappropriate for advocacy. They describe the challenge facing advocacy evaluators:
“Evaluators must acquire and accurately weigh and synthesize imperfect information, from biased sources with incomplete knowledge, under rapidly changing circumstances where causal links are almost impossible to establish. There is a natural temptation to formalize this process in order to create at least the appearance of objective criteria, but it is far better to acknowledge that tacit knowledge and situational judgment are what really underlie good advocacy evaluation, and to find evaluators who can exercise that judgment well. It’s the evaluator, rather than the formal qualities of the evaluation, that matters.”
It is an excellent article, foundational to our thinking. It has deeply influenced not only our professional approach but also the way we think about the purpose and conduct of evaluation.
The origin story - “a shield not a sword”.
Mark and Steven’s original ambition was to write an article that pushed back against pressure from donors to evaluate advocacy work in a way that could “prove” whether it would work or not - prior to being launched. They wanted to encourage people to reflect on the structures, incentives and mental models around evaluation that led to bad practice.
MS: “The incentives, even at the very beginning of a philanthropic venture, are to say: show me that this is going to work, show me when it's going to work, and show me that you have a strategy to get from here to there in two years or three years, or if you're generous, five years. I think it's really good to push back against that.”
ST: “I actually thought it was more of a shield than a sword. It was more something people could use to push back against nonsense.”
In part, this ambition arose from the poor practices and vanity metrics resulting from a “best practice” and metrics-driven way of thinking about evaluation that they observed being demanded by funders at the time.
They wanted to make the case for evaluation approaches that acknowledged the limitations of demanding concrete evidence of success within strict timeframes.
ST: “Whatever gets measured, gets managed, and that can give us vanity metrics - on downloads of a report, or page impressions, or retweets etc. I think you can see the anxiety in that. That there have got to be some breadcrumbs between that and whatever their policy change is or could be.”
They were also keen to challenge the cultural and institutional pressures in philanthropy that were resistant to change and different ways of thinking.
ST: There are really profound isomorphic forces in this field that draw people away from doing something different. Especially in philanthropy, people are really very median hugging. They're very uncomfortable getting away from ‘best practices’.”
MS: “There's a certain amount of self-justification that has to exist in philanthropy and that's where you build all the apparatus of the evaluations to feel like you're actually being rigorous about something you actually shouldn't want to be that rigorous about.”
They were also keen to draw on lessons and approaches from other fields. They wanted to provide alternatives to this overly reductive, predictive, project-based and metrics-focused way of thinking about evaluation and impact.
ST: I thought a lot about military strategy in this context, which is an underestimated resource for people who are in the [social impact] world, because people have been thinking about these more abstract questions for hundreds of years. They do “After Action Reports” and it’s just a regular part of all military action. They are engaging in evaluation and reflection when they write military history. There's formal military history that's written that's designed to extract lessons out of experience and fit them into doctrine. Part of the issue is the military experience is so alien to most people who are in these ‘helping’ professions that that repertoire isn't really available.
They wanted donors to recognise that, particularly in the context of advocacy, relying on the prediction of simple chains of causality will not suffice for effective evaluation. Advocacy has an opportunistic element that can’t be captured by monitoring the aggregate effects of an organisation’s outputs and activities.
ST: “Imagine playing Russian roulette with 20,000 chambers. Everyday you’re spinning it and once in a while, the bullet hits. It's an awesome metaphor from the point of view of the target of political activity. Imagine you are a polluter or something. What you're mostly counting on is the fact that the chamber is empty over and over again. The point of view from the other side - from the advocate - mostly the thing that determines whether it's successful is just that random force of the chamber spinning over and over again, not the bullet that they put into it. Now, I think the bullet they put in is really important because if your moment comes, and you don't have the bullet to take down your target, then you're going to be in trouble - and that's where all the preparation and everything else comes in. But you can't evaluate this.”
What has changed since they wrote the article?
Mark and Steven were not alone in thinking about the common limitations of evaluation, and they feel the field has come to absorb many of the lessons from the original article.
MS: “We entered a field that was not empty. But we didn't know a lot about what was going on in that field. It wasn't as though people doing professional evaluation weren’t trying to think about these issues. I think sometimes they were trying to fit it into quantitative frames that were familiar to them. I feel like the field, which is pretty well developed, has absorbed some of these thoughts. If what we were doing was provocative in 2012, I sure hope it's not as provocative now.”
They believe the concerns that inspired the original article have led philanthropists and foundations to think more about developing organisations and institutions rather than short-term campaign-based objectives. This is because more funders today take a longer-term and more holistic view of how change in politics happens.
ST: “If I were ever doing philanthropy, again, I would always want to say - what's the basic foundation? What is the organisational structure in the long term, year in and year out? Is it engaging the public, is it developing ideas and so forth? Are all of those pieces there as opposed to going step by step towards a campaign type goal?”
MS: “I think we're moving towards a broader understanding that the foundations are more important than the short-term goals. People don't mobilise themselves around childcare, even if that's something they need. They mobilise themselves around a broader sense of their own meaning and what the country means and so forth. And channelling everything through short term specific goals I think is something that we'll get away from a little bit in the way that in the electoral world, we get away from throwing everything at a single campaign. And we’ll move a little more towards realising that actually, what you need to do is build the kind of institutions that connect people over a longer period of time.”
Good evaluation is still about making better decisions
The authors have always believed that evaluation should focus on questions that will help an organisation make decisions about how to move forward.
Good evaluation is more important for organisational learning than donor reporting and should focus on what matters to an organisation’s Theory of Change.
ST: “You want to measure the thing that is actually, fundamentally related to your strategy. I'm not a complete measurement nihilist. But I think there's a difference in the measurement you do in order to improve your own organisational functioning, and the audit instinct, which is a powerful instinct. If I was a donor, I would want to see - ‘what are you measuring in order to improve your own organisational functioning based on your own internal Theory of Change?’. But I worry a little bit that once the donor starts looking at it that that thing then turns into auditing”
It also needs to think more about contribution in context rather than attribution in isolation, recognising the importance of luck, opportunity, and those wider structural issues Steven described as “the wheel of history”.
ST: “You can't say, well, these people actually passed a bill and therefore their advocacy strategy was great because maybe their advocacy strategy was terrible and they just lucked out that the things spun the right way. And that's why people almost always overinterpret moments of political success.”
In the end, the practice of evaluation is most useful when you are trying to learn things you don’t know, rather than prove things you thought you did.
MS: “Evaluate things you actually want to know about. Because so much evaluation is just self-justification. Trying to put some quantitative models on what you already think, as opposed to looking at what's the thing I don't know that I actually really do want to know. And that thing might not be: was this a good grant or not? It might be something much deeper. So, trying to identify the things you don't know, and really want to know, is surprisingly a missing piece in a lot of evaluations.”
Conclusion - less elusive, but still a craft
In our experience, the lessons of this important article still ring true today and whilst much has changed, the challenges raised remain relevant in fields beyond advocacy.
Evaluation needs to link to an organisation’s strategy, not a donor’s preferences. A good outcome doesn’t validate a bad strategy, a bad outcome doesn’t invalidate a good strategy, and it’s very hard to tell the difference.
The evaluator matters more than the metrics. The practice of evaluation is about judgement, not certainty. The best unit of evaluation is the whole organisation, in context, over the long term.
Many people we work with recognise these principles. Organisations talk about being ‘learning’ organisations but find it hard to break out of the audit mindset, especially when they have built a big infrastructure to deliver it. It is still common to find evaluation requirements focused on ‘proving’ the impact of interventions to a standard that would satisfy an audit committee.
For Firetail and the clients we work with, Teles and Schmitt’s article has been a vital contribution to reframing the debate, and we are incredibly grateful to them both for their work, and for taking the time to talk to us about it.
—-
About Mark Schmitt and Steven Teles
Mark Schmitt is an American political scientist and author, and is Director of Studies and of the Political Reform Program at the New America Foundation.
Steven Teles is Professor of Political Science at the Johns Hopkins University, and Senior Fellow at the Niskanen Center.
Quotes have been lightly edited for grammar and brevity.