Amnesty International Campaign Manual

EVALUATION

Campaigners and those who assist them want to know whether their efforts are actually having the intended impact. If not, they want to know whether there are different things they might try that could be more effective. Evaluation (and the related activity, monitoring) is a tool that all campaigners use to improve the effectiveness of their campaigning. It is a way of learning from and building on experience in order to do things better in the future.

Contents

The importance of evaluation / 272

Why do you want to evaluate? / 272

What do you want to evaluate? / 273

    Evaluation of outcomes / 273

    Evaluation of methods / 273

    Choosing what to evaluate / 273

Timing / 273

Resources / 274

What information is required? / 274

Assessment / 275

Presenting the results / 276

“Who shall stand guard to the guards themselves?”

Juvenal, 60-130 AD, Roman satirist

The importance of

evaluation

The time spent on evaluation should not be seen as lost to campaigning, but rather as time used to enhance the impact of future work. Useful and practical evaluation can be done with relatively small amounts of time and other resources, and does not require specialist expertise.

Sometimes it is desirable to use an expert outsider, but evaluation done by campaigners themselves should be an essential and integral part of campaigning.

Evaluation simply means collecting information and using it to judge the merit or worth of something. The judgment might be about whether an activity was “un/successful” or “in/effective”; it might also include an assessment of the reasons for success or failure, of changes that might improve the effectiveness of the activity, or of additional or alternative activities.

The term “monitoring” is generally used to refer to the regular and systematic collection of information about what is happening. This might be the extent of media coverage, the number of people writing letters on behalf of a prisoner of conscience and the responses being received, or the number of requests for specific publications.

Some people also use the term “monitoring” to describe the process of using the information to assess how well things are going, and to make decisions about whether changes are needed. In this sense, monitoring is the same kind of activity as evaluation.

To avoid confusion, this chapter uses “monitoring” to refer only to the regular and systematic collection of information that describes “what is”. Evaluation refers to the use of information to assess whether there is a gap between “what is” and “what should be” or “what could be”, and if there is, how it might be closed.

The following questions may help you decide when to evaluate your campaigning, what to evaluate and how to do the evaluation.

gQUESTIONS

g    Why do you want to evaluate the campaigning and who wants to use the evaluation?

Is it to inform the membership about what has been achieved or to help the campaign organizers improve the quality of the work, or both?

g    What do you want to evaluate?

Is it just the impact or changes that the campaigning has had, or the campaigning methods as well?

g    Was it the right strategy?

g    When do you need the results of the evaluation?

Is there, for example, a planning meeting scheduled that needs the information to make decisions about future activity?

g    What resources are available or needed for the evaluation?

How much time and money should you devote to it?

g    What information is needed and how will it be collected?

What is the best you can get with the resources you have available?

g    What does the information tell you about how well you are doing and what changes might be needed in your campaigning methods?

g    What are the most effective ways of telling the relevant people of the results? Is it in a newsletter, at a meeting or in a written report?

Why do you want to evaluate?

You need to be clear about “why” you want to evaluate your campaigning, and the related issue of “for whose benefit”, in order to decide such things as “what aspects” will be evaluated, and when and how the results will be reported.

The most important reason for evaluating is to improve future campaigning techniques. The main “audience” for such evaluation should be the campaigners themselves. Campaigners need to know what worked well and why in order to build upon their strengths. They need to know what did not work well and why in order to take steps to overcome weaknesses.

Another reason for evaluation is to inform AI members and others about the results of the activity which they have assisted. They contribute time and perhaps other resources to the campaigning and are entitled to know what has been achieved. If campaigners do not keep them informed, the members and others may not be willing to help in the future.

A third possible reason is “accountability” _ to report on activities to the management committee or the annual general meeting of members.

An evaluation may have more than one purpose. In this case each purpose should be considered in determining what will be covered, the timetable, presentation and other aspects.

What do you want to evaluate?

There are two types of issues that could be the focus of monitoring/evaluation:

N    Outcomes: has anything changed as a result of your campaigning? What impact have you had?

N    Methods: what things did you do in order to bring about the changes? How well did they work?

Evaluation of outcomes

Some evaluations are restricted to gathering and assessing information about whether changes have taken place. The most important changes that evaluation should examine are those indicated by the objectives in the plan, i.e. the intended changes. An evaluation might also note unintended effects that may be important to future campaigning. There might be unintended positive effects (for example, you found a valuable new ally who might cooperate with you in the future) or negative ones (for example, you offended someone you might have wanted as an ally).

Evaluation of methods

Evaluation should also look at the campaigning methods, even if it is clear that the objectives were successfully achieved. “Methods” is used here in the widest sense and includes strategies, techniques, resources (people, money, etc.), and how the work was organized, managed and implemented. Only by understanding what worked well and what did not, and the reasons for success or failure, can monitoring/evaluation help you to build on strengths and find remedies for weaknesses.

Choosing what to evaluate

There is often not enough time, money or interest to evaluate thoroughly every aspect of campaigning. You have to make choices about what you will cover, and should do so on the basis of two criteria:

N    What is most useful to you and other people for whose benefit you are doing the evaluation? What issues are of greatest importance or interest to them? If the campaign had a number of objectives and a variety of methods, were some more important than others?

N    What is practical, taking into account the limited resources and the deadline for using the information?

Timing

Evaluation has to be useful to justify the resources expended on it, and timeliness is an important element of usefulness. Sometimes, the results of evaluation are needed by a specific time or date that cannot readily be altered. This may have significant implications for what is evaluated and the methods used to collect and assess the information. For example, if the deadline is relatively short, you may have to restrict the number of outcomes that you will try to examine and the number of people you can interview.

If you consider it essential that a particular issue be thoroughly evaluated and there is no specific deadline, you should set the deadline on the basis of the length of time required to do the research adequately.

Resources

Only one resource is absolutely essential for evaluation: time. One or more people have to collect information, people may have to spend time providing the information, the information needs to be assessed, and the results considered by those who are interested in it.

Other aspects may require money for things such as postage and printing.

There is no formula for determining how much time and money you should spend on evaluation. If you have some flexibility, then the “budget” should reflect the answers to these questions:

g    How useful would it be to have the information that evaluation could provide? Will it help you to improve the quality of a technique that is critical in future campaigning?

g    How much time/money will it cost to collect and analyse the information that is required to produce useful evaluation?

Can you do useful evaluation with no money and little time? Yes! Lack of money and time are problems that confront people in the IS and Sections with paid staff as well as groups whose members are all volunteers.

What information is required?

The information you need is determined by the focus of the evaluation. Is it going to look only at the changes the campaigning has brought about, or methods as well? For example:

N    if the campaign objective is “to persuade the government to sign an international human rights agreement”, you will want to know what response the minister gave when AI representatives met her or him, and, if the government had announced its policy on the agreement, what it is. Can you find out why -- was it the public pressure, the quality of the presentation, etc?

N    if you want to evaluate the usefulness of campaigning materials, you will want to know what the people to whom you provided the materials thought of them. Did they find the materials easy to understand and use? Were the materials produced and distributed in a timely manner? Did the materials persuade and enable them to take action?

The ways in which you collect the information depends on the kind of information you need as well as factors such as its accessibility, your resources and the deadline.

Some information may be readily available. For example, you can monitor coverage of your media releases by designating people to scan specific newspapers and magazines, and listen to/watch specific radio and television news programs. You can collect information about the impact of an exhibition by counting the number of people who attend and by having a visitors' book in which they are invited to indicate their background (such as whether they are AI members) and write comments. Important events may be documented in minutes of meetings to monitor whether or not people have done what they said they would do, on time, and if not, why not.

A big advantage of planning to evaluate when you are working out a campaign strategy is that you can identify at the beginning the information that will be needed and along the way can consider ways of collecting at least some of that information. For example, it is far more difficult to evaluate your media impact if you have not monitored it during the campaign.

Other information required for evaluation may have to be collected by means such as interviews and written questionnaires. If you are not familiar with designing and using questionnaires, try to obtain advice from someone who is experienced in these methods or from books on the subject.

There is some important information that you will not be able to collect. One reason is that it may be inaccessible. For example, governments often give vague or non-committal responses to requests for information until they think it suits their interests to announce their policy. Another reason is that the methods needed to collect the information may be too expensive and time-consuming. For instance, a common campaigning objective is “to raise public awareness” about an issue. To get direct evidence about this needs surveys before and after the campaign. Human rights campaigning organizations can rarely, if ever, afford to do this. In these circumstances, you may be able to identify and collect other information that is relevant to the issue you are evaluating. For example, in relation to the objective of raising public awareness, you may want to look at:

N    the extent and quality of media coverage _ using the media is one of the most common methods used by AI and other campaigning organizations on the assumption that the public reads and is influenced by “favourable” stories;

N    the response of people who are not AI members _ did they join AI, donate money, sign a petition or request literature? If you update membership statistics on a monthly basis, you can see whether there was an increase in the number of people who joined AI following the launch of a campaign, compared with previous months, or the same month a year earlier.

Assessment

Assessment is the process of using the information to accomplish the aims of evaluation _ to answer questions such as “are we making progress?”, “are we using our resources well?”, “is there something different we should try?”

Assessments can be quite different. One kind is factual _ what happened and why? Sometimes a factual assessment can be confidently made. At other times it cannot be made because we do not have the information that is needed.

In some situations it is relatively easy to claim a link between what we did and the outcome. For example, an AI speaker addressed a trade union meeting and the union then agreed to send a letter in support of an appeal for the release of a prisoner of conscience. In other situations we do not have enough information to be certain about the factors that led to the events that occurred. For example, if AI is one of a number of organizations that successfully lobbied the government to make a particular decision, it is unlikely you will be able to assess the exact contribution you made to the outcome.

A second kind of assessment is “making value judgments”, such as whether an activity was “successful” or whether results were “worthwhile”. Making a value judgment requires not only information about what happened and why, but also criteria against which “success” and “achievement” can be assessed. Success is more difficult to define where:

N    Your plan does not specify targets in objectives or methods, such as “to increase public awareness”. How big an increase is a good outcome? How many signatures on a petition or number of people attending a meeting is a satisfactory result?

N    Your target represents what you “hope” will happen, rather than a well-founded expectation. This is generally the case where an activity is new.

N    Your target was partially achieved _ you aimed to gain support from 10 trade unions and seven agreed.

Where there are no explicit success criteria, the person doing the assessment should be wary of imposing her/his values. The expectations of people involved in a campaign may differ, so it is important to state what happened (“our campaign launch was covered in three newspapers”) as well making judgments (“the coverage was good/poor”) that reflect your own expectations. If you do want to make a judgment, you should state the criteria on which it is based. For example, “the letter-writing campaign was a success _ 300 people participated this year, compared to 200 last year”.

A third kind of assessment involves making recommendations about whether an activity should be continued in its current form, changed or discontinued, and whether a different activity should be initiated. Making recommendations involves making factual judgments about both the past (why did something work or fail?) and the future (how will something work?). Recommendations may be based on an assessment of the information collected for the evaluation, or on information related to other situations, such as successful campaigning techniques used by other organizations, or both. If you want to recommend something because it worked elsewhere, think carefully about the circumstances under which it worked. Are there any critical elements which are different to the situation you are evaluating, such as the resources available?

Presenting the results

How the evaluation should be presented depends on a number of factors, including:

N    who the monitoring/evaluation is for;

N    what they want it for;

N    when they want it;

N    whether a formal record is required.

For example, if an evaluation is to provide information to a large and widely dispersed group of people, you could prepare a detailed report and send it to them, or you could provide a summary in a newsletter, advising that a detailed report is also available on request.

If the evaluation is intended only for the campaign team, the most effective means of communication may be a short outline paper and an oral presentation. This would give the members the opportunity to seek clarification and elaboration, and to discuss how to use the findings. The booklet produced by AI during the 1996 campaign for the establishment of a permanent international criminal court. The campaign was evaluated, both at the IS and by the rest of the movement.

© ai

Amnesty International Campaign Manual