Postcard from Ethiopia: How to measure our impact

16th April 2015

Claire Allan

Head of Programme Quality and Impact, Farm Africa

As an economist, I have always found it logical that NGOs should be expected to demonstrate the impact of their work. Once we have set our objectives, we need to know whether we are achieving them and how we might do better.

That means measurement.

At Farm Africa, we support smallholder farmers to increase their productivity and grow their incomes. If we want to be sure we are delivering this, we need data.

Putting it into practice

But, it is one thing to recognise the need for measurement and quite another to actually do it in practice. Collecting income data from the farming families we work with is very different to asking people in the UK what they earn.

For one thing, they are unlikely to have payslips to refer to. Most households are home to a number of people who earn income, each doing a range of different jobs throughout the year, both on the farm and off it. Some are paid ‘in kind’ – with food rather than cash – and even those doing only agricultural work might produce and sell up to ten different crops at varying prices throughout the year.

All this combines to make information on people’s incomes notoriously unreliable – and unreliable data is no good for measuring results.

New technique

So, when I was invited to join some of my Ethiopian colleagues to learn and apply a relatively new technique that promises to be “a reliable, standardised method of collecting and using household income data”, I was excited to say the least.

The individual household method (IHM) was developed by Evidence for Development in an attempt to improve the quality of household income data in developing countries. It does this by using a standard method of collecting data that can be used in many different countries and settings, and specially designed software which allows data to be checked and analysed quickly.

Focus group, Oromia, Ethiopia

After a day in the classroom learning the theory, we headed to Golla village in central Ethiopia to talk to small groups chosen by community leaders for their knowledge on specific subjects. This preliminary survey is critical for building up a picture of the local economy that interviewers can then refer to during household interviews.

I joined the employment group and learned lots about the different types of work done in the village and going rates of pay. The group told us, for example, that wealthier households often hire workers for an entire year to help with the full agricultural process. In exchange, the worker is given board and lodgings and one quarter of the land to work for himself, supplemented with a small cash salary.

Household interviews

Although time consuming, this context information proved extremely useful in the later household interviews. Knowing what we did about the terms of annual contracts, we were able to ensure that hired workers were included in the list of people to be fed when calculating the household’s basic food needs. This avoided overestimating those household’s food surplus (the amount of food available for the household to sell).

Understanding how we change lives

Because of the complex nature of the areas we work in, we are unlikely to ever be able to fully measure – or even describe – every impact that our projects have. However, as we improve the reliability of the data we can reasonably collect, we at least begin to understand the size of the changes we are bringing to people’s lives and we can use that information to continually improve the services we provide. And that, after all, is exactly why NGOs should be measuring their impact in the first place.

This was first published by Farm Africa.