When evidence isn’t clear

18th February 2014

Wolf Ellis

Associate

Poverty measurement and targeting methods must be transparent. The lack of important details in a recent report highlights deeper problems with using proxy indicators to target aid and welfare programmes: the resulting household selections can be regressive and almost arbitrary.

Practical and reliable estimates of wealth and poverty are becoming increasingly important as more countries introduce social protection systems, and as development programmes place a greater emphasis on targeting. However, there is not yet a widely-accepted ‘gold standard’ measuring method – and in the absence of this, it is difficult to assess the relative merits of competing approaches.

Given this uncertainty (and the serious consequences of wrong measurements in many programme settings: the wrong people get the aid), it is important that studies include a full description of their method(s) of measurement, or at least provide references that offer this explanation. Household Registration and Targeting in the Hunger Safety Net Programme 2, a report from the ‘Evidence on Demand’ programme (no relation to EfD), illustrates the problem of evaluating new methodologies where key information is missing.

The report is presented as outlining “the rationale behind the targeting approach for the Kenya Hunger Safety Net Programme (HSNP)”, and describes a complex targeting procedure which includes an interesting way to combine absolute and relative measures of poverty while making practical use of previous research. The key question is how this compares with other approaches: is the procedure a good enough method of measuring poverty and practically and reliably targeting the poorest households?

It’s difficult to say, based on the information provided. The crucial omission is that the ‘proxy means test (PMT) score’ consumption estimate – used as the poverty measure that plays a central role in both deciding settlements’ quotas of HSNP beneficiary households and then selecting beneficiary households within the settlements – is based on a poorly-described relationship between household characteristics and household consumption, apparently observed from the poorest four counties’ data from the 2005-06 Kenya Integrated Household Budget Survey (KIHBS).

Some very brief information is given about how PMT scores are produced, but there is no information on the particular household characteristics measured, their assigned indicators or the coefficient weights used, even though the choices made about these can all greatly affect outcomes. This makes the model and its relationship impossible to replicate or properly evaluate.

There is also a lack of information on the PMT scores’ reliability in relation to other indicators of local poverty. The report does contain a comment noting that some of the HSNP programme’s consumption estimates are “all within the error range of the estimate from the KIHBS 2005-06 survey data”, and this is interpreted as “a basis for some confidence in… the validity of the relationship between… household characteristics and consumption”. However, this interpretation is potentially self-referential: since household characteristics are all that have been recorded for the HSNP programme, the similarity between the estimates could merely imply that there has been little change in these characteristics.

The HSNP targeting method is highly dependent on the 2005-06 KIHBS data, but there is no clear explanation of why it is used as such a cornerstone, how reliable and complementary that data was, or whether the relationship between household characteristics and household consumption observed in these four counties then might have changed in the seven years between 2006 and 2013.

Proxy means tests are widely seen as an apparently objective and pragmatic shortcut for identifying poor people, but it’s questionable whether the HSNP should have used these ‘consumption-through-characteristics’ estimates for its targeting. The paper explains that this decision was taken because actual household consumption data would have been “more timely and expensive to collect” from the almost 375,000 registered households, but notes as one of several bullet-pointed “proxy means test weaknesses” that the chosen method still requires large amounts of data collection. 40 questions are needed per household to generate a consumption estimate (with no mentions of the capacity of the HSNP Secretariat collecting the data, or whether any answers were verified).

Indeed, a further acknowledged weakness is that the model can’t estimate consumption for households that don’t respond to all questions. We are not told how many households are excluded as a result of this, or whether the small computers into which questionnaires were directly entered might have prevented questions from being left unanswered. At the very least, there do seem to be consumption estimates for significantly more than 100,000 households. However, we know from Kidd and Wylde’s extensive assessment of proxy means test methodologies for AusAID that in one study in Sri Lanka such exclusions would have reduced the total number of households included from 5,619 to 344.

For the HSNP model there is also “31% of variation in consumption that is not explained by PMT” – “so some households’ actual consumption [is] not well estimated” even for the original KIHBS 2005-06 data. At almost a third of the data, this is a problematic basis for deciding which households are selected or bypassed for potentially vital support. Kidd and Wylde also find that proxy means tests are regressive – PMTs seem to systematically under-estimate the consumption of the richest households and over-estimate the consumption of the poorest households – and that generally “eligible households have no better than a one-in-two chance of being selected”. They conclude that a PMT “therefore functions… like a simple rationing mechanism that selects some poor and non-poor but excludes large numbers of eligible poor from receiving benefits and support.”

Houses in Oyam district, northern Uganda. For targeting purposes, household characteristics may not be the best indicators of consumption.

Despite some interesting ideas, the contribution to poverty measurement and targeting debates of Household Registration and Targeting in the Hunger Safety Net Programme 2 is limited for now by its lack of the substantive details and explanations that would be needed for testing and wider use of its model. More information on these would help.

This lack of information is revealing in one sense, however, highlighting wider problems with targeting methodologies that are based on proxy means tests. The apparently clear procedures for statistical models like these make it possible for them to be portrayed as ‘objective’, but such statements obscure the many methodological decisions and assumptions upon which the models are built: even a model derived from choices that each appear to be entirely objective and evidence-based has little value if the final household selection is almost arbitrary in practice.

Considering PMTs’ tendency to target the wrong people, policy makers should be cautious about their accuracy – and all of these factors need to be presented transparently for thorough assessments of how proxy means tests compare to other methods of estimating wealth and poverty.