Towards better monitoring and evaluation
Katarina Mathernova, Joost de Laat, Sandor Karacsony* Русский/Russian
Identifying the poorest and most vulnerable communities is key for achieving results on Roma inclusion. Sandor Naske/Chachipe Youth Photo Contest
BRATISLAVA, Slovakia – 26 April, 2012 – The April 2011 European Union Framework for National Roma Integration Strategies calls upon European Union Member States to include strong monitoring and evaluation components.
Monitoring and evaluation is critical for results on Roma inclusion
This was underscored during recent conferences on the topic organized by the governments of Slovakia and Bulgaria in partnership with the World Bank, UNDP, the Open Society Foundation, the Euorpean Commission (EC), the Slovak Governance Institute, and the Poverty Action Lab Europe (J-PAL).
The conferences highlighted several well-established monitoring and evaluation tools, and there are encouraging international collaborative efforts to promote and use these tools to achieve results for Roma inclusion. In particular, the conferences distinguished four main areas:
- Ensure good targeting of inclusion programmes by identifying the poorest communities.
- Identify the critical gaps in employment, education, housing, health, etc., between the poorest communities and the general population, and monitor progress in closing these gaps by institutionalizing the collection of comparable survey information.
- Devise Roma inclusion programmes focus on achieving results in closing the most critical gaps by institutionalizing results frameworks and programme monitoring.
- Establish continuous learning and innovation, by institutionalizing evaluations and social policy experimentations to learn which approaches are most (cost-) effective to scale up.
Ensure good targeting of inclusion programmes through poverty mapping
Key indicators of poverty are based on measures of income, such as the ‘at-risk-of-poverty’ indicator used in the European Union (EU), or measures of household expenditure. Some countries, such as Denmark, have centralized databases with detailed information on incomes for all residents.
Many countries, however, do not have this information, either because income information is not sufficiently accurate due to the relatively large share of the informal economy, especially among the poorest, or because such a centralized database simply does not exist.
National household surveys such as the EU Statistics on Income and Living Conditions (EU-SILC) collected annually in each of the member states by the national statistical institutes collect detailed income information but the sample sizes — several thousand households per country — are not large enough to accurately identify spatial pockets of poverty.
National censuses, currently being implemented and analyzed throughout the EU, collect information on each household, but often lack sufficiently detailed income indicators necessary to calculate poverty rates.
Fortunately, identifying the poorest and most vulnerable communities can be achieved through ‘small area estimations’, or simply poverty mapping, whereby household surveys such as the EU-SILC are combined with information from the national censuses to estimate poverty rates at, for example, municipal level, and show these on spatial maps.
The World Bank, in partnership with statistical offices around the world, has produced small area poverty maps for more than 70 countries.
Throughout 2012-2013, the European Commission and the World Bank will seek to work with national statistical institutes to produce more such maps and provide an important tool to better target inclusion resources.
The experience of the recent crisis map of the least developed microregions by the Government of Hungary and the Open Society Institute shows that such maps can be used to increase the allocation of EU funds towards the poorest regions, including poor and predominantly Roma communities.
Identify the gaps in human development outcomes and monitor progress
In 2011, the European Commission/UNDP/World Bank and the EC/EU Agency for Fundamental Rights combined efforts to compile two large household surveys of Roma communities and of non-Roma living nearby in European countries with large Roma populations. Collecting detailed information on employment, education, housing, health, finance, migration, and rights, these rich data sources are already informing national Roma integration strategies in several countries and forming the basis of new analysis and policy advice in, for example, early childhood education and financial inclusion (World Bank reports forthcoming, 2012).
While in the interim such efforts are critically important, the conferences also highlighted how the nationwide sample of the annual EU-SILC survey can be expanded, for example bi-annually, to include extra households from the marginalized communities identified through the poverty maps.
This would have the advantage of using the same EU-SILC questionnaire and the same implementing agencies — the National Statistics Institutes — to simultaneously measure outcomes and monitor progress among both the poorest communities and the general population. As such, it would also allow governments to perform standardized monitoring of progress on inclusion using existing instruments for reporting.
Ensure programmes focus on the most critical areas
The draft guidance document Monitoring and Evaluation of European Cohesion Policy highlights that it is often difficult to demonstrate the value of a policy because programmes frequently focus on spending rather than achieving well-defined results on outcomes, such as improving job prospects, keeping children in school and learning, etc.
Fortunately, there are basic tools — results frameworks — which clearly articulate the results chains by summarizing how the project envisions that inputs (financial and human resources) will translate into specific activities that will in turn lead to specific — monitorable — outputs (e.g. number of unemployed who have received job training), which in turn will contribute to achieving the ultimate desired impacts (results).
To institutionalize their use in Roma inclusion projects and programmes, governments may consider capacity building, requiring their use for funding proposals, and using crowd-sourcing innovations to support transparency and accountability of projects. For example, the World Bank is currently collaborating with the Slovak Government and UNDP to map European Social Fund projects using a mapping tool developed under the World Banks’s Open Aid Partnership programme.
Establish continuous learning and innovation
Often, there are many different policy options that seek to achieve similar policy outcomes. Social policy experimentations provide rigorous evidence on project impacts through randomized counterfactual impact evaluations.
Promoted by the EU PROGRESS facility, and implemented around the globe by governments, civil society, and international organizations, these can identify the most cost-effective interventions and build public support around proven programmes for Roma inclusion.
Additionally, seeking direct community level feedback on interventions can add critical local perspectives for social innovation.
For example, as part of the monitoring and evaluation component of the ongoing European Parliament Roma Pilots, the Directorate General for Regional Policy has provided financing to the World Bank, UNDP, Roma Education Fund, Slovak Governance Institute, and Hungary’s Kiut microfinance program, to design and implement a local system of data collection whereby partner organizations — mainly small NGOs — collect beneficiary level outcome indicators and project feedback that is entered in an online tool for project monitoring accessible by project partners.
There are several practical monitoring and evaluation tools available and several pilot programmes are successfully implementing them. The use of these tools can be expanded and implemented more widely and systematically, often at relatively little additional costs, to achieve better results for Roma inclusion.
*Katarina Mathernova is Senior Advisor at the World Bank; Joost de Laat is Senior Economist at the World Bank. Sandor Karacsony is a consultant with the World Bank and the Open Society Institute.
