Monitoring and evaluation

Defining monitoring and evaluation

Monitoring and evaluation (M&E) are integral and distinct parts of strategy and implementation. They are critical tools for forward-looking strategic positioning, organisational learning and for sound management. In terms of M&E with and for youth there are two key areas to focus on:

1) Donor agencies’support, encouragement and advocacy for the continued development of international youth development indicators and a set of targets on key areas for youth development: see the Commonwealth/UNDESA case study 18.

2) Direct engagement working with young evaluators, monitoring and assessing interventions: see case study 19 and case study 20.

Young people can be engaged in a variety of ways including designing indicators and methodology, data gathering, report writing and participating in review processes.

Added value of working with youth

  • Involving, training and supporting young people who are the target of a programme to take a lead in monitoring and evaluation can produce more accurate data (often youth respond better to youth).
  • Employing young people and supporting them in roles as volunteers or interns can be cost effective. Young people’s contributions often outweigh the financial implications.
  • If they come from the target community, young people can be important communicators, promoting community support and engagement.

“Participation of young people in evaluations may lead to issues being identified which might otherwise be overlooked. For example, in the evaluation of the Families Orphans and Children Under Stress (FOCUS) programme in Zimbabwe, children and young people identified stigma and sexual abuse as major issues.” Family Health International

The barriers and how to overcome them

Initially, young people, no less than adults, will probably lack technical M&E skills. It can be effective to hold training sessions with adults and young people together in order to simplify and clarify M&E terminology and to build the adult-youth partnerships, which can foster discussion and collaboration.

Some stakeholders will be sensitive about discussing faults or failures with young people, i.e., those perceived to be junior or inferior in status. It is therefore important to emphasise/introduce the skills that a young evaluator can bring, for example, the ability to reach out to other peers and use of local languages that can help minimise social barriers.

 

Valuing young people’s contributions - Photo © Students Partnership Worldwide

Initial stages and indicators

Initial steps

Below are some initial questions you should consider when hiring young evaluators:

  • Is it appropriate for young people to evaluate this programme/project?
  • How can you make contact with young evaluators?
  • Who in your M&E team should they work with? Do they need any training on working with young people?
  • Are you collecting evidence on the benefits of youth participation?

We now turn to some indicators for the three thematic areas of this guide. They offer an initial overview of the key areas to track in relation to youth development in the given sector.

Sample indicators (by thematic area)...

...to be disaggregated by sex, age, and other relevant indicators of exclusion where appropriate

Governance, voice and accountability

  • Budget allocated to national youth council;
  • Diversity and representativeness of youth council/parliament membership;
  • Youth membership of other voluntary civil society organisations;
  • Seats for youth on decision-making bodies (such as committees and councils);
  • Youth voter registration and participation in the electoral process;56
  • Improved accountability mechanisms (internal and external) such as complaints mechanisms and feedback on how ideas have been used.

Post-conflict transitions and livelihoods
  • Improved skills, income, employment/self-employment (including socially excluded groups);
  • Improvements in the sustainability of new or existing economic activities;
  • Improved health (including decreases in sexually-transmitted infections and substance abuse);
  • Enhanced civil society engagement (including reduced crime and violence or a decrease in extremism);
  • Improved social and economic opportunities for young women (which is linked to later marriages and increased personal agency);
  • Increased investments in continuing education by young people and their families.57

Sexual and reproductive health and rights

In relation to MDG 5 and 6: Maternal mortality and reproductive health, combating HIV/AIDs, malaria and other diseases, key indicators include:
  • Reduction in the maternal mortality ratio and a decrease in adolescent birth rates;
  • HIV prevalence among pregnant women aged 15 to 24 decreased;
  • Condom use rate of the contraceptive prevalence rate increased;
  • Percentage of population aged 15 to 24 with comprehensive correct knowledge of HIV/AIDS increased;
  • Contraceptive prevalence rate increased. 

Case studies in this section
We now turn to three case studies demonstrating youth participation in M&E:

18. Country Level Indicators (Commonwealth/UN)

Country Level Indicators

Lack of age-aggregated data and specific, measurable, achievable, realistic and time-bound objectives is a global problem affecting the vast majority of youth plans and programmes.

The Commonwealth Plan of Action for Youth Empowerment 2007-2015 (PAYE) and the World Programme of Action for Youth (WPAY) are framework documents that provide ways forward. The PAYE underpins the work of the Commonwealth Youth Programme (CYP), and is offered as a model strategic plan for ministries of youth, but also calls for mainstreaming across departments. CYP and a number of UN system agencies are collaborating on next steps.

Problems addressed

  • There is a lack of systematic country-level indicators for youth. The PAYE contains rudimentary universal indicators (integrated with the millennium development goals) and is intended to complement M&E frameworks developed at national and regional levels.
  • Governments were initially cautious about measuring progress in youth development.

Objectives

  • To formulate, test and review different approaches to youth development;
  • To define broad, inclusive indicators for monitoring and measuring youth development in the economic, social and political spheres. A number of Commonwealth youth ministers called for a youth development indicators tool for:
    • Setting targets and measuring progress (including progress on the PAYE);
    • Sharpening advocacy for youth development;58
    • Assisting youth mainstreaming within each country, and decision making about allocating scarce resources.

Youth as beneficiaries

Youth development indicators are intended to impact governance, the economy and service delivery in all sectors. Indicators will focus on three key areas: political empowerment, social empowerment and economic empowerment.

Youth as partners

Youth development indicators are intended to employ participatory as well as statistical approaches. Young people are members of the technical advisory committee and will be part of the expert panel.

Process

  • The Commonwealth youth ministers meeting in 2008 endorsed the concept.
  • UNWPAY goals and targets were proposed in October 2008.
  • By February 2009, a number of organisations had revisited and reaffirmed their commitment.
  • High-level discussions on youth development indicators were part of the 47th UN Commission on Social Development.
  • CYP, ILO, UNDESA, UN Dept Statistics and UN Latin America made recommendations on next steps.
  • It is anticipated that by 2012, four Commonwealth countries will have piloted revised indicators.

Results

Development of the indicators is at an early stage, however:

  • PAYE has been endorsed by all 53 Commonwealth countries. There have been public launches in a number of countries including Anguilla, Guyana, Barbados, Solomon Islands, Papua New Guinea, Tonga, Vanuatu and Cook Islands.
  • CYP has used PAYE as the basis of national youth policy development (a majority of Commonwealth countries) and subsequent action planning. Youth has also been mainstreamed into the M&E framework for all Commonwealth Secretariat technical assistance.
  • WPAY has guided national youth policies (most recently Cambodia’s) and regional initiatives such as the African Youth Charter and the Ibero-American Youth Convention.

Lessons learned

  • Successes in national youth policy formulation, including participatory processes, have often been followed by failures not only in M&E.
  • M&E at national level should be combined with efforts to mainstream youth, as stand-alone ministries lack power and resources.
  • Malaysia and Brazil (with UNESCO support) have developed national level M&E frameworks.

Potential challenges

  • Cultural differences in defining youth demographic, and focus and prioritisation of indicators.
  • Gaps in technical understanding and guiding philosophy (paradigms) as well as resources.

 

For further information contact:

Commonwealth Youth Programme,

http://www.thecommonwealth.org/subhomepage/152816/

Commonwealth Youth Forum 7 - Photo © Nia Lole

  • 58. The Human Development Index has been a foundational basis for PAYE, however a simple international ranking index is not the objective of the Commonwealth Youth Development Indicators.
Additional Resources: 

1) Example indicators from the Commonwealth Plan of Action for Youth EmpowermentExample youth Indicators

2) UNDESA on Youth Development Indicators: http://bit.ly/9XMZUk

Youth Engagement Lens: Beneficiaries
Operational Area: Monitoring and Evaluation

Resource: Example indicators from Youth Development Index

The following indicators are adapted from Commonwealth PAYE

Economic indicators

  • Reduce by half the proportion of people living on less than a dollar a day;
  • Increase the percentage of young women and men in formal and non-formal employment by 2015;
  • All governments to enact pro-youth employment policies and incentives by 2010;
  • All sectors/ministries to allocate at least 25% of their total annual budget in support of youth development mainstreaming;
  • Numbers of people receiving entrepreneurship training as part of their formal education;
  • Numbers of young people receiving micro-credit;
  • Numbers of young people undergoing apprenticeships or vocational/business skills training.

Political indicators

  • Human rights as part of school curricula and youth development training;
  • Numbers of young people involved in governance, democracy and human rights education as a) educators, and b) beneficiaries:
  • Youth membership of electoral commissions:
  • Ratification status of human rights instruments.

Social indicators

  • Percentage increase in social sector allocations and spending for young people;
  • Percentage decreases in incidence of youth crime, substance abuse, teenage pregnancy and suicide;
  • Decrease in HIV prevalence amongst youth demographic;
  • Numbers of families benefiting from training (counselling, communication skills and other).

The paradigm: empowerment

Commonwealth youth ministers and heads of government have endorsed the view that:

“Empowering young people means creating and supporting the enabling conditions under which young people can act on their own behalf and on their own terms, rather than at the direction of others. These enabling conditions fall into four broad categories: i) an economic and social base; ii) political will, adequate resource allocation and supportive legal and administrative frameworks; iii) a stable environment of equality, peace and democracy; and iv) access to knowledge, information and skills, and a positive value system.

“...A positive value system cannot be simply taught as if it were another kind of information. A value system is demonstrated, by actions. By their actions, youth leaders and other role models in government, civil society, media and education need to demonstrate that:

  • All young women and men have grounds for self-worth;
  • People of all backgrounds and in all circumstances deserve the respect and understanding of others;
  • Violence is not the way to resolve conflicts;
  • Traditions are often things of value, but are not to be followed blindly;
  • Change is a constant and must be faced with hope and creativity;
  • Consensus can be reached through dialogue and debate;
  • Human rights are to be respected.”
Reaching out to youth - Photo © Nia Lole

19. Youth Empowerment Programme (NAC, Uganda)

Youth Empowerment

Youth-led monitoring and evaluation (M&E) facilitates the design of realistic and practical tools, as well as building transferable skills and ensuring that young people’s input to decision-making is informed and consistent.

The Youth Empowerment Programme (YEP)60 has young people leading field- based M&E as part of their activities on sexual and reproductive health and rights (SRHR), livelihoods and conflict resolution with their peers in schools and communities. Their experiences were discussed and recorded during a youth guidance project workshop in Uganda.

Problems addressed

  • How to support and train young people to take ownership of and successfully implement M&E activities in programmes that involve them;
  • Youth engagement in all aspects of programming and encouraging youth-led analysis.

Objectives

  • To provide young people with skills and support in both quantitative and qualitative research methods.
  • To use M&E data during the course of the programme to improve programme delivery.

Youth as partners

Young people are given the skills to conduct a range of monitoring and evaluation activities.

Youth as leaders

Young evaluators help develop indicators and write final reports and recommendations.

Process

  • Interactive, group-based training on:
    • What M&E is and its purpose in development programmes;
    • Evaluation tools: focus group discussions, life skills checklists, knowledge assessments (particularly on SRHR and livelihoods) and surveys;
    • Monitoring tools: daily log sheets and books, demonstration and replication records, activity/event reporting forms, participant lists and monthly progress reports;
    • Trainees practice the tools by evaluating the training itself.
  • Young people go to the field and are responsible for:
    • Pre-testing and contributing to the design of M&E tools;
    • Monitoring the programme in the field – which involves them keeping daily logs of all activities, with participants disaggregated by age, gender and in- or out-of-school status;
    • Implementing behavioral surveillance surveys and youth knowledge, attitudes and practice surveys.
  • Staff visit the field to: 
    • Give young people feedback on the quality of their M&E;
    • Support data analysis and feed learning back into programme delivery.
  • Young people take part in the annual programme evaluations.

Results

  • Monitoring data has enabled programmes to prioritise who target beneficiaries should be;
  • Young people have increased voice, influence and agency within the YEP.
  • Young people made refinements to tools, such as:
    • Rephrasing of questions, especially in the local language when phrases are not easily translated;
    • Improvements to the behaviour surveillance survey questions, format and guidelines;
    • More user-friendly data collection methods.

“M&E enables young people to experience firsthand the issues affecting development and the impact of development interventions in order to learn best practices to further instigate change within their communities and country and to develop vital skills for their professional development. Young people are an untapped resource primarily perceived as only the target of programme interventions instead of being effective in the implementation of a programme and its monitoring and evaluation." Natalie Newell, M&E co-ordinator, YEP/SPW Uganda

Lessons learned

  • Length of training needs to be increased to a minimum of two weeks. Volunteers must be thoroughly trained about cultural norms, sensitivity and confidentiality.
  • On-the-job training empowers most, and therefore periodic training is vital.
  • Schedule training for particular interest groups, e.g., after 5pm is often best for students.
  • Training should be skills-oriented, practical and conducted in local languages.
  • Standardised training across the country is not always suitable; customise training based on target groups.

Potential challenges

  • Achieving gender balance for focus group discussions in M&E data collection; young women may need to ask permission of male relatives to take part.
  • Volunteers deviating from agreed methods, such as males asking sensitive SRHR questions of females.

For further information contact:

SPW Uganda, http://www.spw.org

 

Gaining vital skills for the future - Photo © Yarri

  • 60. 2010-2012 funding from DFID-CSCF
Additional Resources: 

1) SPW M&E training VPE Initial Training Session #2_17Feb09 (2) for young evaluators

2) SPW Uganda Volunteer SPW UGANDA VOLUNTEER ME GUIDELINES 2009 (2)

3) Shared Learning Network 4 Uganda report: http://bit.ly/90nUck

Youth Engagement Lens: Beneficiaries, Partners
Operational Area: Monitoring and Evaluation

20. Measuring Adolescent Empowerment (UNESCO, Nepal)

Measuring Empowerment

In accordance with UNESCO’s strategy of action with and for youth, which strives to involve young people as equal partners in all aspects of project planning, implementation and evaluation, the Section for Youth collaborated with Youth Initiative to monitor and evaluate a pilot on ‘Breaking the poverty cycle of women’ in two districts of Nepal. Peer-group monitoring and evaluation was expected to generate a better reflective mechanism to evaluate progress from the recipients’ viewpoint and to contribute to the capacity-building of youth organisations active in social development. Youth Initiative was responsible for carrying out the M&E which was simultaneously conducted in Bangladesh, India, Nepal and Pakistan.

Problems addressed

  • Lack of outreach by peers during M&E data collection.
  • Greater understanding of literacy and life skills for poor and excluded ethnic groups is needed in Nepal.
  • Lack of awareness on violations of adolescents’ rights, including trafficking and gender-based violence in Nepal.

Objectives

  • To contribute to the capacity-building of youth organisations active in social development, including community learning centres.
  • To involve young people as equal partners in all aspects of project planning, implementation and evaluation.
  • To contribute to mid-term evaluation of adolescents’ empowerment project.

Youth as beneficiaries

One thousand and fifty adolescent girls (10 to 19 years) and 1,000 boys involved in the project were affected by the evaluation findings.

Youth as leaders

M&E led by a youth organisation: Youth Initiative. Eight adolescents (four boys and four girls) acted as peer evaluators. The evaluation was designed, conducted and reported on by a youth-led organisation.

Process

  • Youth Initiative selected to carry out M&E at an early stage of the whole project;
  • Orientation of young evaluators, including lessons learned from previous cycle;
  • M&E training and discussion;
  • Eighteen village development committees were visited for M&E activities, whereby focus group discussions (using random sample techniques), individual interviews (with adolescents, parents and others) and observations (on hygiene, transport, energy use, health, use of public space etc) were conducted;
  • Comparisons were made between adolescents within and outside the project.

Results

“Before we never heard about outcome and impact level (results). Now we would like to focus on outcome and impact level.” Young evaluator

  • Skills and knowledge of young evaluators.
  • General M&E findings:
    • The project was found to have an impact on discussion, decision-making and volunteering on HIV/AIDS and public health issues (fire risks, sanitation, nutrition). However, broader SRHR knowledge was lacking.
    • Legal literacy (rights) classes were popular with adolescent girls, but few were aware of legal advice services.
    • Findings supported recommendations to target illiterate out-of-school youth, and produce more practically oriented and attractive materials.
    • Overall quality of the programme depended on facilitators’ levels of motivation, especially given their inadequate salary.

Lessons learned

  • Young evaluators can uncover new issues, such as the need for establishing separate classes for newer and older project participants.
  • Providing incentives for young beneficiaries or survey respondents is vital, such as offering refreshments.
  • Facilitators may have access problems in remote rural localities, so providing logistical support (bicycles in this case) is crucial.
  • It is important to ensure there are minimal gaps between evaluation cycles.

Potential challenges

  • Seasonal weather patterns (in this case, the rainy season of June to August) can disrupt data collection;
  • Fear of open exchange of views due to conflict (in this case Nepalese-Maoist insurgency);
  • Shy or uninterested respondents. A large number of active respondents fell outside the planned age-range, and were 20 to 25 years old.

For further information contact:

info@youthinitiative.org.np or see www.youthinitiative.org.np

Exploring gender dynamics in Nepal - Photo © Students Partnership Worldwide

Additional Resources: 
Youth Engagement Lens: Beneficiaries, Partners, Leaders
Operational Area: Monitoring and Evaluation