Evaluation
Since the Seed Alliance’s inception, AFRINIC, APNIC and LACNIC have supported the implementation of evaluation mechanisms that allow them to communicate a clear message to their communities about the impact of their regional Grants and Awards programs, as well as the effectiveness of the support this funding provides to the program recipients. Evaluation efforts were conducted at the grant recipient level, at the regional program level and at the Seed Alliance level, as follows:
- Grant recipients: Technical reports submitted by grant recipients to each regional program were reviewed and the evaluation criteria applied on score cards to assess their impact. Score cards are visible when hovering over the Supported Projects map and when following each project information sheet.
- Regional program: Every regional program applied internal mechanisms according to performance review tools from their respective organizations to assess program performance.
- Seed Alliance: The Results Assessment Framework (RAF) allowed to monitor the outcomes of the program, based on a set of defined indicators. Additionally, IDRC commissioned an External Evaluation.
Of the evaluation efforts conducted, is worth highlighting:
Improvements to Selection Processes
Since the beginning of the Seed Alliance, evaluation and communication concepts have been incorporated in all systems and processes to collect datasets for further analysis. Basic statistics are collected through the online application system. Improvements to how data is collected through the applications process have been implemented based upon the feedback received from applicants. The online applications forms have been reviewed through a simple, but effective, continuous improvement mechanisms. Analysis of previous data collected was also conducted to improve the application and selection processes. A geographic distribution and summary of statistics is available under the Applications Received section.
Continuous Improvements to Reporting Templates
FIRE Africa, FRIDA and ISIF Asia continued to improve the template in line with the recommendations made by a consultant, hired by FIRE, who was in charge of the technical and financial reports generated by the Grants and Awards recipients. The consultant’s remit was to ensure:
- That objectives, milestones and achievement timelines were clearly highlighted
- That the project managers clearly understand what was expected of them in the reports.
Information sessions about how the report templates should be used was provided to funding recipients from 2012 to 2015, which helped them to gain the required expertise to create adequate reports.
Regional Evaluation: FIRE and ISIF Asia approach to evaluation
Back in 2011, when negotiations with IDRC started for the initial grant that enabled the Seed Alliance to establish itself, the three regional programs agreed to take part in an innovative approach to evaluation:
- Through a mentoring process, using Utilization-Focused Evaluation (U-FE) as a framework for evaluation
- Using Research Communications (ResCom) as a framework to support communication strategies, linked to research findings at both the program and recipients level
This work built on the experience gained by the ISIF Asia program during its involvement with the Developing Evaluation Capacity in ICTD (DECI) project in 2010. DECI was funded by IDRC.
Mentoring for FIRE and ISIF Asia was offered through a funding partnership. The costs of this process were shared, and used funds from the IDRC and Sida grants for ISIF Asia. Financial support from the DECI-2 project (also funded by IDRC) was received for FIRE. AFRINIC and APNIC signed separate MoUs with the DECI-2 team, which defined the terms and conditions of the work to be undertaken and outlined the funding for the activities to be implemented. In practice this meant that ISIF Asia benefited from an additional CA$ 35,000 for evaluation and research communications mentoring from the DECI-2 project. This arrangement enabled ISIF Asia to make considerable savings, as well as to receive world-class support during the development of the evaluation frameworks and communications for development strategies. See Full list of U-FE and ResCom activities conducted from 2012-2015.
- FIRE Africa: For FIRE Africa, the focus of the DECI-2 support was to mentor the FIRE Africa team in U-FE and ResCom, to elicit uses and Key Evaluations Questions (for the U-FE component) and communication audiences and objectives (for the ResCom planning). DECI-2 mentors recommended the the team should focus one of the evaluation uses on the RAF data demands (i.e. an accountability use) and another on learning, methods, or process.
- ISIF Asia: In the case of the ISIF Asia projects mentored by DECI-2, it helped funding recipients to design and conduct their own evaluation plans and communications strategies. The focus of the DECI-2 support was to mentor three Grant recipients in U-FE and ResCom, to elicit uses and Key Evaluations Questions (for the U-FE component) and communication audiences and objectives (for the ResCom planning). The mentoring was conducted using a combination of online and face-to-face mechanisms. Three organizations that received funding during the 2014 Grants round where selected to receive DECI-2 support:
- CIIAG (Cook Islands)
- Nazdeek (India)
- Operation ASHA (Cambodia)
Regional Evaluation: FRIDA’s Evaluation approach
The original plan was that Seed Alliance funds were also going to support the U-FE & ResCom activities for the FRIDA program. However, LACNIC made the decision to incorporate a different evaluation approach and rescind the MoU with the DECI-2 project. Funds not used for U-FE & ResCom were returned to the DECI-2 project and reallocated by LACNIC to support their own evaluation work for the FRIDA program.
Harmonizing the DECI-2 and Sida Approaches
A technical report template that detailed project activities and lessons learned was developed using a framework that tried to reconcile both IDRC and Sida requirements. This template was reviewed by DECI-2 mentors. DECI-2 mentors tried to organize a visit to Sida offices during the first term of 2014 to have the opportunity to present more on their approach and try to find mechanisms to support the harmonization of the two evaluation approaches, but unfortunately it was not possible to confirm Sida’s staff availability.