B.2 Assessing the UAS policy and strategy
B.2.1 When is the right time to assess?
Policy makers, regulators and in some cases universal service fund administrators have developed UAS strategies and plans that focus on meeting digital UAS goals and targets. Many UAS related strategies are now in their second or third 3- or 5-year cycle, and yet it is not clear whether they have had the desired impact or not. In some cases, there is clear evidence that targets have not been met. In many cases, the ineffectiveness of strategies and plans can be linked to design, process, and deadlines.
Contrary to popular practice, a strategy should not be reviewed at the end. The discovery of the wrong technology choice, the insufficiency of training being provided, or the absence of relevant content to spur uptake only at the end of a project leads to wasted financial and human resources and with limited or depleted funds, to correct errors that could have been addressed early on.
Continuous evaluation and assessment is a key ingredient for successful implementation. This is a key aspect of all UAS projects. It requires that monitoring and evaluation (M&E) take place throughout the life of the project. The outcomes must be communicated broadly to all stakeholders to ensure accountability and transparency and ultimately sustainability.
B.2.2 Assessment fundamentals
Before commencing with the review, the ‘owner’ of the UAS strategy, digital strategy or UAS fund strategy (usually the government ministry responsible for ICT or the fund) should consider the following fundamentals:
Criteria
Description
Timing
It is important that the strategy assessment does not commence too late, or too long after the implementation has commenced. In the advanced stages of strategy implementation, it will become increasingly difficult to make changes in response to challenges identified, it will also increase the cost of the assessment exercise. Outcomes of the assessment need to be integrated into the strategy implementation.
Leadership
Governance arrangements established at the outset should include leadership for the programme. Leadership should be committed to outcomes and thus to the completion of a proper assessment. Lack of leadership is a risk to the project.
Ownership
Governance arrangements established at the outset should include ownership for the programme, this can be joint ownership, but if so, roles and responsibilities must be clearly stipulated. Lack of ownership impacts accountability and is a risk to the project.
Collaboration
The assessment should be a collaborative process. All partners, beneficiaries and relevant policy-makers and regulators must be involved in the process alongside any other relevant stakeholders.
Integrity and transparency
The assessment must be transparent, there must be a desire to identify challenges and risks, and to understand and obtain outcomes that will lead to effective implementation. The integrity of the assessment is key.
Outcomes-driven
The goals and outcomes that were defined early on need to guide the project.
Evidence-based and data-driven
Evidence gathered through the assessment should inform outcomes. It is key that appropriate and good quality data is gathered. It is furthermore important that the right tools to utilise the data are in place, for example mapping/GIS tools, models, etc
Funding
There must be adequate funding for all aspects of M&E built into the project financing. This includes funding for objective and professional strategy assessment which can require a significant commitment of resources
Capacity
There must be staff resources dedicated to M&E, and specifically to strategy assessment and fund assessment, as applicable.
Source: Adapted from A Guide to Making Strategic Assessments (Australian Government, https://awe.gov.au
Checklist: UAS Strategy Assessment
✅ An assessment framework has been published and agreed
✅ The budget has been analysed and the relevant financials have been audited by independent financial advisors.
✅ Each pillar or programme is assessed independently, as well as collectively. The actual outcomes achieved have been measured against desired outcomes.
✅ Agreed upon, credible baseline and evaluation data has been used, including indicators;
✅ A broad range of stakeholders has been consulted (a) on the framework; and (b) on the outcomes of the assessment, including the budget.
✅ The granting process is transparent and bids were advertised publicly.
✅ The outcomes have been published.
NOTE: Has the strategy and/or fund performance been evaluated periodically throughout its existence (continuous assessment/M&E)? Have the reports been made available to the public? Were any adjustments made throughout the strategy period on the basis of evaluation outcomes?
Download checklist