Digital Skills Assessment Guidebook

Acknowledgements

The Digital Skills Assessment Guidebook was prepared by Chris Coward, Principal Research Scientist and Director, Technology & Social Change Group (TASCHA), Stacey Wedlake, Research Coordinator and Analyst, TASCHA and Allison J. Anderson, Research Assistant, TASCHA, under the supervision of Susan Teltscher, Head, Capacity and Digital Skills Development Division, ITU Telecommunication Development Bureau, with inputs from Halima Letamo, Capacity and Skills Development Officer, ITU. The work was carried out under the overall supervision of Eun-Ju Kim, Chief a.i., Digital Knowledge Hub Department, ITU Telecommunication Development Bureau. The publication was edited by Anthony Pitt, consultant to ITU, with graphic designs developed by Elena Stankovska-Castilla, Associate Capacity Development Officer, ITU and Justine Depoisier, ITU intern. Special thanks goes to the ITU Publication Production Service and the ITU branding teams for the desktop publishing and cover design.

Disclaimers

The opinions, findings and conclusions expressed in this publication do not necessarily reflect the views of ITU or its membership.

All reasonable precautions have been taken by ITU to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader.

ISBN

978-92-61-31091-2 (Paper version)
978-92-61-31101-8 (Electronic version)
978-92-61-31111-7 (EPUB version)
978-92-61-31121-6 (Mobi version)

© ITU 2020

International Telecommunication Union
Place des Nations
CH-1211 Geneva, Switzerland

Some rights reserved. This work is licensed to the public through a Creative Commons Attribution-Non-Commercial-Share Alike 3.0 IGO license (CC BY-NC-SA 3.0 IGO).

Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited. In any use of this work, there should be no suggestion that ITU endorse any specific organization, products or services. The unauthorized use of the ITU names or logos is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the International Telecommunication Union (ITU). ITU is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. For more information,
please visit https://creativecommons.org/licenses/by-nc-sa/3.0/igo/

Foreword

In an increasingly digital world, information and communication technologies (ICTs) play a key role as development enablers that can facilitate countries’ capabilities to reach the Sustainable Development Goals (SDGs). Added to that, the COVID-19 global health crisis that emerged in early 2020 dramatically underlined – as perhaps nothing else could have – the vital importance of meaningful connectivity to people’s livelihoods, employment, health and well-being, education, and social participation. Yet, according to ITU statistics, almost half the world population is still not using the Internet.

One of the main barriers to greater uptake is a lack of skills required to use digital products, services and content, particularly among populations in poor countries and among marginalized groups. Our reliance on digital networks during the pandemic, and the corresponding rapid growth in the production and delivery of a huge range of new digital goods and services, emphasizes the critical importance of rapidly improving access to digital skills training, particularly for vulnerable nations and communities.

Taking advantage of the many opportunities offered by ICTs, while at the same time coping with a highly digitalized economy, demands different types of digital skills. The changing technological landscape – including ICT-enabled education, digitized communication and media tools, automation in the home and workplace, increased collaboration through social media, and growing data processing capabilities – requires an ever-evolving skills base.

Digital skills are fast becoming essential for people to navigate ordinary day-to-day activities such as using a mobile phone to transfer money to family members via digital financial services, using the Internet for remote education classes and to research, prepare and deliver coursework, and acquiring basic skills for staying safe online. In the workplace, digital skills are required for an ever-growing number of jobs, and even in traditionally manual sectors like agriculture, digital applications are beginning to make inroads, with a corresponding requirement for some level of digital literacy. Developing a digitally skilled population calls for the involvement of different institutions: universities need to undertake research and determine requirements; government institutions need to develop appropriate policies; training institutions need to deliver relevant digital skills training; and the private sector and civil society organizations need to support digital adoption and use.

As necessary digital skills continue to grow in number and complexity, countries are in need of well-structured approaches to identify current digital skills levels and manage future requirements.

To assist Member States in this process, ITU’s Digital Skills Assessment Guidebook has been designed to serve as a comprehensive, practical step-by-step tool for national digital skills assessments. The guidebook can be used to determine the existing supply of a digitally skilled cohort at a national level, to assess skills demand from industry and other sectors, to identify skills gaps, and to develop policies to address future digital skills requirements. It is designed for use by policy-makers and other stakeholders, such as partners in the private sector, non-governmental organizations, and academia.

This guidebook draws on, and complements, the ITU Digital Skills Toolkit published in 2018. Whereas the toolkit was designed to help policy-makers develop national digital skills strategies and roadmaps, this guidebook focuses on helping them identify national skills gaps and requirements, which can then be addressed through targeted digital skills development policies and strategies. In other words, the results of a digital skills assessment exercise can serve as a concrete and necessary input into the national policy-making process.

This guidebook also serves as a contribution to the ITU-ILO Digital Skills Campaign, which is part of the ILO Decent Jobs for Youth initiative launched in 2016. The campaign aims to boost youth employment through digital skills by incentivizing and encouraging partners to commit to deliver digital skills training to young people, particularly in the developing world. So far, pledges to train more than 15 million people have been made through the campaign. This guidebook is one of ITU’s contributions to the knowledge products that support the campaign, assisting stakeholders in the design and delivery of effective training policies and programmes to ensure that young people are equipped with the right skills to take their place in the digital economy.

I trust the ITU membership will welcome this guidebook as an indispensable tool, which will facilitate their work and encourage them to undertake the important task of assessing their national digital skills readiness for improved policy-making.

Doreen Bogdan-Martin
Director
Telecommunication Development Bureau, ITU

Table of Contents

Acknowledgements

Foreword

List of tables and figures

Introduction to the Digital Skills Assessment Guidebook

Chapter 1: Review of existing work on national digital skills assessments

Introduction

Approaches for assessing digital skills levels

National-level digital skills assessments

Understanding current and future digital skills needs

Conclusion

Chapter 2: Assessment of current digital skills levels

Step 1: Assemble team

Step 2: Decide what to assess

Step 3: Collect and analyse data

Step 4: Data dissemination

Conclusion

Chapter 3: Understanding current digital skills needs and gaps

Step 1: Administer desk review

Step 2: Choose methods

Step 3: Focus on key sectors

Step 4: Conduct a gap analysis

Step 5: Communicate the digital skills supply, demand and gap to wider audiences

Conclusion

Chapter 4: Forecasting future skills requirements

Step 1: Understand future technology trends

Step 2: Conduct anticipation exercises

Step 3: Make strategic decisions

Conclusion

Chapter 5: Conclusion

References

Appendix

List of tables and figures

Tables

Table 1: International digital skills assessments

Table 2: Methods to assess current skills needs

Table 3: Identify existing data sources

Table 4: Who, what and how

Table 5: Comparison of data-collection methods

Table 6: Identify key sectors and skills

Table 7: Forecasted trends and their impact

Table 8: Review of development plans

Table 9: Guide to data gathering

Table 10: Factors that affect demand

Figures

Figure 1: Overview of assessment approach

Figure 2: Overview of assessment approach

Figure 3: Overview of how to forecast future digital skills requirements

Introduction to the Digital Skills Assessment Guidebook

Information and communication technologies (ICTs) continue to grow in number, sophistication and complexity. These technological advancements drive changes in labour markets, creating and increasing the need for countries to develop a digitally skilled population in order to expand economic participation, drive economic development and compete in the global economy. To accomplish this, policy-makers need to identify current digital skills levels and determine and anticipate current and future digital skills requirements in their countries.

The International Telecommunication Union (ITU) works to strengthen capacity in the field of ICT and digital technology development, as articulated through resolutions of its Plenipotentiary Conference and its World Telecommunication Development Conference (WTDC). Objective 3 (Output 3.3) adopted by WTDC (Buenos Aires, 2017) mandates ITU to elaborate human skills development policies and guidelines in telecommunications/ICTs for its members, especially developing countries, in order to assist them in enhancing their human and institutional capacity and setting up their national digital strategies. In this connection, ITU has developed a Digital Skills Toolkit (ITU, 2018a) that can be used by both developing and developed countries as input into their national digital skills strategies. Having a national digital skills strategy before embarking on a digital skills assessment approach will provide a helpful foundation for the work outlined in this guidebook (OECD, 2016, p. 74). Other organizations have developed and discussed frameworks that can be used to undertake national digital skills assessments; however, there is no practical document or guide that provides a step-by-step approach which countries can follow to assess their national digital skills.

The level of technological development in each country will influence its digital skills levels and requirements. In developing countries, the levels of and requirements for digital skills, as well as the processes for assessment or evaluation of these skills, will be different from those in developed countries. It is therefore important for ITU to put in place mechanisms to support these efforts and initiatives.

In the light of the above, this document provides a practical guide that will assist countries, especially in the developing world, to undertake national digital skills assessments in order to determine their current digital skills levels as well as their current and future digital skills requirements. The roadmap is designed for use by policy-makers and other stakeholders, such as partners in the private sector, non-governmental organizations and academia, who may need to undertake skills assessments at the national level.

The guidebook comprises the following five chapters:

• Chapter 1 reviews existing work on national digital skills assessments, discussing the advantages and disadvantages of digital skills assessment tools that can be employed as part of a national-level assessment.

• Chapter 2 provides a step-by-step approach for assessing the current supply of digital skills.

• Chapter 3 provides a step-by-step approach for determining a country’s current level of demand for digital skills and identifying any skills mismatch.

• Chapter 4 addresses future technology trends and how to conduct digital skills anticipation exercises.

• Chapter 5 contains a summary and conclusions.

• In the Appendix, readers will find additional resources and tools which policy-makers and stakeholders can use in developing ideas suited to their specific country or context.

Chapter 1: Review of existing work on national digital skills assessments

Introduction

This chapter reviews existing work on digital skills frameworks and corresponding digital skills assessment tools, discussing the advantages and disadvantages of each tool as part of a national assessment. In considering the various assessments, the discussion covers how digital skills assessments incorporate the types of and levels of digital skills established in ITU’s Digital Skills Toolkit. Digital skills assessments are broken down into three categories, namely self-assessment, knowledge-based assessment and performance-based assessment, and the role and purpose of each of these types of assessment is discussed. In addition, the chapter reviews digital skills assessments at the national level, examines the advantages and disadvantages of national assessments and provides examples of countries where digital skills assessments have been undertaken. Finally, a variety of methods available for countries to conduct an assessment of their current digital skills requirements and gauge their current skills mismatch are considered. In later chapters, the report builds upon this knowledge to support a nation’s understanding of its future digital skills needs.

Types of digital skills

Before considering how to evaluate digital skills, it is first necessary to understand what digital skills are and how they are categorized. Digital skills, sometimes also called digital competences or competencies, encompass the “knowledge and skills required for an individual to be able to use ICTs to accomplish goals in his or her personal and professional life” (Commission on Science and Technology for Development, 2018, p. 4). Given the pace of change in technology and digital work opportunities, digital skills denote a broadening spectrum of skills, which changes over time. Digital skills include a “combination of behaviours, expertise, know-how, work habits, character traits, dispositions and critical understandings” (Broadband Commission for Sustainable Development, 2017, p. 4). They thus include not only technical skills but also cognitive skills as well as non-cognitive soft skills such as interpersonal skills and communication skills.

People use digital skills to engage with a wide variety of digital technologies: desktops, laptops, mobile phones and other Internet-enabled or “smart” devices. Some of these skills may be device-specific, for example using a keyboard or mouse, while others may be more universal, such as using effective web search terms.

Levels of digital skills

Digital skills might be better understood by classifying them into proficiency levels. ITU’s Digital Skills Toolkit divides skills levels into three categories: basic, intermediate and advanced (ITU, 2018a, p. 5). Placing these skills on a continuum provides a pathway for study. For example, a person typically needs to achieve basic skills before moving on to intermediate or advanced skills.

Basic

Basic digital skills provide the foundation for using ICTs. In some communities, these skills are applied entirely on mobile devices. Elsewhere, mastering basic skills entails interacting with several types of device. Basic skills include:

• Using a keyboard or touchscreen to operate a device

• Using software to download apps and create documents

• Completing basic online transactions such as making Internet searches, sending and receiving e-mails, filling out a form.

These skills may be acquired through formal training, through self-teaching or from a peer. Basic skills make it easier for people to communicate with others and to access and use public and private services (ITU, 2018a, p. 6).

Intermediate

Intermediate skills enable people to use digital technology in “meaningful and beneficial ways” (Broadband Commission for Sustainable Development, 2017, p. 27). In contrast to the more universal basic skills, a person will need different sets of intermediate skills depending on their goals and needs, and their vocation. For example, depending on the type of job in which they are employed, a person may need digital graphic design skills in addition to word processing (ITU, 2018a, p. 6). As technology changes and grows, the number of skills that fall under “intermediate” continues to evolve and expand. In the recent past, colleagues could only collaborate virtually by passing text back and forth via e-mail; now, work teams can collaborate using video, text and voice on a wide variety of platforms. People generally learn intermediate skills through formal education, from peers or through self-study (e.g. online tutorials).

Advanced

ICT specialists use highly specialized, advanced skills in professions such as computer programming, software development, data science and network management. Like intermediate skills, advanced skills and jobs that require them continue to grow in number and scope. Some of the newer skillsets include:

• Artificial Intelligence (AI)

• Big data

• Cybersecurity

• Digital entrepreneurship

• Internet of Things (IoT)

• Virtual Reality (VR).

People acquire advanced skills most commonly through advanced formal education, but other pathways exist, such as coding boot camps or online training. (ITU, 2018a, p.6)

While this report is founded on the notion of basic, intermediate and advanced skills, it is important to be aware that some digital skills frameworks also divide skills into different sets of proficiency levels.

Digital skills frameworks

Before discussing approaches to assessing digital skills levels, we need to touch on the importance of digital skills frameworks. It is beyond the scope of this guidebook to assess the digital skills frameworks themselves, but it is important to understand the major digital skills frameworks because many (although not all) assessment approaches are constructed around a particular framework. A digital skills framework provides a means of categorizing and organizing the complexity and range of digital skillsets. Frameworks create a common language and sometimes prescribe proficiency levels or learning outcomes (Vuorikari & Punie, 2019, p. 1). Digital skills frameworks are used to inform policy, instructional planning, and assessment tools (ITU, 2018a, p. 7). This section gives a brief overview of four of the major frameworks currently used for policy and measurement.

Digital Competence Framework for Citizens (DigComp)

The European Commission’s Joint Research Centre first published the Digital Competence Framework for Citizens (DigComp) in 2013, and updated it in 2017. The framework includes five competence areas: (1) information and data literacy; (2) communication and collaboration; (3) digital content creation; (4) safety; and (5) problem solving. The framework also specifies the knowledge, skills and attitudes required for each competence, with eight proficiency levels (Carretero, Vuorikari & Punie, 2017). Developed for and employed mostly by Member States of the European Union (EU), DigComp has been used as a foundation for developing strategy, education programmes and assessment tools in over 20 countries in Europe and around the world (Kluzer & Pujol Priego, 2018, p. 8).

Digital Literacy Global Framework (DLGF)

In order to make DigComp more applicable in developing countries, the Digital Literacy Global Framework (DLGF) developed by the United Nations Educational, Scientific and Cultural Organization (UNESCO) used empirical studies from economically diverse countries. The goal was to create a framework to serve as the foundation for indicator 4.4.2 of Sustainable Development Goal (SDG) 4: “Percentage of youth/adults who have achieved at least a minimum level of proficiency in digital literacy skills”. The development of the framework included a technical review of more than 40 global digital literacy frameworks, which were then mapped against DigComp. In the end, DLGF supplemented the existing DigComp framework with two additional competence areas, namely “devices and software operations” and “career-related competences”, and one additional competence under the “problem solving” competence area, namely “computational thinking” (Law, Woo, de la Torre & Wong, 2018, pp. 23–25).

Digital Skills to Tangible Outcomes (DiSTO)

Another major framework for understanding digital skills was developed by a group of researchers at the London School of Economics, led by Alexander van Deursen and Jan van Dijk, among others, and has been incorporated into the Digital Skills to Tangible Outcomes (DiSTO) project that “develops and improves measures and models of people’s digital skills, digital engagement and outcomes of … ICT use” (London School of Economics and Political Science, n.d.-a). Van Deursen and van Dijk have been conducting extensive research for many years under the banner of “Internet skills”. Their model uses the term “digital media” to describe “computers, mobile telephony, digital television and the Internet” (van Deursen & van Dijk, 2015). The framework organizes skills into four domains:

• Operational skills - the skills to operate digital media

• Formal skills - the skills to handle the special structure of digital media such as menus and hyperlinks

• Information skills - the skills to search, select and evaluate information in digital media

• Strategic skills - the skills to employ the information contained in digital media to reach personal or professional goals.

The original 2009 work has been updated and expanded several times, and now includes a survey tool adapted and tested in Australia, Brazil, Chile, the Netherlands, the UK, the United States (US) and Uruguay. The methodology tested self-assessments with performance-based tools, allowing researchers to proxy items based on actual performances. Thus, it produces valid results in terms of skills level (van Deursen, van Dijk & Peters, 2012).

New Essential Digital Skills Framework

The United Kingdom (UK) Department of Education’s new Essential Digital Skills Framework is designed to support adults in enhancing their digital skills. The framework is focused on “the skills needed to safely benefit from, participate in and contribute to the digital world of today and the future”. It encompasses five categories of skills: communicating, handling information and content, transacting, problem solving, and being safe and legal online (Essential Digital Skills Framework, 2018). According to a blog post, the framework was originally created in 2015 by Go ON UK, a digital skills non-profit organization, and was updated in 2018 through consultation with a steering group of technology companies, banks, business consortia and civil society and in collaboration with over 400 cross-sector organizations (Ryder, 2018). The skills in the framework inform the UK Consumer Digital Index, a basic digital and financial skills survey of 9 000 UK residents that was last conducted in 2019 (Lloyds Bank, n.d.).

Discussion

Countries take a varied approach to engaging with assessment frameworks. Some countries build their own skills frameworks, while others rely on frameworks developed for broader use, such as the more than 20 countries that are employing DigComp. Still other countries rely on additional frameworks, not reviewed here, such as the Certiport International and Digital Literacy Certification (IC3) programme or the International Computer Driving Licence (ICDL) (Law et al., 2018, pp. 30-33). This guidebook is informed by all the various approaches adopted, since different ways of categorizing digital skills lead to different approaches to measuring digital skills.

Approaches for assessing digital skills levels

This section focuses on reviewing and evaluating existing digital skills assessment approaches. These tools were identified from the above digital skills frameworks, academic articles and research produced by international organizations. Digital skills assessments were chosen for their potential to be deployed or adapted for organizational or country-level use. We define and explore the three types of assessments: self-assessments, knowledge-based assessments and performance-based assessments. A full table of assessment tools can be found in the Appendix. The question of how these can be used to measure a country’s current digital skills levels will be covered in Chapter 2.

Self-assessments

Self-assessments measure digital skills by asking participants to rate their own level of knowledge, ability, confidence or usage. Questions tend to use pre-defined scales such as Likert (e.g. scale 1 - 5), multiple-choice, or true or false. For national skills measurement efforts, self-assessments are typically administered in the form of a survey. Since self-assessments are relatively quick to take, they are often paired with other types of assessment. The length of such assessments and total skills covered can vary.

The main advantage of self-assessments is that they are the easiest and least costly to create, deploy and score (ITU, 2018c, p. 41). Self-assessments can also cover an almost unlimited range of skill types, from basic to advanced. Additionally, self-assessments allow a person to reflect for themselves upon their own strengths and weaknesses (Kluzer & Pujol Priego, 2018, p. 35). However, this type of assessment comes with significant disadvantages. One is that people often find it hard to assess their own skills and abilities with any degree of accuracy (Litt, 2013, p. 620). Demographic factors such as gender, income and prevailing social groups also skew the way a person assesses his or her skills (ITU, 2018c, p. 42). Despite these drawbacks, self-assessments remain popular.

Examples of self-assessment surveys

Self-assessment surveys can easily be attached to existing surveys or other large sampling measures. ITU and Eurostat, the EU’s statistical office, are examples of organizations that incorporate self-reporting surveys as part of their large data-collection processes. The total number of skills-specific questions is lower than with other methods, since the surveys also cover other topics. In ITU’s ICT Households questionnaire, question HH15 asks about nine ICT skills, mostly computer-based, covering basic and intermediate skills, with one computer programming question (ITU, 2018b). The rest of the survey covers other ICT access and use questions. Eurostat has developed a digital skills indicator based on DigComp. A person reports whether they performed various activities under four competence areas: information skills, communication skills, problem-solving skills and software skills. A person receives a rating of “no skills”, “low”, “basic” or “above basic” (Eurostat, n.d.-a). The use of Eurostat’s digital skills measure has been limited to European countries.

Other assessments are deployed as standalone surveys. Digital Skills to Tangible Outcomes (DiSTO) was originally constructed and validated in the UK and the Netherlands (van Deursen, Helsper & Eynon, 2014). More recently, through partnerships, the surveys have been used in Australia, Chile, Brazil, Uruguay and the US as part of targeted research projects (London School of Economics and Political Science, n.d.-b). DiSTO uses a Likert scale and covers online and mobile skills. Another survey developed as part of a research project is the ICT Skills Indicator (ISI). This online survey, using a Likert scale, asks about advanced ICT skills of people aged 16 – 35 years in small island developing states (Redeker & Sturm, 2019).

Ikanos, created by the Basque Government, gives suggestions for local ICT training opportunities. It also provides a digital profile that identifies the digital knowledge, skills and attitudes that a professional needs to have for different occupations, including ICT skills essential for the occupation and sector concerned as well as necessary transversal digital skills. These profiles address current digital skills needs based on the World Economic Forum’s (WEF) Industry 4.0 and DigComp (Kluzer & Pujol Priego, 2018, p. 80).

Knowledge-based assessment

Knowledge-based assessments test skills using questions about factual or procedural knowledge (Kluzer & Pujol Priego, 2018, p. 35). The assessment results are usually presented as a set of responses to multiple-choice questions, and may produce a more accurate picture of abilities than self-assessments (Kluzer & Pujol Priego, 2018, p. 35). Advantages of knowledge-based assessments are that they can test skills at lower cost and with less effort than other testing methods. However, they sometimes focus too much on features of the technology itself and not on how to use digital skills to solve a real-life problem (Sparks, Katz & Beile, 2016, p. 12). For example, such assessments will test knowledge on what an e-mail icon looks like rather than how to send an e-mail with an attachment. Most assessments focus on desktop or laptop skills, but some of the skills could transfer to mobile devices and need to be conducted in a controlled environment for the highest degree of accuracy.

Performance-based assessment

Performance-based assessments measure actual performance on digital skills in realistic scenarios using tools such as browsers and word-processing software (Kluzer & Pujol Priego, 2018, p. 35). Some of the assessments are conducted in a laboratory or software simulation, while others use the computer’s existing software. Performance assessments are the most valid method of measuring digital skills (ITU, 2018c, p. 42). However, these tests are the most expensive to conduct and most time-intensive for users, which makes large-scale implementation difficult (Kluzer & Pujol Priego, 2018, p. 35). Such tests are often deployed in school settings where processes for national testing already exist.

Three of the tests reviewed are run worldwide, and participating countries can compare their data within and across countries (Table 1). These assessments are conducted in cycles of varying frequency and target different age groups.

Table 1: International digital skills assessments

National-level digital skills assessments

This section offers several examples of countries where digital skills assessments have been undertaken at the national level. These examples showcase different approaches to conducting national digital skills assessments based on countries’ unique goals and activities.

Uruguay: Digital Skills to Tangible Outcomes (DiSTO)

In 2017, the Research Group on Uruguay, Society and Internet (GIUSI) within the Department of Communication of the Catholic University of Uruguay conducted a survey to understand how Uruguayans access and use the Internet. Part of the survey contained self-assessment digital skills questions based on the DiSTO framework. The nationally representative survey randomly called residents aged 18 and older who have a mobile number. Respondents were asked to rate themselves on a scale from 1 to 5 in respect of their ability to do 13 tasks without any assistance, including: “I know how to remove friends from my contact lists” and “I know how to create something new from images, songs and videos that I find online”. The study found that, although most Uruguayans are online, there were significant differences across ages and education levels in their reported abilities to accomplish certain online tasks (Dodel & Aguirre, 2018).

Peru: Programme for the International Assessment of Adult Competencies (PIAAC)

The Organisation for Economic Co-operation and Development (OECD) Programme for the International Assessment of Adult Competencies (PIAAC) is an assessment focused on cognitive and workplace skills needed for successful participation in 21st-century society and the global economy. The most prominent survey it conducts is the Survey of Adult Skills, which measures proficiency in information-processing skills like literacy, numeracy and problem solving in technology-rich environments (defined as “the capacity to access, interpret and analyse information found, transformed and communicated in digital environments”), and looks at how adults use these skills at home, at work and in community engagement (Directorate for Education and Skills, n.d.). Proficiency is measured according to a 500-point scale, divided into levels. The survey is conducted in around 40 countries, which allows comparison between countries (OECD, n.d.).

Earlier instances of the Survey of Adult Skills (aged 16-65) were conducted in OECD countries between 2011 and 2012, but the scope has since been expanded to include other countries. In 2017-2018, OECD ran the survey in Peru. The survey contains performance-based and self-assessment questions. The results showed that, on average, adults in Peru had lower levels of proficiency in the three areas than adults across OECD countries. In addition, within this low proficiency in information-processing skills, there was a high variability in scores, revealing large disparities in educational attainment and parental background. The survey also showed that the skills mismatch in Peru was the severest among any of the countries participating in the survey (Directorate for Education and Skills, n.d.). Participation in the survey gives Peru detailed information on the country’s digital and other skills and allows comparison with OECD and non-OECD countries.

France: Pix

Pix, run by the French Ministry of Education, is an online platform developed in 2016 that assesses and certifies digital competences based on DigComp (Pix—Cultivez vos compétences numériques, n.d.). Any French speaker can assess his or her skills using a performance-based assessment. At the end of the test, the person receives a digital skills profile with targeted recommendations for future learning. Digital skills can be officially certified by taking a test “under strict examination conditions” (Kluzer & Pujol Priego, 2018, p. 99). Employers and schools can also create accounts to coordinate assessment of employees’ and students’ digital skills. By 2020, Pix will replace the current secondary school Internet certificate (Vuorikari & Punie, 2019, p. 7). Even though Pix is currently only available in French, it is constructed on an open-source platform that can be adapted to different languages, frameworks or skillsets (Laanpere, 2019, p. 13).

Kenya: Basic Education Curriculum Framework (BECF)

Kenya’s Ministry of Education created a national-level education framework, the Basic Education Curriculum Framework (BECF), that partially maps onto the DigComp framework. The BECF is “an outcome of extensive stakeholder engagement, a national needs assessment study, deliberations from a national curriculum reform conference and several benchmarking studies” (Kenya Institute of Curriculum Development, 2017, p. 3). The BECF is focused on digital literacy core competence, and targets K-12 students. The curriculum includes an online digital literacy platform with materials that support teacher professional development. It uses competence-based skills assessments using formative and summative methods (Law et al., 2018, p. 43). Advantages to this approach include involving a broad spectrum of stakeholders, creating the framework at the national level and aligning the framework with national and regional development strategies, as well as targeting one group (K-12) for assessment and training. The main disadvantage is that there is no separate digital skills measurement tool.

Discussion

The examples cited highlight different approaches to conducting national digital skills assessments and serve as a reminder that countries will choose varying approaches according to their goals and activities. For instance, the Uruguayan implementation stood out in contrast to the other examples as the only national-level self-assessment survey. As mentioned previously, this assessment approach is less valid than performance-based assessments. However, it makes sense to take the opportunity to ask digital skills questions as part of a larger, national survey on Internet use.

Besides the differences in the assessment approaches themselves, the examples discussed also reveal differences in terms of assessing adults or youth. The Peruvian PIAAC and Uruguayan DiSTO assessments test the abilities of adults, whereas the French Pix and Kenyan BECF focus on youth. Looking more broadly at adults has advantages insofar as it will yield a greater sense of the skills currently used in work and society. However, assessing the skills of graduating secondary school students is useful because they will soon be part of the workforce, and the testing can take place within established educational systems.

Understanding current and future digital skills needs

This section reviews a variety of methods that can help countries gauge their current digital skills needs and skills gap at the national level. These established methods allow countries to leverage their own government, academic and industry expertise to identify their unique requirements. The same methods will serve to identify current skills needs and assess skills gaps in the immediate and the near term. Anticipating future skills needs will be addressed later, in Chapter 4. Current skills needs assessments can be conducted annually, while future skills anticipation exercises are generally conducted less frequently and might cover a time span of up to 10 years or more (OECD, 2016, p. 42).

Methods to assess current digital skills needs

The International Labour Organization (ILO) and OECD created an overview of potential methods for understanding skills needs for the G20 Employment Working Group. These methods are applicable to all countries and can be adapted to focus only on digital skills, as opposed to/instead of all employment skills (ILO & OECD, 2018). They can be used to gauge the current demand for digital skills across the economy or within a specific sector. In this section we adapt the ILO and OECD overview for the purposes of understanding current skills needs.

Table 2: Methods to assess current skills needs

This guidebook will look in more detail at how to choose between the different methods and what stakeholders to include in Chapter 3.

Skills surveys

In addition to the methods outlined above, there are publicly available survey data that quantify labour-force skills requirements and skills mismatches for some countries. One such dataset is issued by Eurostat. Eurostat recently published new experimental data on skills mismatch, which reports over-qualification rates and job mismatches by sector and field of education. It does not specifically include ICT professions in the sector analysis, but it does include computing in the field of education. The data are experimental and only for EU countries, but both the data and the methodology are publicly available and could be adapted for other countries (Eurostat, n.d.-b).

Additionally, there are survey data produced by PIAAC-TRE, which conducts a survey about people’s jobs and what skills they use at work. The survey asks respondents if they have the skills required for their job, and whether they need more training. It includes questions on digital skills, such as “What level of computer use is/was needed to perform your job/last job?” and “Do you think you have/had the computer skills you need/needed to do your job/last job well?”. These data are publicly accessible; however, they are mostly limited to OECD countries (OECD, n.d.).

Conclusion

This chapter has explored previous research on classifying digital skills and assessing current digital skills levels and requirements. Digital skills were classified into three proficiency levels: basic, intermediate and advanced. We will use these classifications throughout the guidebook to identify what skills a country wants to assess. Next, the chapter reviewed types of digital skills assessment and highlighted different countries’ approaches to assessing current skill levels. Lastly, various methods to assess a country’s current skills needs were examined. As we will show in more detail in later chapters, a country can choose a combination of methods based on available data and expertise. Some countries have already conducted a baseline assessment of digital needs, whereas others are embarking on efforts to understand their digital skills supply and demand for the first time. This guidebook is designed to assist countries in their approach, whatever their starting point; but it is not meant to be prescriptive in advocating any one approach.

Chapter 2: Assessment of current digital skills levels

As discussed in Chapter 1, several approaches exist for assessing a country’s current digital skills levels or supply of skills. Each country’s particular capabilities, existing processes and needs will determine which approach is best to adopt. This chapter outlines the steps for selecting the appropriate digital skills assessment approach for your country (see Figure 1). First, it addresses how to manage the assessment process, which includes choosing a governance model and selecting stakeholders. It then goes on to review how to identify your country’s existing data and resources, consider the demographic focus, and choose an approach. The last two steps cover data collection, analysis and dissemination. This step-by-step guide is not meant to be rigid. Countries should adapt it to their specific needs.

Figure 1: Overview of assessment approach

Step 1: Assemble team

Conducting a national digital skills assessment takes considerable time and resources. Creating a structure to manage the process will help guide the work through each of the steps – measuring the supply of skills, understanding the demand for skills (Chapter 3) and forecasting future needs for skills (Chapter 4).

Choose a governance model

Creating a strong governance structure early on will help you collect the data you need in order to understand your country’s digital skills levels. There are multiple ways to approach governance of a national digital skills assessment. The three main governance models for skills assessments are the “policy model”, the “independent model” and the “hybrid model” (OECD, 2016, p. 77). Whichever approach you choose, the lead agency or group should have extensive experience in managing large-scale data-collection and data-analysis projects.

• Policy model: Governance is managed by those who will use the information in their decision-making for a specific purpose, such as the ministry of labour, the ministry of education, or the ministry of digital economy, ICT or telecommunications. However, a risk of this model is that the assessment elaborated may be so narrowly focused on one policy that it is not useful for other stakeholders.

• Independent model: Governance is led by an agency that is independent from the users of the information, such as a university or a national statistical office. The use of the skills assessment for policy is not predefined, but the assessment can be seen as a “general tool for policy-making”. One risk of this model is that the final output may not fully address all of the requisite characteristics for policy-making, so the work should be scoped carefully.

• Hybrid model: The hybrid model is a combination of these two approaches to governance. In this model, a skills assessment exercise might be led by a government ministry but involve close collaboration with public employment services or other governmental or non-governmental organizations.

Choose the approach that makes most sense in relation to your country’s resources and existing approaches to governance. If your resources are more limited, the policy model will help keep the assessment narrowly focused. However, digital skills are inherently cross-cutting and relevant to the work of almost every ministry. An independent model or hybrid model will enable broader use of the assessment results.

Engage stakeholders

After choosing a governance model, identify and convene a group of stakeholders. Given that digital skills affect every aspect of work and life, we recommend that a broad, representative group of stakeholders be included in the process of considering which digital skills assessment to undertake. You will continue to involve these stakeholders in work to understand current skills needs (Chapter 3) and forecast future digital skills requirements (Chapter 4). This section, adapted from ITU’s Digital Skills Toolkit, points to some of the stakeholders that might be considered, but the list is not exhaustive and will vary from country to country. If you have previously worked with stakeholders on a national digital skills strategy, consider reconvening that same group.

Government agencies and regulatory bodies

Many government agencies or ministries may already be involved in some type of skills development or assessment efforts (British Council & ILO, 2014, p. 20). Even if not specifically related to digital skills, these may be useful when evaluating existing data and institutions to use in the digital skills assessment. Consider the following:

• Digital economy/ICT/telecommunications

• Labour/workforce/industry development, including small and medium-sized enterprises (SMEs)

• Education, including education boards

• Statistical offices

• Human resource development

• Rural development

• Regional/subnational governments

• Other sectors utilizing ICTs/digital technologies (e.g. agriculture, finance, trad, transport, etc.)

Educational institutions

In most countries, all levels of schooling support digital skills acquisition. Bringing together representatives from all levels (e.g. primary, vocational training institutes, and university levels), as well as from non-traditional education, will give a more complete picture of the current education environment. Technical education programmes may have particular insights into supply and demand for different types of digital skills.

Private sector

Broad engagement from the private sector will bring an important perspective on what digital skills the workforce currently uses on the job. These stakeholders will become even more significant when it comes to ascertaining skills needs and future requirements (Chapters 3 and 4). Invite participants from the main industries in your country, from multinational corporations and small businesses. SMEs, start-ups and the ICT ecosystem will also provide important perspectives on emerging advanced skills. Chambers of commerce and trade unions will not only bring a broader understanding from their members but can also be key allies in advancing work.

Civil society

Civil-society organizations (CSOs), including non-governmental organizations (NGOs) and community-based organizations (CBOs), contribute important perspectives in two main areas: connection to underrepresented groups, and experience in non-formal digital skills training. Underrepresented groups in society are often equally or more at a disadvantage in terms of digital skills acquisition. CSOs can often also access those working in the informal sector, which accounts for a large part of most emerging economies (World Bank, 2019a, p. 94). When assessing nationwide skills levels, it will be very important to consider how to reach these groups in order to provide an accurate picture.

Additionally, CSOs and public libraries often provide digital skills training for adults as well as other types of continuing education, and thus have first-hand experience of the general skills abilities of populations outside formal schooling or the workforce. Make sure to include CSOs that have connections to people working in the informal economy, who may well be omitted from national labour statistics or skills initiatives based in workplaces.

Discussion

After identifying stakeholders, the lead agency will decide whether to engage these groups through formal collaboration or through an informal, ad-hoc process (OECD, 2016, p. 91). Formal collaboration may take place through a ministry advisory board or on the basis of a previously created Digital Skills Charter.1 An informal process can take the form of a working group or roundtable, and works best with a clear objective and realistic timeline.2 Establishing these structures will help with the next steps of identifying existing capabilities and processes and ultimately selecting an assessment approach.

Step 2: Decide what to assess

Inventory of existing data and resources

Before deciding what data to collect, you need to find out what data already exist and how they are collected. For example, your country’s national statistical office or education ministry may collect digital skills data through national surveys or school testing. Knowing about these sources will help establish whether you need merely to adapt the existing approach in line with new national priorities, or to create an entirely new method. First, conduct a desk review of existing data and resources in your country. Cast a wide net. An existing national survey or test may perhaps not contain a digital skills component, but adding to or building on existing processes will greatly decrease the time and resources needed. The number of data sources available will vary from country to country. Some possible data sources may be large-scale educational testing or population surveys.

Potential sources include:

• Educational demographic data

– Undergraduate and graduate enrolment and graduation rates in computer science, engineering, technology

– ICT apprenticeship programme outcomes

– Number of university professors in computer science, engineering and technology, lecturers with PhDs

– Schools with ICT curricula

• Other educational data

– Secondary and university exit exams

– Secondary-school educational testing

– Adult skills testing

– Vocational schools’ certification exams

• Surveys

– National population surveys

– Labour force surveys

– Students (secondary to university), IT apprenticeships, recent graduate surveys.

Not every country will possess every type of data. Some countries will display substantial data gaps and lack resources, and may not have existing robust statistical offices (Adinde, 2019, p. 37). Working with stakeholders, use the tool below to help identify possible existing data sources and processes. This tool is adapted from the OECD Skills Strategy Diagnostic Report: Italy (OECD, 2017, p. 233).

Table 3: Identify existing data sources

Decide the “who, what and how” of the assessment

Based on existing resources and your stakeholder input, consider who and what to assess and how often to collect data. For example, if you have an engaged education ministry that already conducts yearly national testing among secondary students, you may decide to test secondary-school students. However, based on stakeholder input, if it is more valuable to test adults of workforce age, you may need to create a new assessment tool. Whoever you decide to test, work with your stakeholders to collect specific data on underrepresented groups.

Use the tool below to define the requisite characteristics of your assessment approach. These characteristics include the who (target group to be assessed), what (skills you will be assessing) and how (manner in which the assessment will be conducted - frequency, administrative logistics and implementation timeline).

Table 4: Who, what and how

Now that you and your stakeholders have an understanding of the existing data landscape in your country and your desired assessment characteristics, you can move on to the next step: collecting and analysing data.

Step 3: Collect and analyse data

Compile existing data

Once you know who and what skills levels you want to assess, return to your data source table from Chapter 2. Look for two pieces of information: what supporting data your country already collects within the scope of your assessment, and what existing data-collection methods you can exploit. Let’s assume, for example, that your stakeholder group decided to measure the basic and intermediate digital skills of all secondary-school students and the advanced digital skills of university graduates. Your country has an existing national exam that all secondary-school students take before graduation. This exam does not contain questions about digital skills. However, it constitutes an opportunity to incorporate an additional assessment focusing on digital skills. For university graduates, you can use your country’s existing data on what degrees are awarded to see what percentage of degrees are awarded in technology and engineering. Now you will be able to choose the assessment approach that best fits your needs. (More information in regard to assessing advanced ICT skills can be found in a section below.)

Choose a data-collection approach

The decisions concerning the desired focus of the assessment will directly inform what approach to choose for collecting data. The table below compares the two main methods of collecting digital skills assessment data: population survey and national testing.

Table 5: Comparison of data-collection methods

The method you choose will influence what type of assessment you undertake: self-assessment, knowledge-based or performance-based. The Recommendations on Assessment Tools for Monitoring Digital Literacy within UNESCO's Digital Literacy Global Framework suggest combining two or three types of digital skills assessments. If a nationwide performance-based evaluation is not feasible, you can combine a nationwide skills self-assessment with a more limited performance-based test. If you have more extensive resources, combining a self-assessment with a performance-based evaluation allows a comparison between a population’s perceived and actual skills abilities (Laanpere, 2019, p. 13).

After you decide which approach fits your needs and capability, decide whether to use or adapt an existing assessment tool as detailed in the Appendix. You can also see which tools offer internationally comparable data. If you need internationally comparable data, your tool selection and leeway to adapt resources is much more restricted. After making your selection, review your “who, what and how” as decided upon by your stakeholders. Will the data collected clearly meet your needs? Revise your plan as necessary to fill the data gaps and make sure that important considerations have not been left out (Mackay, Gallo, Husch & Rak-sakulthai, 2015, p. 35).

Assessing different ICT skills levels

If you have decided to assess advanced ICT skills, you might need to collect additional data. As mentioned in Chapter 1, in contrast to the more generally applicable basic and intermediate skills, advanced ICT skills are highly specialized. Some self-assessment surveys may have one or two questions that cover advanced ICT skills such as coding, but these typically do not drill down to the level of detail required to assess the applicability of these skills for employment.

Knowledge-based and performance-based testing primarily covers basic and intermediate skills, too. However, by contrast, the Finnish ICT Skills Test is an example of how to integrate advanced ICT skills into a knowledge-based test. The test consists of multiple-choice questions on information security, programming, database operations, information networks and server environments (Kaarakainen, Kivinen &Vainio, 2018, p. 359).

As the main provider of formally trained advanced ICT specialists, universities will be your main source of the additional data you will most likely need in order to assess advanced skills levels. The most useful data will be graduation rates in ICT fields, technology, engineering, mathematics and computer science as compared with non-technology degrees. You can also look at data such as the number of technology-related university researchers and lecturers with PhDs. In order to train future technologists, teachers need to be highly skilled as well (African Capacity Building Foundation, 2017, p. 29). Using employer surveys or roundtables can help identify people trained outside the university environment. Identify employers that hire or need employees with advanced ICT skills. Then ask if they have unmet skills needs, and in what specialties. As you become more familiar with your country’s advanced ICT skills needs, you will be able create more specific assessments targeted at those specific skills.

Data analysis

After data collection, you can now move on to analysing the data. In Step 2, you identified the entity that will conduct the analysis. The responsible entity should be one that has highly skilled, experienced statisticians, such as a government statistical office or university. The output of the analysis should be accessible for other researchers to pursue unofficial analysis. During the analysis, pay attention to the previously identified underrepresented groups. Comparing these groups’ skills levels with those of the population as whole may reveal skills inequalities. You will apply the collected data immediately in dissemination efforts (Step 4), in understanding current needs (Chapter 3) and in anticipating future requirements (Chapter 4).

Step 4: Data dissemination

In view of the significant effort involved in collecting data, it is worth sharing the data with others who can put them to good use. Accordingly, identify with whom you will share the data. Of course, you will first want to share the results with your stakeholder group. Next, aim for a wide distribution with institutions and policy-makers in the areas of education and training, employment and community-based organizations. Consider sharing the data publicly, online, for easy access and to leverage potential for others to discover and use them.

Consider how you will want to share the data. For example, if you intend to share raw data, know what file formats are most widely used by professional and amateur statisticians in your country. Reports, presentations, webinars and websites may also be needed, depending your audiences. During the dissemination effort, collect reactions and feedback to inform future data collection.

Conclusion

Countries’ digital skills levels vary depending on a variety of factors such as technological development, existing infrastructure, private-sector investment and higher-education institutions. Chapter 2 has described the requisite steps to assess the current supply of digital skills levels in a country. First, the governance model and stakeholders to manage the assessment process are selected. Then your stakeholder group decides what skills to assess based on your country’s existing resources and needs. All this preparation sets the stage for robust data collection and analysis. The prior decisions around the desired focus of the assessment directly influence the data-collection approach. Finally, you can share the data with your stakeholder group and other wider audiences. The understanding of the supply of your country’s skills thus obtained will lay the foundation for the work described in the next two chapters, on determining current skills needs and forecasting future requirements.

Endnotes

1 For more details on creating a Digital Skills Charter, see ITU, “Digital Skills Toolkit”, p. 24 (ITU, 2018a).

2 For more details on stakeholder consensus building for developing skills assessments, see OECD, “Getting Skills Right”, pp. 86-91 (OECD, 2016).

Chapter 3: Understanding current digital skills needs and gaps

After a country’s assessment of existing digital skills is complete, thus affording it a greater understanding of the supply of digital skills, it can turn attention to determining its current level of demand for skills. The process of assessing current digital skills requirements will also expose a country’s national skills mismatch. A skills mismatch is defined as the gap between an individual’s job skills and the demands of the job market, including being over-skilled or under-skilled for current market opportunities (OECD, 2016, p. 12). An understanding of the current skills supply and demand will inform a country’s policies aimed at reducing the skills mismatch.

This chapter outlines the practical steps that a country can take to conduct a demand assessment. The focus here is on current digital skills requirements as they are revealed by industries and other stakeholders in individual countries. The next chapter will consider how to forecast longer-term digital skills needs based on countries’ decisions about how they wish to engage in global trends.

Figure 2: Overview of assessment approach

We recommend the following approach to assess a country’s skills demand. First, given that this exercise follows on from the assessment of the supply side (Chapter 2), it is recommended that the same system of governance and coalition of stakeholders be used to lead and direct this phase of assessment. A select group of researchers should be identified from the lead agency or organization, or a consultancy commissioned, to conduct the assessment. Once this research team is in place, it should undertake a desk review of available international and national data sources to gather existing information on skills requirements and skills mismatch. In this phase, the research team might also choose to identify the country’s sectors of importance for deeper focus. Next, it will need to choose the appropriate method for data gathering based on specified goals, infrastructure and resources. Finally, the research team will need to carry out data gathering, followed by analysis and broad reporting of the outcomes.

The approach is centred on addressing the following key questions:

Step 1: Administer desk review

A focused desk review using a variety of sources will provide an overview of the available statistics, as well as the current literature and discussion on the demand for digital skills in the country. Although not all countries will have available data from all the sources mentioned here, a number of potential international and national-level quantitative and qualitative data sources that might form part of the desk review are presented below.

International datasets and studies

Several international organizations provide resources on current skills needs at the regional and country level, in the form of qualitative and quantitative data. For example, ITU and ILO publish numerous studies on current needs for the workforce and addressing skills mismatch. They have a wide geographical range of reporting, although they do not cover all countries. The World Bank and LinkedIn have collaborated to provide a dataset on industry employment shifts, talent migration, industry skills needs and skills penetration that covers 140 countries (World Bank, 2019b). Additionally, the ILO statistics database, ILOSTAT, provides data for most countries on industry employment share. The share of employment by industry can be a useful measure to inform decisions on which industries to include in the skills demand assessment as a high priority (ILO, n.d.). Finally, a number of organizations produce studies on current digital skills requirements by sector. For example, WEF’s work in connection with the Fourth Industrial Revolution identifies demand for specific skills in different industry sectors (World Economic Forum, 2016, p. 22).

National datasets and studies

As discussed in Chapter 2, national datasets and studies should be consulted in this phase. Some areas of data that may already be collected by existing processes and agencies include the following information:

• Region-specific or local assessments of skills demand

• Labour-market trends, such as flows in and out of employment, by occupation and sector

• Vacancy surveys

• Existing sector studies from chambers of commerce, chambers of industry and trade unions.

Academic research

Finally, national universities or colleges, as well as national research institutions or think tanks, may have completed research on skills demand or skills mismatch. These reports can be included in the initial desk review.

Step 2: Choose methods

Once Step 1 is completed, the research team should choose the appropriate methods to respond to data gaps uncovered in the desk review. In Chapter 1 of this guidebook, some of the potential methods for determining current skills demand were outlined. This section builds on that background and provides more details on how to select the appropriate approach in the light of a country’s unique resources and constraints. A holistic approach is recommended whereby more than one method is used to identify digital skills needs, because each exercise has its own strengths and weaknesses. Within this approach, the methods might also include a combination of qualitative and quantitative data.

Qualitative methods

Qualitative methods can provide a holistic review and allow direct user engagement in the assessment. Industry roundtables, focus groups or subject-matter expert interviews can be carried out with strategic stakeholders, such as major industry leaders, sector skills councils, sector bodies, policy stakeholders and leaders from universities and technical schools.

These focus groups and interviews are useful for discussing the types and levels of digital skills required by different sectors and the skills gaps that exist. On a practical level, this could include asking some of the following questions:

The benefit is that qualitative methods may yield a deeper understanding of the issues at hand and enable the discussion to follow the nationally selected digital skills assessment framework; the disadvantages are that this approach might be non-systematic or non-representative and could prove time-consuming.

Quantitative methods

A number of quantitative methods can be employed, such as surveys and forecasting models. The quantitative methods vary in complexity and cost, and here we address some of the advantages and disadvantages of the different methods.

Surveys can be conducted at many different levels: industry-wide, within firms, looking at vacancies or among a subset of the population, such as recent graduates. Surveys conducted with employers or employees can speak directly to perceived skills requirements. This could enable sectoral needs to be identified, including detailed information on capabilities, competences and required skills. Employee surveys can detect a skills mismatch by asking workers whether they feel over- or under-qualified for their current job. However, response rates for employer and employee surveys are often low and large samples are needed to obtain robust data, which means that this approach might be expensive. Another kind of survey at the employment level are vacancy surveys that focus on why posted vacancies remain unfilled. Additionally, graduate surveys could be used to provide information for improving the quality of training programmes. These are relatively low cost and easy to execute, but the disadvantage is that they are confined to workers’ early market experiences. These surveys can be based on the skills identified in the supply-side nationally selected digital skills assessment framework.

Discussion

While there are many different methodological solutions that the stakeholders can pursue, the most appropriate methods will be geared to uncovering data that were not readily available in the desk review and that can be compared with the data collected in the digital skills assessment. A number of the methods outlined in this section serve not only to consider current digital skills needs, but also to see what is already known about the digital skills mismatch. For example, vacancy studies highlight current digital skills needs, but also flag mismatches between the skills people possess and those needed in order to meet the requirements of particular jobs.

Step 3: Focus on key sectors

A further approach consists in conducting a deeper dive into the digital skills requirements of key sectors as part of a national skills needs assessment. The first step in this approach is to identify the main sectors of importance in the country, including both traditional key sectors and emerging high-growth sectors for the country. There are several metrics which might be useful for identifying key sectors, including a sector’s share of GDP or share of employment and a sector’s growth potential. The chosen sectors of importance can be targeted in a deep dive. Next, identify what the major occupations are within those sectors, using the same datasets or through consultations with sector experts. After identifying the sectors of interest, focus on gathering more detailed data in the sectors concerned, using a mix of the qualitative and quantitative methods discussed earlier in the chapter. In this phase, it is important to consider both small and larger firms in the sectors. The following table can support the exercise.

Table 6: Identify key sectors and skills

Step 4: Conduct a gap analysis

After completing the data gathering, conduct a gap analysis to gain an understanding of any mismatch between the nation’s current digital skills supply and its current digital skills demand, using information gathered in the steps laid out in Chapters 2 and 3. Some skills mismatch information is readily available in data compiled as part of the desk review, such as vacancy studies and information gathered from employers about skills mismatches, while other data collected needs to be analysed in order to gauge the skills mismatch.

Below are some potential ways of examining the data in order to identify digital skills gaps:

• For basic and intermediate skills, compare the results of the skills supply with the required skills levels identified by partners. Depending on the assessment framework chosen, this could entail comparing levels of different competences or composite scores. The difference between the levels or scores required by partners and the outcome of the supply-side assessment defines the skills mismatch.

• Compare skills mentioned in vacancy surveys explaining why positions have gone unfilled with outcomes of supply-side skills assessments.

• Compile information from sectoral studies, both surveys and qualitative research, about the difficulty of recruiting appropriate candidates.

• Review any employee surveys and find out if they feel over- or under-qualified for jobs, and in what digital skills areas.

• Assess graduation rates for specialized digital skills fields, as well as average growth rates in particular fields of study over a period of time, as compared with employment rates for specialized digital skills fields, ensuring that education is responsive to skills supply and demand (OECD, 2016, p. 24).

Discussion

Understanding a country’s digital skills mismatch is an important step towards informed policy-making. While there are many different approaches that may be adopted to understand a country’s level of digital skills mismatch, we recommend a broad approach, to include as many different methods and data points as possible, subject to any constraints. It is also recommended that the discussions and surveys of current skills needs be tailored to the digital skills assessment chosen in Chapter 2, for ease of comparison.

Step 5: Communicate the digital skills supply, demand and gap to wider audiences

Once the assessments are complete, develop a communication and reporting strategy. The first step in formulating a dissemination strategy is determining to whom assessment information is to be communicated. This information has numerous applications in the areas of employment, education and training, and policy-making, so we therefore recommend broad distribution of the results to policy-makers, stakeholder groups, educators, employers and individuals (ILO & OECD, 2018, p. 19). In addition to the stakeholders who are part of the coalition, distribute outcomes to the industries, organizations and educational institutions that you included in the data-gathering process. Also include government ministries and, while this assessment is focused on national-level requirements, consider distribution to regional or local governments and agencies. Furthermore, the data are not only useful for government entities, but also for civil society and social partners.

Next, decide how information will flow between the assessment team and stakeholders, considering the best way to communicate findings to different stakeholders (Mackay et al., 2015, p. 36). Different audiences will be interested in different pieces of information and will likely engage with the data through different formats, including: written reports, websites, conferences/events/seminars, social media and traditional media outlets like newspapers, magazines, radio and TV, among others (ILO & OECD, 2018, p. 19). Finally, as a part of the communication plan, collect feedback to inform future assessment work.

Conclusion

Skills mismatches are costly for a nation as they can increase labour costs, lower productivity and slow the adoption of important technological innovation. While skills mismatches are in some ways inevitable, there are methods for understanding where these mismatches occur which serve to inform future policy and decision-making. This chapter describes an approach to understanding skills needs and gaps: administering a desk review, choosing methods, staying focused on key sectors, and conducting a gap analysis. The availability of monetary and human resources will affect a country’s choice of methods. Many of the methods identified in this chapter can be utilized for forecasting future digital skills requirements, too. Lastly, the question of how to communicate the findings to wider audiences was covered in more depth. In the next chapter we will discuss how to conduct digital skills anticipation exercises.

Chapter 4: Forecasting future skills requirements

Globally, rapid changes are taking place in the labour market due to technological changes, as well as globalization and worldwide demographic change. These changes affect all aspects of people’s lives, from agriculture and education to health, and are especially relevant as they relate to changing national digital skills requirements. Each country will be affected by the ongoing changes in different ways, but nations seeking to understand future skills needs should be aware of several overriding technological developments, such as artificial intelligence (AI), big-data analytics, cloud computing, Internet of Things (IoT) and robotics. These new digital technologies have the potential to change the future of work through digital entrepreneurship, freelancing and offshore services. Additionally, future digital skills requirements will be affected by trends towards “green jobs”,[1] such as “smart” electricity grids, “smart” transport systems and “smart” buildings for improving environmental performance.

This chapter will plot how to understand future trends in work and jobs and how to conduct skills anticipation exercises. It will then discuss how countries can consider making different choices about how to engage with these trends, based on their national development plans, the role of the formal and informal economy in their country, and their country’s unique needs and competitive advantage. The chapter also includes case studies on how countries approach the task of identifying future digital skills requirements.

Figure 3: Overview of how to forecast future digital skills requirements

Step 1: Understand future technology trends

Several international organizations, such as ITU, ILO, WEF, the World Bank and OECD, produce frequent studies on the future of work and the future of jobs, often with a sharp focus on understanding needs in the digital transformation space.[2] The various reports reviewed below can serve as a starting point for gaining a greater understanding of global technology trends that might inform future digital skills needs. A focused desk review with these sources, plus any local or regional sources, will allow stakeholders to understand anticipated global technology trends that will affect their nation’s economy. Stakeholders can then discuss what impact the trends are likely to exert on their own country and sectors.

ILO: The Future of Work and the informal economy

ILO’s Global Commission on the Future of Work started in 2017 and its landmark report Work for a Brighter Future is broadly concerned with the ways in which new technologies will affect how people work and their social protections (ILO, 2019). The Commission includes representatives from developed and emerging economies. They focus on the ways in which people are and will be entering into non-standard forms of employment, and the opportunities brought by technology, as well as the concerns for workers on digital platforms. Though the Future of Work initiative does not conduct comprehensive projections of technology or industry trends, it touches on many of the global technology trends, such as how demand will continue to increase for industrial robots, new forms of telework and ICT-mobile work, as well as green jobs. Updates to the Commission’s research and trend reporting is ongoing, and it covers both developed and developing countries in its work.

More than 60 per cent of the world’s labour force is employed in the informal economy, the overwhelming majority in emerging and developing countries. ILO’s Women and Men in the Informal Economy report paints a statistical picture of where the informal economy is most present and whom it includes (ILO, 2018). Understanding the magnitude of and trends within the informal economy will support the creation of strategies and interventions to facilitate the transition of workers into the formal economy.

ITU

The ITU Digital Skills Toolkit helps countries by providing guidelines to develop a digital skills strategy. It is intended for policy-makers as well as partners in the private sector, non-governmental organizations and academia. Its overarching aim is to facilitate the development of a comprehensive digital skills strategy at country level, and provide an initial categorization of different digital skills levels. The toolkit notes that digital skills are essential in opening the door to a wide range of opportunities in the 21st century. Countries that implement comprehensive digital skills strategies ensure that their populations have the skills they need to be more employable, productive, creative and successful while ensuring they remain safe, secure and healthy online. Critically, digital skills strategies need to be updated regularly to respond to the emergence of new technologies and their impact on the digital economy and digital society. (ITU, 2018a).

ITU also publishes the Digital Skills Insights, an online publication which puts together scholarly articles with a focus on the impact of digital transformation on capacity and skills development (ITU Academy, n.d.). The publication seeks to provide a body of knowledge that will facilitate academic research and innovation, exploring the linkages between emerging technologies and capacity development. It features current and new thinking that will contribute to informed policy debates and decisions among policy-makers and regulators, as well as helping the private sector to anticipate and plan for human capital requirements and skills development in order to remain competitive in a rapidly changing ICT environment.

WEF: Future of Jobs

In 2016, WEF produced its first Future of Jobs Report, which included data from its Future of Jobs Survey (World Economic Forum, 2018). The report identifies emerging job skills needs in nine economic sectors and 15 developed and emerging economies. The report is not a skills assessment framework and does not specifically focus on digital trends, but it can inform a country’s decision-making about how global trends will affect current and future digital skills needs. The most recent Future of Jobs Report includes a survey of companies in 20 countries that make up 75 per cent of the world’s GDP, as well as regional-level analysis focused on labour-market trends across economic sectors and regions. The regional-level analysis included over 80 countries. The report projects technologies that companies will likely adopt in the near term, provides examples of new roles needed as well as roles that will become redundant in the future, and highlights digital skills that are likely to be in demand in the coming five years. It also projects industry-, country- and regional-level needs. WEF updates the report each year. Another WEF project, the Reskilling Revolution, is designed to help leaders in the private and public sectors to create and pilot reskills initiatives (World Economic Forum, n.d.).

World Bank: 2019 World Development Report: The Changing Nature of Work

The World Bank has been publishing the World Development Report since 1978. The 2019 report uses the Human Capital Index (HCI) to predict that in countries with lower human capital investments the future workforces will be only one-third to one-half as productive in comparison with countries with a healthy and educated population. In particular, the report recommends increasing investments in early childhood education to best prepare people for the changing, future labour market. The report repeatedly emphasizes that countries need to invest in the human capital of their citizens to increase their chances of success in the global market (World Bank, 2019a). HCI scores are available for 157 countries (Human Capital Project, n.d.).

The report notes that technology changes the workforce in three ways: the geography of jobs, through changing global value chains; how people work (specifically the gig economy); and the importance of transferable technical skills to succeed. The section on the informal work sector emphasizes the need to move people to formal private employment. Specific analysis on the future of work emphasizes that there is always a degree of uncertainty in predicting future trends, but governments should focus efforts on job creation and “protect the vulnerable while still encouraging employment” (World Bank, 2019a, p. 31).[3]

OECD: Going Digital Toolkit and Measuring the Digital Transformation

OECD’s Going Digital Toolkit helps countries assess their current state of digital development and to understand broad trends in digital (OECD, 2019b). The toolkit provides data corresponding to access, use, innovation, jobs, society, trust, market openness, growth and well-being for OECD countries and eight additional countries, including some developing countries. It is not intended to be an index, but to provide key indicators for policy-makers to monitor their digital transformation and its impacts.

Building on the toolkit, OECD’s report Measuring the Digital Transformation: A Roadmap for the Future gives a broad picture of technology trends and, though not directly related to measuring digital skills, details how to measure the impact of the digital transformation on economic sectors, health systems and labour-force productivity and workers. It also details how to measure digital infrastructure and access to ICTs (OECD, 2019c). OECD released the toolkit in early 2019 and plans to develop corresponding resources in 2020.

McKinsey Global Institute (MGI)

McKinsey & Company’s internal think tank focuses on understanding the evolving global economy. MGI has done and continues to produce research addressing the impacts of automation on the workforce. One example is MGI’s study Jobs lost, jobs gained: Workforce transitions in a time of automation, which examines how technological innovation and automation, such as AI and robotics, will reduce the demand for some jobs, change the skillset required for others, and create new opportunities for employment growth through 2030 (McKinsey Global Institute, 2017). The report considers the future of work in selected countries.

Documenting forecasting trends

The above research covers a broad reach across many sectors and countries and is continually updated. The resources below are narrower in focus, but may be helpful for certain countries. You may also find more specific resources from local universities and think tanks. Your stakeholder group may also possess additional resources.

Use the table below to log trends from the above-listed sources that are most pertinent to your country. Understanding these trends can inform the anticipation exercises that will take place in Steps 2 and 3 of this chapter.

Table 7: Forecasted trends and their impact

Discussion

Global technology trend assessments are important because skills needs change rapidly as a result of technological transformation and shifts in labour-market needs. Not every country will engage with each emerging trend. Countries need to understand how the changes might affect their key economic sectors. Armed with this knowledge, policy-makers can predict what digital skills needs will emerge from these changes and prevent costly imbalances between future skills supply and demand.

Step 2: Conduct anticipation exercises

Countries will make different choices about how to engage with emerging trends, based on their own national development plans, demographics and other changes, economic sectors of importance, and competitive advantage. Step 2 sets out how countries can conduct anticipation exercises based on their existing strategies and economic sectors of importance, taking into account the trends that are covered in Step 1. In Step 3 we will speak more about what factors can guide strategic decision-making.

Given that the anticipation exercises are a continuation of the skills assessment work described in Chapters 2 and 3, we recommend using the same set of stakeholders to lead and direct this phase of work, which will include:

• A desk review of national development plans and other internal factors that will guide the anticipation exercises

• Data gathering to determine future digital skills needs in economic sectors and industries of importance to the country.

While anticipation exercises can be carried out for any time period, the most common approach countries adopt is considering a five to 10-year outlook.

The anticipation exercise is centred on addressing the following key questions:

Conduct a desk review of development plans

Many countries elaborate national development plans that articulate a vision and goals for a country’s future growth in response to key trends and changes in the macro environment. There might be multiple national development plans created by different agencies.

The stakeholders should review all existing national development plans, and consider what economic sectors are mentioned and the skills necessary to meet the development plans’ goals in the future. Countries can also consult their national digital skills strategies to ensure they are targeting and incorporating the same strategic digital skills needs.[3]

Below is a table from ITU’s Digital Skills Toolkit that can assist in the process of identifying targets for future digital skills requirements.

Table 8: Review of development plans

Gather data to understand existing economic sectors of importance

Much like under the process in Chapter 3, it is recommended that anticipation exercises be conducted with the key sectors identified during the current needs assessment. The anticipation exercises could be carried out at the same time as the current needs assessment, or at a different time. As documented in Chapter 2, we recommend bringing together a broad, representative group to capture the depth and breadth of influence digital skills have on individuals and communities, both now and in the future. These groups include:

• Government agencies and regulatory bodies

• Educational institutions

• Private sector

• Civil society.

For more details on what potential stakeholders to include, refer to Chapter 2, Step 1: Assemble team.

Next, we set out potential questions to ask key sector stakeholders about digital skills requirements stemming from forecasted technological changes. As mentioned previously, it is important to consider including both small and large firms from the sectors. These questions and the table that follows can support the anticipation exercise. During the analysis, the resulting information can be combined with the information on global trends elicited under Step 1.

A table to assist in data gathering may be found below.

Table 9: Guide to data gathering

Now your stakeholders can compile and compare the two lists of anticipated digital skills requirements. There should be some degree of overlap, depending how recent your country’s latest development plans are. You will use this list in the next step, for making strategic decisions.

Step 3: Make strategic decisions

Countries might also consider what their future digital skills requirements are not just by evaluating their current key sectors, but by making strategic decisions about what sectors to create, expand or enter in the future based on the country’s current digital skills strengths, infrastructure capabilities (electrification, Internet penetration, etc.) and competitive advantage. Some countries might have already created and distributed their future strategic goals through national development plans or digital strategy plans (see Step 2 in this chapter), while others might need to consider their strategic goals for the first time, or revisit them after conducting skills assessment and anticipation exercises.

Each country will engage with strategic decision-making in a different way, but here are some questions to consider as stakeholders review the data gathered in Steps 1 and 2:

One way to think strategically about how to engage is by making an inventory of different factors that might affect demand for digital skills in the future, such as changes to the country’s economic sectors of importance or demographic trends. Below is a table from ITU’s Digital Skills Toolkit that can prove useful for reviewing some of these factors.

Table 10: Factors that affect demand

After completing this exercise, determine which factors can be influenced and which cannot, so as to inform policy choices going forward. For example, demographic trends involving birth rates or overall aging of the population are probably beyond the scope of this particular stakeholder group. However, the stakeholder group may feel the need to shift education policy or industry investments to meet future digital skills needs.

Conclusion

New digital technologies will influence the future of work. However, exactly what that future looks like will differ from country to country. This chapter has provided an overview of how policy-makers can forecast their own country’s future digital skills requirements. Stakeholders continue to be important. Your governance model established in Chapter 2 will make it easier for you to tap into stakeholders’ expertise in how global technology trends will make an impact locally. Combining data from international and local resources will allow policy-makers to make strategic decisions about where to invest in order to enable a country to compete in the future. These strategic decisions will directly influence what digital skills will be required in the future.

Endnotes

[1] The ILO defines green jobs as “decent jobs that contribute to preserve or restore the environment, be they in traditional sectors such as manufacturing and construction, or in new, emerging green sectors such as renewable energy and energy efficiency”. See: http://www.ilo.org/global/topics/green-jobs/news/WCMS_220248/lang--en/index.htm .

[2] To read more about global trends and how they will influence skills needs, see relevant publications by OECD (OECD, 2012; OECD, 2019a), WEF (World Economic Forum, 2018) and ILO (ILO, 2019).

[3] The ITU Digital Skills Toolkit provides policy-makers with detailed guidance on developing and implementing a digital skills strategy at the country level that could be useful if the country has not already created a digital skills strategy (ITU, 2018a).

Chapter 5: Conclusion

Digital transformation is rapidly driving changes in labour markets in almost every sector, as diverse as agriculture, education, environment, finance, health, trade, transportation, tourism and the environment, among others, creating an increasing need for countries to develop a digitally skilled population to be competitive and employable in the global society and economy. In order to accomplish this, policy-makers must identify current digital skills levels and understand what digital skills are necessary to meet current and future needs in their countries. Each country will choose to engage with digital transformation in its own way; but, to do so, policy-makers need to understand their country’s technological assets, challenges and opportunities.

This guidebook is designed to provide as much flexibility as possible for each country to choose an approach that fits its resource constraints and unique goals. Each country has different digital skills needs and requirements based on its level of technological development and its economic sectors. Furthermore, assessment methods will depend on a country’s resources and stakeholder engagement. There is no one-size-fits-all method. Policy-makers should engage with partners in the private sector, non-governmental organizations and academia to craft the assessment approach that matches the country’s needs and goals.

Assessing the demand for and supply of digital skills also supports the United Nations 2030 Agenda for Sustainable Development and attainment of the Sustainable Development Goals (SDGs). Digital technologies are key enablers of the SDGs, and a more digitally literate population therefore increases countries’ capacity to effectively use ICT products and services for development.

In particular, SDG 17, target 17.9, aims to: “Enhance international support for implementing effective and targeted capacity-building in developing countries to support national plans to implement all the sustainable development goals, including through North-South, South-South and triangular cooperation” (United Nations, 2015, p,27). This SDG emphasizes the need for countries to fully operationalize technology and innovation capacity-building mechanisms and enhance the use of enabling technologies. This can be achieved meaningfully if countries have assessed their digital skills levels and are fully aware of which skills are available in the country and which skills need to be developed in order to make use of enabling technologies.

The actions outlined in this guidebook can also help achieve SDG 8: “Promote sustained inclusive and sustainable economic growth, full and productive employment and decent work for all” by prioritizing digital skills development in national youth employment and entrepreneurship strategies in all countries. Additionally, a country can measure SDG 4’s thematic indictor[1] 4.4.2, “Percentage of youth/adults who have achieved at least a minimum level of proficiency in digital literacy skills” if performance-based testing is used to measure current digital skill levels (UNESCO-UIS, 2019, p.57).

Policy responses related to the capacity to grow necessary digital infrastructure (electrification, Internet penetration, etc.), address skills mismatches in the education sector and promote the development of specialized digital skills are a necessary component of being prepared to meet future digital skills requirements. This calls for strong partnerships between policy-makers, academia, the private sector and the public at large, to match skills demand with skills supply, anticipate future skills requirements and assess the relevance of skills development interventions. By involving high-level representation, such as agency heads, ministers, CEOs and directors, among many others, political engagement facilitates commitment and improves the chances of success of digital skills assessment exercises. Armed with an understanding of current gaps and future requirements, policy-makers can then begin to craft their policy responses.

The digital skills assessments are not one-off activities, and countries may find themselves having to repeat them periodically. The need to repeat assessments could stem from a review of digital skills strategies or other government interventions making it necessary for a country to track technological developments as well as skills demand and supply.

This guidebook is intended to contribute to countries’ efforts to assess digital skills supply and demand and bring clarity to anticipating future digital skills needs. It is the result of a review of digital skills frameworks and other products, synthesizing the information and consolidating it into a simple step-by-step process that countries can follow. Whatever the motivation for undertaking a digital skills assessment exercise, countries are encouraged to refer to this guidebook to enhance their work.

Endnotes

[1] Thematic indicators measure aspects of SDG 4 not covered by the global indicators (UNESCO-UIS, 2019, p.41).

References

Adinde, I. (2019). Rethinking capacity building in the digital era: The African context. ITU Digital Skills Insights, 33–42.

African Capacity Building Foundation. (2017). Africa capacity report 2017: Building capacity in science, technology and innovation for Africa’s transformation. ACBF. https://elibrary.acbfpact.org/acbf/collect/acbf/index/assoc/HASH01ad/e44e7241/b749d69a/1a6c.dir/ACR2017%20English.pdf

British Council & ILO. (2014). Skill Assessment in India: A discussion paper on policy, practice and capacity. British Council and International Labour Organization (ILO). https://www.britishcouncil.in/sites/default/files/ilo_british_council_skill_assessment_in_india.pdf

Broadband Commission for Sustainable Development. (2017). Working Group on Education: Digital skills for life and work. https://broadbandcommission.org/Documents/publications/WG-Education-Report2017.pdf

Carretero, S., Vuorikari, R. & Punie, Y. (2017). DigComp 2.1: The Digital Competence Framework for Citizens with eight proficiency levels and examples of use. Publications Office of the European Union. https://ec.europa.eu/jrc/en/publication/eur-scientific-and-technical-research-reports/digcomp-21-digital-competence-framework-citizens-eight-proficiency-levels-and-examples-use

Commission on Science and Technology for Development. (2018). Building digital competencies to benefit from existing and emerging technologies, with a special focus on gender and youth dimensions. United Nations Economic and Social Council. https://unctad.org/meetings/en/SessionalDocuments/ecn162018d3_en.pdf

Directorate for Education and Skills. (n.d.). PIAAC country note Peru. OECD. https://www.oecd.org/skills/piaac/publications/countryspecificmaterial/PIAAC_Country_Note_Peru.pdf

Dodel, M. & Aguirre, F. (2018). Uruguay, sociedad e internet: Principales resultados de la encuesta WIP+DiSTO Uy 2017. Universidad Católica del Uruguay. https://ucu.edu.uy/sites/default/files/facultad/fch/uruguay_internet_2019.pdf

Ecorys UK. (2016). Digital skills for the UK economy. Department for Business Innovation & Skills; Department for Culture Media & Sport. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/492889/DCMSDigitalSkillsReportJan2016.pdf

Essential Digital Skills Framework. (2018). https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/738922/Essential_digital_skills_framework.pdf

Eurostat. (n.d.-a). Individuals who have basic or above basic overall digital skills by sex (tepsr_sp410). Retrieved 18 March, 2020, from https://ec.europa.eu/eurostat/cache/metadata/en/tepsr_sp410_esmsip2.htm

Eurostat. (n.d.-b). Experimental statistics – Skills. Retrieved 18 March, 2020, from https://ec.europa.eu/eurostat/web/experimental-statistics/Skills

Human Capital Project. (n.d.). [Text/HTML]. World Bank. Retrieved 20 December, 2019, from https://www.worldbank.org/en/publication/human-capital

ICDL Arabia. (n.d.). Retrieved 11 October, 2019, from https://icdlarabia.org/

ILO. (n.d.). ILOSTAT - The leading source of labour statistics. ILOSTAT, International Labour Organization (ILO). Retrieved 21 February, 2020, from https://ilostat.ilo.org/

ILO. (2018). Women and men in the informal economy: A statistical picture. Third edition [Report]. ILO. http://www.ilo.org/global/publications/books/WCMS_626831/lang--en/index.htm

ILO. (2019). Work for a brighter future—Global Commission on the Future of Work [Report]. ILO. http://www.ilo.org/global/publications/books/WCMS_662410/lang--en/index.htm

ILO & OECD. (2018). Approaches to anticipating skills for the future of work [Report]. ILO and OECD. http://www.ilo.org/global/about-the-ilo/how-the-ilo-works/multilateral-system/g20/reports/WCMS_646143/lang--en/index.htm

International Finance Corporation. (2019). Digital Skills in Sub-Saharan Africa: Spotlight on Ghana. IFC. Retrieved 18 December, 2019, from https://www.ifc.org/wps/wcm/connect/ed6362b3-aa34-42ac-ae9f-c739904951b1/Digital+Skills_Final_WEB_5-7-19.pdf?MOD=AJPERES

ITU. (2018a). Digital Skills Toolkit. International Telecommunication Union (ITU). https://www.itu.int/en/ITU-D/Digital-Inclusion/Documents/ITU%20Digital%20Skills%20Toolkit.pdf

ITU. (2018b). ITU ICT Households Short Questionnaire 2018. ITU. https://www.itu.int/en/ITU-D/Statistics/Documents/datacollection/ITU_HH_SQ_2018.pdf

ITU. (2018c). Measuring the Information Society Report: Volume 1. ITU.

ITU Academy. (n.d.). Digital Skills Insights. ITU. https://academy.itu.int/main-activities/research-publications/digital-skills-insights

Kaarakainen, M.-T., Kivinen, O. & Vainio, T. (2018). Performance-based testing for ICT skills assessing: A case study of students and teachers’ ICT skills in Finnish schools. Universal Access in the Information Society, 17(2), 349–360. https://doi.org/10.1007/s10209-017-0553-9

Kenya Institute of Curriculum Development. (2017). Basic Education Curriculum Framework. KICD, Republic of Kenya. https://kicd.ac.ke/wp-content/uploads/2017/10/CURRICULUMFRAMEWORK.pdf

Kluzer, S. & Pujol Priego, L. (2018). DigComp into action - Get inspired, make it happen. A user guide to the European Digital Competence Framework. (JRC Science for Policy Report). Publications Office of the European Union. 10.2760/112945

Laanpere, M. (2019). Recommendations on assessment tools for monitoring digital literacy within UNESCO’s Digital Literacy Global Framework. United Nations Educational, Scientific and Cultural Organization (UNESCO) Institute for Statistics (UIS). https://doi.org/10.15220/2019-56-en

Law, N., Woo, D., de la Torre, J. & Wong, G. (2018). A Global framework of reference on digital literacy skills for Indicator 4.4.2. UNESCO Institute for Statistics (UIS). http://uis.unesco.org/sites/default/files/documents/ip51-global-framework-reference-digital-literacy-skills-2018-en.pdf

Litt, E. (2013). Measuring users’ Internet skills: A review of past assessments and a look toward the future. New Media & Society, 15(4), 612–630. https://doi.org/10.1177/1461444813475424

Lloyds Bank. (n.d.). Lloyds Bank UK Consumer Digital Index 2019. Retrieved 13 November, 2019, from https://www.lloydsbank.com/banking-with-us/whats-happening/consumer-digital-index.asp

London School of Economics and Political Science. (n.d.-a). DiSTO Project: From digital skills to tangible outcomes. London School of Economics and Political Science (LSE). Retrieved 20 March, 2020, from https://www.lse.ac.uk/media-and-communications/research/research-projects/disto/home.aspx

London School of Economics and Political Science. (n.d.-b). DiSTO Surveys: From digital skills to tangible outcomes – Improving measures and models of digital engagement. LSE. Retrieved 20 March, 2020, from https://www.lse.ac.uk/media-and-communications/research/research-projects/disto/surveys.aspx

Mackay, A., Gallo, I., Husch, J. & Rak-sakulthai, V. (2015). Skills assessment for national adaptation planning: How countries can identify the gap. United Nations Institute for Training and Research (UNITAR). https://www4.unfccc.int/sites/NAPC/Documents/Supplements/UNITAR%20sanap%202015.pdf

McKinsey Global Institute. (2017). Jobs lost, jobs gained: Workforce transitions in a time of automation. McKinsey & Company. https://www.mckinsey.com/~/media/mckinsey/featured%20insights/future%20of%20organizations/what%20the%20future%20of%20work%20will%20mean%20for%20jobs%20skills%20and%20wages/mgi%20jobs%20lost-jobs%20gained_report_december%202017.ashx

National Association of Software and Services Companies. (2018). Hon’ble Prime Minister Narendra Modi unveils ‘Futureskills’ platform to upskill technology professionals in India. NASSCOM. Retrieved 3 December, 2019, from https://www.nasscom.in/sites/default/files/media_pdf/honble-prime-minister-narendra-modi-unveils-futureskills-platform-to-upskill-technology-professionals-in-india.pdf

OECD. (n.d.). Survey of adult skills - PIAAC data and tools. Organisation for Economic Co-operation and Development (OECD). Retrieved 24 October, 2019, from https://www.oecd.org/skills/piaac/data/

OECD. (2012). ICT skills and employment: New competences and jobs for a greener and smarter economy. OECD Digital Economy Papers, No. 198. OECD Publishing. http://dx.doi.org/10.1787/5k994f3prlr5-en

OECD. (2016). Getting skills right: Assessing and anticipating changing skill needs. OECD Publishing. http://dx.doi.org/10.1787/9789264252073-en

OECD. (2017). Skills strategy diagnostic report: Italy. OECD. https://www.oecd.org/skills/nationalskillsstrategies/Diagnostic-report-Italy.pdf

OECD. (2019a). Going digital toolkit. Going Digital Toolkit. OECD. https://goingdigital.oecd.org/en/

OECD. (2019b). Measuring the digital transformation—A roadmap for the future. OECD Publishing. https://www.oecd.org/publications/measuring-the-digital-transformation-9789264311992-en.htm

OECD (2019c). Employment Outlook 2019: The future of work. OECD iLibrary. https://www.oecd-ilibrary.org/sites/9ee00155-en/index.html?itemId=/content/publication/9ee00155-en

Pix—Cultivez vos compétences numériques. (n.d.). Pix.Fr. Retrieved 28 October, 2019, from https://pix.fr

Redeker, D. & Sturm, I. (2019). ICT skills in small island developing states: ICT capacity building, economic opportunities and brain drain. ITU Digital Insights, 73–84.

Ryder, J. (2018). The basic digital skills framework is being updated—Now’s your chance to have a say and shape its future. Digital Skills and Inclusion. https://digitalinclusion.blog.gov.uk/2018/01/23/the-basic-digital-skills-framework-is-being-updated-nows-your-chance-to-have-a-say-and-shape-its-future/

Sparks, J. R., Katz, I. R. & Beile, P. M. (2016). Assessing digital information literacy in higher education: A review of existing frameworks and assessments with recommendations for next-generation assessment. ETS Research Report Series, 2016(2), 1–33. https://doi.org/10.1002/ets2.12118

Telenor Group. (2018). Realising digital Myanmar: Leapfrogging to an inclusive digital economy. Retrieved 12 December, 2019, from https://www.telenor.com/wp-content/uploads/2018/02/Telenor-Realising-Digital-Myanmar-Report-06-February.pdf

UNESCO-UIS. (2019). SDG 4 Data Digest – How to produce and use the global and thematic education indicators. UNESCO Institute of Statistics (UIS). http://uis.unesco.org/sites/default/files/documents/sdg4-data-digest-2019-en.pdf

United Nations. (2015) The 2030 agenda for sustainable development, A resolution adopted by the General Assembly of 25th September 2015, https://www.un.org/en/development/desa/population/migration/generalassembly/docs/globalcompact/A_RES_70_1_E.pdf

van Deursen, A. J. A. M., van Dijk, J. A. G. M. & Peters, O. (2012). Proposing a survey instrument for measuring operational, formal, information, and strategic Internet skills. International Journal of Human-Computer Interaction, 28(12), 827–837. https://doi.org/10.1080/10447318.2012.670086

van Deursen, A.J.A.M., Helsper, E.J. & Eynon, R. (2014). Measuring digital skills. From Digital Skills to Tangible Outcomes project report. Retrieved 26 October, 2017, from https://www.oii.ox.ac.uk/archive/downloads/publications/Measuring_Digital_Skills.pdf

van Deursen, A. & van Dijk, J. (2015). New media and the digital divide. In J. D. Wright (Ed.), International Encyclopedia of the Social & Behavioral Sciences (Second Edition), 787–792. Elsevier. https://doi.org/10.1016/B978-0-08-097086-8.95086-4

Vuorikari, R. & Punie, Y. (2019). The use of reference frameworks to support digitally competent citizens – the case of DigComp. ITU Digital Skills Insights, 1–11.

World Bank. (2019a). World Development Report 2019: The changing nature of work. World Bank. 10.1596/978-1-4648-1328-3

World Bank. (2019b). World Bank Group—LinkedIn digital data for development, jobs, skills and migration trends. Data Catalog. https://datacatalog.worldbank.org/dataset/world-bank-group-linkedin-dashboard-dataset

World Economic Forum. (n.d.). Reskilling Revolution Platform. World Economic Forum. Retrieved 21 February, 2020, from https://www.weforum.org/projects/reskilling-revolution-platform/

World Economic Forum. (2016). The Future of Jobs: Employment, skills and workforce strategy for the Fourth Industrial Revolution. Global Challenge Insight Report. http://www3.weforum.org/docs/WEF_Future_of_Jobs.pdf

World Economic Forum. (2018). The Future of Jobs Report 2018. Insight Report. http://www3.weforum.org/docs/WEF_Future_of_Jobs_2018.pdf

Appendix

Appendix: Digital skills assessment resources and tools

Tool, Author, Date published

Where tool is used

Overview

Skills evaluated

Format

Target group

Frequency of use

Administration

Supporting materials

Advantages

Disadvantages

Compass,[1] Compass Digital Skills, 2018

France, Ireland, Italy, Romania

Following free sign-up, users are prompted to take a skills assessment. Answers denote novice or advanced rating in three categories: communication and collaboration, digital content creation, and safety. Users are then prompted to choose a career pathway.

Basic; Intermediate

Knowledge-based assessment

Young adults

Not Applicable (NA): Self-led training

Online assessment before starting training

Courses with certificates

Connected to training, employment-focused

Career pathways may not apply outside of these countries.

Digital Competence Wheel,[2] Center for Digital Dannelse, 2018

Denmark

The tool builds upon DigComp and includes 4 competence areas with 16 competences and 54 measurable aspects of digital competence. A person rates their knowledge on a scale of 1 - 7. The tool is free for anyone to use. Organizations can sign up for a paid subscription to customize the wheel and track employees' abilities.

Intermediate; Advanced

Self-assessment

Adults

NA: Test and map own proficiency

15-minute test taken individually or coordinated by an organization

Personal competence wheel, recommendations on how to strengthen digital competence, examples, and exercises of each competence

Gives individualized score reports for each of the 16 competences. Offers suggestions on how to strengthen ability in each competence. The wheel is being developed for different subject areas.

No knowledge-based items; only self-reporting. Exercise suggestions require work by the individual. Example suggestion: "I can learn how to convert a Word document to a PDF". No link to how to do that. Diagrams very detailed and challenging to read for the average user.

Digital Competency Profiler,[3] Educational Informatics Lab, University of Ontario Institute of Technology, 2015

Canada, Georgia, Ukraine

Rates frequency of and confidence with use of a particular device. Covers computer, mobile and other smart devices. Based on General Technology Competency and Use (GTCU) framework. Breaks down into "four orders of use of digital technology": technical, information, social and epistemological.

Basic; Intermediate

Self-assessment

Adults; University students

NA: Test own digital competen-
cies

Online assessment

Results report

Focus on different types of devices: mobile and "smart" devices

Limited output documentation

Digital Economy and Society Index (DESI),[4] European Commission, ongoing

EU Member States, Iceland, Norway, Switzerland, EU candidate countries and potential candidate countries

The human capital indicator is founded on DigComp. Based on responses, a person is rated to having "basic" to "above basic" skills.

Basic; Intermediate; Advanced

Self-assessment

Adults

Every 2 years

Collected every 2 years by countries' national statistical institutes or relevant ministries

Part of the DESI, a composite index of connectivity, human capital (digital skills), use of the Internet by citizens, integration of technology and digital public services

Measured in its current form since 2014. Comparable across countries where data collected.

Limited to countries with more advanced data collection

Digital Skills Accelerator,[5] Erasmus+ (European Union), 2019 (ongoing)

Belgium, Ireland, Poland, Spain, United Kingdom

Self-rate within the 5 DigComp competence areas using a 1 - 6 proficiency scale

Basic; Intermediate; Advanced

Self-assessment

Adults

NA

Online assessment

Personalized radar chart, training recommendations

Receive personalized diagram showing strengths and weaknesses. Gives training recommen-
dations.

Students responsible for assessing their own performance

Digital Skills to Tangible Outcomes (DiSTO),[6] London School of Economics and Political Science, 2015 (ongoing)

Australia, United Kingdom, Netherlands, Chile, Brazil, Uruguay, United States

Surveys on digital skills, Internet users and outcomes of Internet use

Basic; Intermediate

Self-assessment

Youth; Adults

As often as preferred; generally, once

Paper or online survey

Academic publications and reports, research on digital exclusion in some countries using heatmaps

Based on theory, originally tested for validity and reliability in both UK and Netherlands. Expanded to other countries through research partnerships. Tools continue to be tested.

Only tested and used in UK and Netherlands. Focused exclusively on online skills.

Education & Skills Online,[7] Organisation for Economic Co-operation and Development (OECD), N.D.

Available in 10 languages

The online version is available for individuals or institutions. Results are mapped onto PIAAC results. It also tests other domains: literacy, numeracy, technology-rich environments, reading (optional). Optional non-cognitive assessments (skill uses, behavioural, career interest, subjective well-being).

Basic; Intermediate; Advanced

Performance assessment; Self-assessment

Adults

NA: Self-assessment

Online assessment

eLene4work,[8] Erasmus+ (European Union), 2017

Belgium, Finland, France, Germany, Greece, Italy, Poland, Spain, United Kingdom

After rating level of soft skills and digital skills (based on DigComp), creates a learning agenda on what areas to improve upon. The orientation guide gives suggestions of online courses (MOOCs) or other learning options to improve employability.

Basic, Intermediate

Self-assessment

Students; Adults

NA: Self-assessment

Freely available online questionnaire

Orientation guide provides pathways to continue learning

Focus on soft and digital skills. Freely available.

No way currently to collect scores aggregately

ICT Households Short Questionnaire,[9] International Telecommu-
nication Union (ITU), 2018 (ongoing)

About 80 countries globally

Sample questionnaire on household ICT access and individuals' use of the Internet, mobile phone ownership and ICT skills. Currently under revision to add more skills and mobile technologies.

Basic; Intermediate; Advanced

Self-assessment

All over 15 years of age, but different age cut-offs

Annually

Questionnaire publicly available for use and can be compared internationally. Methodology has been tested for almost a decade.

Limited questions on ICT skills. Mostly focuses on computer skills and not on other types of digital literacy.

ICT Skills Test,[10] University of Turku (Finland), 2017

Finland

42 performance-based items, which were then categorized into 17 fields of ICT skills. These 17 fields are based on the Finnish National Core Curriculum, the content of the eSkills certification programmes of the Finnish Information Society Development Centre, and the requirements of information and communication studies in Finnish universities of applied sciences.

Basic; Intermediate; Advanced

Knowledge-based assessment

Upper compre-
hensive, secondary students and teachers

NA

Online multiple-choice questionnaires

Asks about advanced ICT skills

Only been used for research in Finland

ICT Skills Indicator,[11] University of Bremen & Institute for Pacific Research, 2019

Fiji, Micronesia, Palau, Samoa, Tonga

Used in 5 small island nations. Online survey to measure advanced digital skills in young people (18-35) to create an "ICT skills indicator" (ISI), and participation in ICT capacity-building workshops. Uses Likert scale to ask respondents to self-report skills in three areas: ability to “manage content of a website”, “design a website”, and “write a computer program using specialized programming language".

Advanced

Self-assessment

Young people (16 -35)

NA: Self-assessment

Online questionnaire using Google Forms

None

Short survey that focuses on more advanced skills

Limited scope of advanced skills. However, approach of only asking about advanced skills may make sense if only looking for individuals with the capacity to be employed in advanced ICT jobs.

Ikanos Competencia Digital,[12] Basque Government, 2012

Spain

Aligned with DigComp. Certifies digital competence in the general population and specific, local jobs. Includes self-assessment test, occupational digital profiles customized for local industry.

Basic; Intermediate; Advanced

Self-assessment

Adults

NA: self-assessment

Self-guided computer-based assessment of 30 questions designed to be completed in 15 minutes, also works on mobile

Personalized score report; directs participants towards local ICT training opportunities; certification a system; guide to help intermediaries to compare a user's test results and occupational profiles

Takes participants’ context into account by including questions on home ICT equipment,
Internet connection and how skills were acquired or certified

The score report resembles a test-based assessment, but it is just attaching labels to participants’ ratings of their own abilities.

International Computer and Information Literacy Study (ICILS),[13] International Association for the Evaluation of Educational Achievement (IEA), 2018

Chile, Denmark, Finland, France, Germany, Italy, Kazakhstan, Republic of Korea, Luxembourg, Portugal, Russian Federation (Moscow), Uruguay and United States (2018 cycle)

Test-based assessment measures student's ability to use computers to investigate, create and communicate in order to participate effectively at home, at school, in the workplace and in the community. Has been administered in 21 countries.

Basic; Intermediate; Advanced

Self-assessment; Performance-based assessment; Knowledge-based assessment

8th-grade students (average age 13.5)

Every 5 years

Computer-based assessment

International report and databases

International comparisons. Incorporates both assessment-based and self-reported ICT skills. Countries can participate in ICILS 2023.

Infrequent (every 5 years)

International Computer Driving Licence (ICDL),[14] ICDL Foundation, n.d.

Over 100 countries

Internationally recognized computer skills certification. Assessment of computer essentials, online essentials, word processing and spreadsheets, as well as software-specific skills.

Basic; Intermediate

Knowledge-based assessment; Performance-based assessment

Adults

NA

Administered at ICDL testing centres

Can be completed with or without ICDL training courses

Proctored at test sites and can be taken with or without participating in ICDL training courses

Focused on computers and software for the workplace (no mobile and limited Internet). No advanced skills. Focuses on the technical aspects of using technology independently and not how those technical skills can be used in real-life situations to solve problems.

Internet and Computing Core Certification Digital Literacy Certification (IC3),[15] Certiport, N.D.

78 countries

IC3 Global Standard 5 covers "living online", "computing fundamentals", and "key applications". "Computing fundamentals" includes "mobile devices" domain. The exam is mapped onto DigComp.

Basic; Intermediate

Knowledge-based assessment; Performance-based assessment

Adults

NA

50-minute exam at an authorized testing centre

Training, certification

"Computing fundamentals" includes "mobile devices" domain

Focuses on the technical aspects of using technology independently and not how those technical skills can be used in real-life situations to solve problems

National Assessment of Educational Progress (NAEP) - Nation's Report Card: Technology & Engineering Literacy (TEL) Assessment,[16] National Center for Education Statistics, US Department of Education, 2018

United States

Uses "problem-solving tasks based on interactive scenarios reflecting realistic solutions". Measures three content areas: "technology & society", "design & systems" and "information & communi-
cation technology". In each content area, a student needs to demonstrate application of a TEL "practice." Practices are divided into three categories: "understanding technological principles", "developing solutions & achieving goals" and "communicating & collaborating". School also reports on what TEL-related subjects are addressed in the curriculum and school demographics.

Basic; Intermediate; Advanced

Performance assessment with self-assessment

Grade 8

Annually in grade 8

Delivered via laptops. Includes 15 scenario-based tasks and 77 discrete questions. Testing time is 60 minutes. "Problem-solving tasks based on interactive scenarios reflecting realistic solutions". 600 schools participated in 2018.

Regional, state and national reports broken down by different demographic groups

Tasks mimic real-life scenarios. Sample tasks available on the website.

Only used in the US. Started in 2014 and revised and administered in 2018.

National Assessment Program (NAP) ICT Literacy,[17] Australian government, 2014 (ongoing)

Australia

Measures student competences regarding accessing, managing, evaluating and developing new understandings of information, as well as communicating with others and using ICT appropriately.

Basic; Intermediate; Advanced

Knowledge-based assessment

Students in year 6 and year 10

Annually. Test in grade 6 and again in grade 10

Targeted at students in year 6 and year 10 to see how the age cohorts change in knowledge over time

Currently only used in Australia

Northstar Assessment,[18] Minnesota Literacy Council, 2010 (ongoing)

Australia, Canada, New Zealand, Nigeria, South Africa, United States

Designed to support community education and workforce goals for adults with only basic ICT skills. Provides a free, online assessment tool and works with approved testing centres to offer proctored assessments and certificates in 31 US states, as well as Canada and South Africa.

Basic; Intermediate

Knowledge-based assessment

Adults

NA

Anyone can access the online test for free. Organizations can also pay to become testing centres for official certification.

Certification (if taken at an official testing centre)

Supports basic computer skills often needed for professional work, such as basic computer use, Internet, Windows OS, Mac OS, e-mail, social media, information literacy, Microsoft Word, Microsoft Excel, and Microsoft PowerPoint

Assessment available to anyone, but certification programme dependent on limited test centre locations worldwide

Pix,[19] French Ministry of Education, 2016

France

Online platform for assessment and certification of digital skills to assess and certify digital competences based on DigComp. Proficiency levels are given based on answering "problem statements". The individual uses their digital skills to provide the correct answer.

Basic; Intermediate; Advanced

Knowledge-based assessment; Performance-based assessment

Adults; Students

NA

Online test

Developing pedagogical tools for teachers, free accounts for schools, and pay accounts for other entities

Free assess-
ment for French speakers. Open-source platform (source code available on GitHub). Provides targeted recommen-
dations of learning resources post-assess-
ment. Device agnostic to demons-
trate skills.

Currently available in French

Programme for International Student Assessment (PISA),[20] Organisation for Economic Co-operation and Development (OECD), 2000 (ongoing)

80 countries, 82 languages (2018)

Assessment of reading, science and maths. In 2021, PISA will have a separate ICT assessment. In previous cycles, digital skills were embedded in other sections.

Basic; Intermediate

Performance-based assessment

15-years old

Every 3 years

Computer-based, every 3 years

Internationally comparable data

Can be costly. Repeated infrequently.

Programme for the International Assessment of Adult Competencies in Technology-Rich Environments (PIAAC-TRE),[21] Organisation for Economic Co-operation and Development (OECD), 2011 (ongoing)

Over 40 countries (OECD and non-OECD)

Two-part survey: background questionnaire and direct assessment of cognitive skills (including problem solving in technology-rich environments). Cycle 1 finished in 2018, with 38 countries participating. Cycle 2 will take place in 2021-2022 and data publication in 2023.

Basic; Intermediate; Advanced

Performance assessment with self-assessment

Adults

Once: 2 cycles

Household assessment taking 90 minutes, includes interview and computer test

Allows international comparison. Adaptive design. Self-assessment asks about whether the skills they possess match what their job requires.

Can be costly. Repeated infrequently.

TASK,[22] Erasmus+ (European Union), 2017

Belgium, Spain, France, Italy, Turkey

Tests communication in mother tongue, communication in a foreign language and digital competence (based on DigComp). A student is given a real-life scenario in 1 of 4 different mastery levels in five competence areas. The student performs the tasks and then assesses their performance based on TASK questionnaire.

Basic; Intermediate

Performance assessment with self-assessment

Lower and upper secondary school

NA

Web tool administered by a teacher

Real-life scenarios. Available in English, Italian, French, Spanish and Turkish.

A student is responsible for assessing their own performance.

Endnotes

[1] https://www.compassdigitalskills.eu/

[2] https://digital-competence.eu/

[3] https://dcp.eilab.ca/

[4] https://ec.europa.eu/eurostat/cache/metadata/en/tepsr_sp410_esmsip2.htm

[5] https://www.digitalskillsaccelerator.eu/

[6] https://www.oii.ox.ac.uk/research/projects/measures-and-models-of-internet-use

[7] https://www.oecd.org/skills/piaac/onlineassessmment/

[8] http://elene4work.eu/

[9] https://www.itu.int/en/ITU-D/Statistics/Documents/datacollection/ITU_HH_SQ_2018.pdf

[10] https://dl.acm.org/citation.cfm?id=3237231

[11] https://drive.google.com/file/d/1cG7UmhkMAmfEul5c9lfwxwxgxeVr6A2N/view

[12] https://www.ikanos.eus/en/

[13] https://www.iea.nl/icils

[14] https://icdl.org/

[15] https://certiport.pearsonvue.com/Certifications/IC3/Digital-Literacy-Certification/Overview.aspx

[16] https://www.nationsreportcard.gov/tel_2018_highlights/

[17] https://www.nap.edu.au/nap-sample-assessments/ict-literacy

[18] https://www.digitalliteracyassessment.org/

[19] https://pix.fr/

[20] https://www.oecd.org/pisa/

[21] https://www.oecd.org/skills/piaac/about/

[22] http://www.taskeuproject.com/

Boxes