The ‘hidden data’ that could boost the UK’s productivity and job market

The Career Tech Challenge looked at supporting the development of new solutions to help people find rewarding future careers, including those in low pay or insecure work. We’ve been conducting research with the finalists to explore the process of building data-driven solutions. This report explores the findings and highlights how the CareerTech Challenge innovators used labour market data to support adults to plan for their future career.


28 06 2021



Using Labour Market Data to Support Adults to Plan for their Future Career: Experience from the CareerTech Challenge

Download the report

In the Skills for Jobs white paper that has become a strategic centrepiece for UK government, the Department for Education sets out a plan to align the supply and demand for skills in the job market through a mix of measures that includes employer-led Local Skills Improvement Plans, new financial entitlements for training, and ‘great careers support’.

If these approaches work, they could help bring the UK’s high level of ‘skills mismatch’ to best practice levels, resulting in a boost to productivity of up to 5% (according to the OECD). They could also accelerate recovery from COVID-19, by giving workers the support they need to find new roles that realise their potential, make use of their skills, and offer them a more secure future in a labour market that is being disrupted by new technology.

But how can ‘great careers support’ be delivered at a scale that has a real impact on the labour market? This was a central question that drove the CareerTech Challenge, a partnership between Nesta, Nesta Challenges and the Department for Education which sought to stimulate the development of new tools for career navigation, and uncover ‘what works’ to help adults develop new skills and find good work.

We documented and analysed the innovators’ progress through interviews and research, enabling Nesta to understand the enablers and barriers for the development of ‘great careers support’.

What did and could help innovators deliver tailored and responsive career support?

Out of the 31 innovators supported by the CareerTech Challenge, 20 worked on building and testing new tools to make careers information, advice and guidance more accessible and tailored to the supply and demand for skills in local job markets.

The research revealed an abundance of creativity among the innovators, and demonstrated the value of user testing, co-creation, and partnerships with local stakeholders, such as FE colleges and employers. Yet it also uncovered a major challenge that is holding back innovation in the field: limited and inaccessible labour market data.

The deficiencies in the UK’s labour market data are illustrated by the experiences of the winners of the CareerTech Challenge Prize, the team developing Bob UK,      , a tool which is designed to provide instant, online careers advice and job recommendations based on information about local vacancies and the jobseeker’s skills.

The developers spent time trying to source UK data which directly replicated data sources used to develop the version of Bob that has helped over 250,000 jobseekers in France. However, it became apparent that equivalent sources of data rarely existed. The Bob UK team was able to work around this issue by carefully combining alternative sources of data from a number of UK and non-UK sources.

Many other innovators experienced similar barriers, finding that the publicly available data that could help people to make more informed decisions about their careers is often incomplete, difficult to use, and poorly described. The impact of this is significant.

A shocking insight from the report is that one innovation enabled careers advisors to base course recommendations on labour market information for the first time. Prior to using this tool, such information was too time-consuming for careers advisors to uncover and analyse for it to be of use, and job seekers were given advice that was not based on employer demand for skills.

How better data can help deliver great career support

To address this issue of hidden and missing data and unleash the productivity-raising potential of better skills matching, the report recommends:

  • The creation of a central labour market data repository which collates publicly available information about the labour market.
  • Public data providers should review the quality and accessibility of the data they hold, and make it easier for developers to use.
  • The development of better skills and labour market taxonomies to facilitate consistency between sources and enhance data matching.

Nesta’s Data Analytics team have launched a number of initiatives that aim to fill gaps in the UK’s supply of labour market information:

  • Creating a skills taxonomy: With funding from ESCoE, the team created the first ever data-driven skills taxonomy for the UK that is open. Nesta is currently updating the taxonomy and hope to publish the next version towards the end of this year.
  • Mapping viable transitions for workers: In Mapping Career Causeways, supported by P. Morgan, the team used machine learning to measure the similarity in the skills and work activities required in more than 1,600 jobs. This information can be used to suggest ‘viable transitions’ to job seekers, based on their most recent role. The underlying algorithm can also identify the skills gap between any two jobs which can inform decisions around training. The code for this project is open and is available on Github.
  • Measuring automation risk: With funding from The Gatsby Foundation, the team provided the first ever estimates of automation risk for apprenticeships. Policy recommendations included increasing awareness about the types of tasks that were found to raise risk (such as routine and repetitive activities) and those that lower risk (such as tackling unstructured problems in changeable environments).
  • Providing free insights on skill demands: In partnership with the Department for Education, Nesta is creating an Open Jobs Observatory which will contain free insights from online job adverts, with a focus on the skills requested by employers.
  • Identifying green jobs: Nesta’s next focus is developing a methodology for tagging green jobs. At present, the UK has no way of identifying individual jobs in green sectors. Developing this methodology is a necessary step in transitioning towards a greener economy.

If you are interested in discussing the findings from the report please contact or for more information about Nesta’s new initiatives contact