Will a Skills Passport ever get me through the lifelong learning border?: Two critical challenges facing personalised user models for lifelong learning

2.2.1 Challenge 1: Lifelong Learning requires data from many different systems. Lifelong Learning requires a significant shift in thinking from the norm. Instead of providing stand alone solutions within a specific Educational Technology (EdTech) product we need to think about how a learner might be supported across the many different learning environments that they interface with across a lifetime. Ideally, a lifelong learner model would help people to understand how they learn, and what they need to learn, supporting them in setting and then achieving goals that could be very long term. Importantly, learners should be supported through the many different transitions that they undergo throughout their lifetime: from school to university, to the workplace, and back to further education and professional development [40], helping them to re-frame their thinking about themselves along the way [10].

Kay [28] presented a vision for lifelong learning as early as 2008, when she proposed that we need to put people in control of rich and scruitable lifelong user models which would help them learn how to learn. Key to this proposal, was the argument that user models should be elevated to the status of a first-class citizen, meaning that they should have “intrinsic value independent of any one application”. The paper reviews a number of prototype scruitable user models demonstrating how they can help a learner to understand their own learning process. A more recent update [30] revisits this grand challenge, introducing a notion of a Personal User Model for Life-long Life-wide Learners (PUMLs), and reviews an ongoing program of research on the topic. Importantly, this paper presents a clear conceptual model of how the various user models intersect, and points to the distinction that a PUML is a personal model for one user. However, the way we could gather data generated by a learner over their lifetime was given less attention in these papers.

This is by no means an easy challenge to solve, but a PUML will require precisely this type of advance. This is because learning occurs across formal and informal educational contexts, in a wide array of different environments. People do not just stay within an institutionally authorised Learning Management System (LMS) and its associated supported tools. They can learn anywhere and everywhere, and are becoming increasingly “non-compliant” learners [23]. People use a range of social media to communicate with one another, sometimes supported the workplace, but sometimes not. They make use of sites like StackExchange and Youtube to find answers to challenges they face, enrol in microcredentials and Massive Open Online Courses (MOOCs), interact with coaches and more senior colleagues over coffee to resolve challenges that they are facing, and increasingly turn to ChatGPT to answer questions on the fly. This learning that occurs “in the wild” [33] presents a problem for those who would build lifelong learner models [15]. We need to be able to gather data from multiple places, and it often comes in a wide array of formats.

Comprehensive and scalable solutions for dealing with data from multiple sources have yet to be implemented. Educational data standards are often proposed as a method for delivering data interoperability at the point of data emission, but the process is fraught and adoption has been sporadic at best. Early standards like SCORM², and now the two more modern updates, Experience API (xAPI)³, and IMS Caliper⁴, all attempt to harmonise educational data at its original source using one pre-defined format, but to date they have largely failed [24, 36]. Why the ongoing failure? One explanation is proposed by Feldstein [17], who notes that specifications tend to produce compromises that few people are thrilled with, and often leads to them striking out with their own solutions. Similarly, Stringer et al. [44] have highlighted the social problems facing standards adoption, pointing to: the slowness of the standards development process; researcher resistance; diversity in tools; institutional issues; and a lack of vendor cooperation. Thus, achieving data interoperability is much more of a socio-technical problem than a straight technical one. And yet the bulk of the data standards work in this space has tended to focus upon technical issues alone. An alternative approach involves trying to harmonise data at point of capture [33, 34], while encouraging vendors and large organisations to work towards a more uniform emission of learning data [36]. In this scenario great care must be taken to ensure that the data captured is mapped into educationally meaningful constructs [29], as this helps to ensure that the data is actually useful in a learner model [36]. While such hybrid socio-technical models show promise, they are yet to be implemented at scale across a lifelong learning context. The social barrier remains a difficult one to overcome.

2.2.2 Challenge 2: Lifelong learning requires portable learner models. Even if we could aggregate data from a wide array of learning environments in a meaningful sense, another problem remains. We must also be able to move the resulting learner model between different learning systems. Baker et al. [8] identified this problem as a key grand challenge, calling it the problem of transferability, or the learning systems wall. Critically, each time a learner interfaces with a new learning environment they must endure a cold start as the system knows nothing about them and has to build up a new learner model. Valdés Aguirre et al. [46] suggest that this problem arises because user models in learning environments tend to be more heterogeneous and complex than the models traditionally used in recommender systems (RecSys). For example, while a RecSys would typically contain some basic user information (e.g. location, browser, language) coupled with information like click through rate to other pages, time spent looking at items etc., learner models often contain far more complex constructs such as emotional state and affect, knowledge models, social dynamics etc., which are more difficult to represent across different systems. Furthermore, as machine learning is often used to extract models about these complex constructs, the resulting models are rarely portable to other environments.

While attempts to move learner models and competencies between educational domains have existed for decades (see e.g. [6, 27, 28] for some early attempts) this problem remains largely unsolved. Multiple reasons lie behind this failure, and in fact the problem could again be considered socio-technical. For example, there is a tendency for most groups to work upon improving the learner model in their system rather than working on porting that model to other systems. This is an understandable omission as the models are often remarkably different. Desmarais and Baker [15] provide a comprehensive review of learner and skills-based models, calling attention to the issues associated with aligning ontologies between different models in a long term learner model. This lack of semantic interoperability [11] between different learner models is a key unresolved challenge. Even if this problem could be resolved, where precisely the learner model would be stored, and in what technology stack, remains an open question. While many different proposals have been made [14, 39], the lack of convergence upon one solution is concerning.

3.2.1 Different socio-geographical sectors use markedly different data and metadata. Core to the challenge facing skills passports is the wildly varying data that they will have to aggregate from various socio-geographical sectors, in the form of skills taxonomies, ontologies, graphs, capability frameworks, course information and learning experience data. How will this data make sense as someone crosses the skills sector border depicted in Figure 1? Let us assume that our learner starts in Australia, which appears to be heading towards broadscale adoption of the ASC across the higher and vocational education sectors. As of the December 2023 release, the ASC features three types of skills for 1575 occupations, coming to combinations of: 1686 specialist tasks, 94 technology tools and 10 core competencies. Assume further that our learner interfaces with a tool to claim 5 skills in the ASC along with a related occupation, say the new Data Scientist (ANZSCO 224115) occupation. The resulting simplified “skills passport” is listed in the first two columns of Table 1. However, the claimed strength of a skills passport does not lie in this simple representation of our learner's skills. Rather, a skills passport becomes useful because of its ability to support someone who is not in their desired occupation to see what skills they need to get there, and what training pathway they might take to reach that goal. Thus, using the same ASC representation of skills, a Policy Analyst might be able to see that they could reach their dream job of Data Scientist by acquiring the skills required for a Data Analyst and then perhaps taking some more advanced training while working in that job for a period. (See the last two columns of Table 1.)

But what happens if our user attempts to move from the Australian context across to a a job in Germany? Perhaps they saw an advertisement that requires applications using a CV generated in the Europass system. At this point we start to see where the skills sector border becomes a serious issue to contend with. We can quickly see the complexity of this problem thinking about how skills would be represented in Europass. It is likely that this skills sector would use the multilingual ESCO taxonomy. Version v1.1.1 of this taxonomy stores information about 3007 occupations, over 13,890 skills and qualifications submitted from contributing countries that span 28 European languages. Completed qualifications can be tagged by contributing institutions with ESCO skills, and displayed in a learner's Europass.¹⁰ However, there are currently no cross-walks between ESCO and the ASC.¹¹ This means that the only way our learner could currently apply for their job in Germany is by undergoing a cold start and creating a new skills passport with this new data format; there are no tools to help our learner cross the skills sector border. This has quickly become a problem of user model portability, so we will return to this problem in Section 3.3 when we discuss Challenge 2.

3.2.2 The skills passport would need to interface with a broad array of systems. Our learner will also have to interface with a variety of learning environments, each of which is likely to both consume and produce data in different formats. This brings both a challenge of integrating with the many different learning environments, as well as the problems of data interoperability, at both the syntactic and the semantic levels. The difficulty of doing this well can be seen in the Europass ecosystem. While participating countries can add their qualifications to Europass, and can even label their level according to the European Qualifications Framework¹² (EQF), mapping between these awards remains a difficult challenge. This is exemplified by the need for Europass Mobility templates¹³ and Certificate Supplements.¹⁴ The existence of these templates demonstrates that actually integrating the data supplied by different European countries remains a manual task that is very difficult to resolve. For a skills passport to be genuinely useful it would have to represent both formal and informal learning across the domain where it is widely used, but the challenges associated with doing this in Europass should give us reason to pause — integrating with IT systems representing curriculum is no easy task, and there is no one system available to make this integration easier. Moreover, any skills passport would likely experience a phasing in period before it became a universal representation of a learner's capabilities. This means that along the way the tool would likely need to integrate with a wide range of existing Portfolios (e.g. LinkedIn) as well as numerous Human Resources (HR) systems used by companies that had yet to adopt the skills passport as a representation of capability. Our learner is now likely to be in a situation where they have built up a skill profile representing their data science capability in two government frameworks and numerous private environments, undergoing a cold start at each step along the way.

3.2.3 Skills and capabilities are contextually dependent on where they are demonstrated. One final problem of data interoperability remains to be explored. This one arises from the frequently made claim that a skills passport could help a learner to identify training pathways to change their career. Thus, in the December 2023 version of the ASC, a Policy Analyst has a specialist task (i.e. a skill) named “Analyse data to identify trends or relationships among variables”, which is shared by 38 other job roles, including Data Scientist and Neuroscientist. While it is quite likely that this specialist task is in fact performed by each of these occupations, it is equally probable that each of these occupations performs this task at a very different level of capability. Thus, an “advanced” capability in this skill for the Policy Analyst is unlikely to be considered “advanced” for a Data Scientist. This makes the unambiguous collection of data from people working in the two different occupations a difficult problem to solve. Some work has been completed by ESCO in using context to restrict occupation suggestions [4], but this is a research problem that requires far more work. Even if we can account for the contextuality of skills within one specific framework or ontology, we would still need to be able to port those skills to highly similar skills described in other skills representations. This returns us to a consideration of Challenge 2.

3.5.1 Trustworthy cross-walks between the core taxonomies. First, a substantial gap exists in the technology available to map between skills taxonomies. Table 2 lists likely candidates for core taxonomies with a “Government” label, and looking down the “Cross-walks” column we see that almost none of them have been mapped to each other. For example, there is currently no way for a person to move a skills passport from Australia to Europe, the USA or to Singapore. The only geographical transition that is currently facilitated is between Europe and the USA (or vice versa). While this is a significant advance, and covers millions of potential users, a skills passport or EdTech provider that takes advantage of this technological development is yet to emerge — supporting the claim that the problem is socio-technical.

It is interesting how few cross-walks have been constructed between any skills frameworks and taxonomies. This problem has likely arisen because cross-walks were traditionally created by hand, which meant that they required a substantial labour investment with very little potential reward. However, the emergence of a general drive towards lifelong learning facilitated by global skills passports provides the potential “killer app” on the horizon that could encourage more work in this domain. One early cross-walk was created by Burning Glass (now Lightcast), whose proprietary ontology originally had a mapping to O*Net-SOC and to ANZSCO. The Lightcast “Open Skills” taxonomy²⁷ now maps to O*Net-SOC, however, the ANZSCO cross-walk appears to be depreciated by changes in the taxonomy post merger. However, more open approaches are emerging. For example, a 2022 Masters thesis by Guru Rao [25], used XLNet to map between ESCO and O*Net-SOC with a 69% accuracy over a sample of 200 human labelled occupations. This work also investigated ways in which this matching process might be improved by incorporating domain specific knowledge to bridge between the two ontologies (accuracy 34%), or extending XLNet's vocabulary with domain knowledge (accuracy: 59%), claiming that the relatively small size of the hand labelled validation set led to poorer performance in this case. Even more positive, at a similar point in time, the above mentioned Europe-USA cross-walk was created when the European Commission mapped ESCO into O*Net-SOC using a ML model based upon BERT to speed up matching with human oversight and correction.²⁸ The same methodology could be used to create a similar set of cross-walks between other taxonomies in future work.

Some research has been published which demonstrates how these more global cross-walks might be mapped to geographically local representations of skills and occupations. For example, Dörpinghaus et al. [16] discuss a process by which they constructed a German labour market ontology which takes ISCO occupations as its top level and also maps to ESCO by design. This approach guarantees a cross-walk between three important taxonomies and so is a sound one for supporting both local contextualisation of skills and occupations while retaining the data portability that a skills passport requires. Taking a more automated approach based on word embeddings constructed from a corpus combining both ESCO and the Italian occupation taxonomy, Giabelli et al. [22] mapped the Italian labour market taxonomy into ESCO using FastText, which is an extension of word2vec. This approach did not require the pre-existing mappings between ISCO and the local country specific ontology that was utilised by the previous approach [16], but it also suffered a loss of accuracy as a result which suggests that some human oversight would still be required.

This is because accurate cross-walks between core taxonomies are an essential requirement for a globally useful skills passport. While NLP may go some way towards speeding up the creation of the cross-walks, it seems unlikely that this process will be fully automatable. Restricting the number of core taxonomies used would reduce the amount of work that would need to be performed by humans to ensure that the mappings were trustworthy, which is the underlying reason behind the suggestion that only some skills taxonomies should be approved as core. An ongoing lack of validated publicly available crosswalks will hold back the emergence of a genuinely useful skills passport, and will also encourage the ongoing proliferation of alternative skills taxonomies, private ontologies and graphs. In short, we will see companies, governments and researchers all creating new skills and competency frameworks while it is easier to create a new ontology than it is to map into an existing one. In order to make progress we require national skills bodies to focus more upon this critical enabling infrastructure.

3.5.2 Publicly available NLP tools to support mapping into core taxonomies. This approach would also require the public release of tools to encourage good citizenship across the skills ecosystem. For example, tools that support the mapping of other skills taxonomies and descriptors into those that are designated as core would make it possible for individuals to port their skills passport to core taxonomies in other domains, irrespective of how much work their provider had completed on data portability. While this approach would not guarantee trusted mappings, it would encourage the emergence of tools to support people in moving their passports as required while waiting for improvements in technology, so helping to ensure that individuals are not “trapped” in one skills sector. Individual users, or perhaps even a set of validation services would likely be required to approve the resulting mappings. At UTS, we are investigating the possibility of constructing models that can rapidly map between any two skills taxonomies or capability frameworks. Our solution uses GPT to generate lists of skills that are representative of a given skill or capability. These are then compared using cosine similarity over bags of skills to return a ranked list of likely matches in the second taxonomy.

Similarly, educational institutions would benefit from tools that support the tagging of their curriculum descriptions into core skills taxonomies. These tools could be semi-automated using NLP (e.g. the skills extraction tool released by Singapore²⁹) or require human input to select skills against a defined taxonomy (which would enable instructors to maintain some control over how their subjects are represented in a taxonomy). In both cases human oversight would again be required to approve suggested skills tags or make decisions based upon the resulting analytics. For example, our early work on using NLP tools provided by BG took its skills taxonomy as a controlled vocabulary for automating the Recognition of Prior Learning (RPL) [35], and demonstrated that out of the box NLP tools could be used to build up more extensive services that would help to encourage good citizenship in this space.

3.5.3 A global look up service. For this solution to work it would be necessary to store the information in a skills passport with contextual information designating which taxonomy its skills originated from. Named graphs [12] could perform this role, providing a mechanism for contextualising a skill or occupation listed in a skills passport to a specific taxonomy. However, other approaches are no doubt possible. Government agencies could take on this governance role, signing taxonomies that they had chosen to warrant as core to authorise their use in skills passports they consider valid in their domain. These agencies could then also become responsible for maintaining mapping tools to support translation to other core taxonomies. This brings us to the final critical element for a useful skills passport.

3.5.4 Governance. The above proposal is designed to be localisable to a specific jurisdiction during an initial set up phase. This makes it possible for one domain to start developing a skills passport within their own border, but to plan for eventual user model portability to other domains. However, ensuring that a skills passport could function across all skills-sector boundaries would likely require substantial governance and oversight. It is likely that some sort of governing or standards body would need to be created to manage communication between the different local authorities, and to approve the designation of new taxonomies as core. A set of expectations are likely to be necessary to elevate a taxonomy to core status. Some criteria that seem likely to form an initial minimal set could include:

1. A core skills taxonomy MUST describe each skill it lists with an id, a name, and a brief description for use in NLP.
   
2. The server for the taxonomy MUST provide at least one cross-walk of its skills to another core taxonomy.
   
3. The service SHOULD provide cross-walks to all other core taxonomies.
   
4. A set of public APIs SHOULD be made available to facilitate other mappings and services.
   

Components of this ecosystem already exist. In particular, ESCO is a fully defined ontology that contains ids, names and a description of each entity, and as previously discussed, a cross-walk has been created between ESCO and O*Net-SOC. As such, ESCO is largely in compliance with the above compulsory requirements. Note however, that the ESCO-O*Net-SOC cross-walk concerns only occupations, and that no further NLP services or APIs currently exist. More work is required to achieve all requirements and to map occupations to skills claims in a passport, but this is in principle possible.

3.5.5 An embryonic user model for a portable skills passport. It seems that the technology for creating a minimal viable product (MVP) that could cross one skills sector border (Europe to the USA) is within reach. A skills passport provider that used ESCO (e.g. Europass) should in principle be able to translate any claims that a user made in their passport about previous occupations that they had worked in to the US occupations listed in O*Net-SOC. Furthermore, each of those occupation claims could be linked to the skills listed in both taxonomies right now, which would yield an embryonic skills profile capable of crossing the skills sector border.

Mapping skills across both taxonomies would potentially also help to identify equivalences between the skills listed in each occupation and so support the creation of skills cross-walks between ESCO and O*Net-SOC. This would be an important step for the user model to move beyond a situation where only occupations could be ported between the two geographical domains. Indeed, the entire point of a skills passport is to disaggregate the coarse-grained representation of courses and occupations down to a finer grained skills based user model that would be more suitable for personalisation. Hence this problem of mapping between skills taxonomies remains a critical enabling step.

Such mappings would bring other benefits as well. For example, mapping from occupations to skills in the ESCO-O*Net-SOC cross-walk may also enable the identification of possibilities for contextualising skills with a level designation (e.g. introductory, intermediate and advanced), if certain occupations could be identified as requiring more advanced capabilities. This would assist with the personalisation of training pathways between job roles that shared similar skills but at different levels of capability. An extension of this user model could eventually include the ability for individuals to rate their competency (or not) in a given skill, as well as to claim new skills. Note that this program of work makes the creation of a genuine skills cross-walk more urgent.

More work remains to be completed. However, the lack of progress toward an MVP of this form comes from social factors as much as technical ones, with research teams, commercial providers, and government agencies all tending to work in their own local domains and failing to support the broader global context. If a MVP were to emerge that provided this first simple skills based user model then perhaps more sectors would see benefit in working towards convergence. Thus, an MVP could encourage the mapping of more taxonomies into the already declared core ones, so helping us to develop a socio-technical infrastructure that could in fact cross the skills sector border.