First demonstrator: ESCO ontologies and semantic matching
Last updated
Last updated
ESCO [3] describes in detail the skills required for an occupation. It currently provides around 13900 skills linked to 3008 descriptions of occupations. It aims to support job mobility across Europe by improving labour market integration and efficiency.
ESCO provides a common vocabulary and ontology developed around three themes: occupations, skills, and qualifications. In fact, it is based on three types of terms that support language analysis for text matching, namely:
preferred terms: They are not used for any other occupation or skill and thus are unique. Preferred terms are the ones that best represent an occupation or skill.
non-preferred terms: They can be synonyms but also spelling variants, declensions, abbreviations, etc. They are regularly used by jobseekers, employers, or education institutions to refer to concepts related to preferred terms.
hidden terms: Terms commonly used to address occupations but are considered outdated, misspelt, or politically incorrect. They are useful for indexing, searching, and text mining purposes but are invisible to the end users.
ESCO ontology, definitions, and structures aim to improve the matching of the profiles of learners, job opportunities, and learning resources. ESCO pillars are depicted in Figure 4.
To better understand how the ESCO skills and competences work, we will focus on the skills and competences pillar. At the very top level, four main concepts are found, namely: knowledge (K); Language skills and knowledge (L); Skills (S); and Transversals skills and competences (T). Each one of these top levels contains a hierarchy of concepts that is better illustrated by taking one of them to exemplify its usage. For instance, taking the “Agile project management” skill and walking up the hierarchy, we have the following path:
knowledge > business, administration and law > business and administration > management and administration > ICT project management methodologies > Agile project management.
Furthermore, every skill has the following properties:
Description: textual skill description
Alternative labels: alternative concept that represent a similar skill(s)
Skill type: top hierarchical level
Skill reusability level
Relationships
Broader concepts: related wider concepts
Optional for: related optional concepts
Concept Uri: ontological URI with a link to the hierarchical concept and all its related information.
Skill’s textual descriptions are used to embed a sentence-transformers model and then maps sentences and paragraphs to a 384-dimensional dense vector space that can be used for tasks like clustering or semantic search. The process follows the steps described below.
ESCO Skills descriptions are “embedded” using SBERT[4] with a model called “all-MiniLM-L6-v2”[5].
Course module contents are used as queries against the embedded model (trained with skills descriptions)
then we retrieve the ESCO skill most closely associated with the query (module contents).
The resulting model and index are saved to disk and used for similarity search based on the cosine similarity (other similarity metrics such as k-nearest-neighbor from Faiss can be easily used).
Notice that using all the textual descriptions to create an embedding of ESCO skills and competences allows us to match course textual module contents and obtain a similarity metric from this comparison.
Figure 5 shows a typical course description from a university offering. This is a structured document that contains the module name, professor name, qualifications targets, module contents, teaching methods, and additional relevant information for students.
From this document, we extract the relevant fields for our prototype, as shown in Figure 6.
Notice that the third column in the above figure is the module contents, this is the data field that we use to obtain a best match against ESCO skills descriptions. Thus, we use the course description from the “Agile Software Development” module to obtain the best ESCO skill match. Table 1 shows a typical matching output, including the top 5 results.
One of the main issues we encountered was the large number of embeddings available to vectorise the course’s content, the ecosystem currently provides more than seven thousand[6] models to choose from, focusing on sentence similarity topics. To determine which model to use, we used a subset of 50 WBS courses and carried more than a hundred simulations to determine the best model and which part of the course text to use.
The first question we wanted to answer was to determine which section of a course was the best predictor for finding skills. To achieve this, we used WBS course text split into three categories: course title, course description and course goals. Then we combined the text into different components, for instance, we merge the course title text and the description text, title and goals, goals and descriptions, and so on. This is depicted in Figure 7.
We used all these text snippets to match against ESCO skills description and calculated its semantic similarity based on the cosine metric using a fixed typical embedder (all-MiniLM-L6-v2). Then, the top ten semantic similarity responses were selected and then averaged into groups of the top 3, 5 and 10.
From this experiment, we learned that the best course descriptor to predict skills was the course learning goals followed by the course description, noticed that the combination of learning goals and course description produced poor results, and we hypnotise that this is due to the introduction of new text that increases text noise and dispersion.
The second question we wanted to answer was to determine which embedding models should be used. As previously mentioned, there are more than seven thousand models to choose from. Moreover, these models are changing and improving all the time, so we needed to establish a methodology that would allow us to compare embedders’ performance against each other to determine which embedding is the best to use to suggest ESCO skills.
Figure 8 shows a subset of the model’s comparisons carried out. This subset was selected by taking the top performers in standard benchmarks such as the HuggingFace leaderboard[7] for semantic textual similarity (STS).
As a criterion for model selection, we suggest focusing on the average score of the top 3 results together with the overlapping score. In this case, the overlapping score represents model consistency or robustness, for instance, the gte-large model shows that 30% of the suggested skills were the same using the text for course description and learning goals. All in all, gte-large is the best performer and the all-MiniLM-L6-v2 is the worst (see Figure 8).
These results should be interpreted correctly in the sense that gte-large uses a large context window, so it is slower, thus, more computationally expensive than all-MiniLM-L6-v2. Thus, there is a trade-off between model performance and speed.
The third question we wanted to answer is what skills should be suggested to users based on ESCO. As we have seen previously, the same course can produce different skill suggestions based on the text snipped that it is used (i.e. title vs description vs learning goals). Our overall recommendation is to use the intersection set of suggested skills, as depicted in Figure 9.
