AI-driven role matching: linking candidates to jobs using machine learning in big structured data
Exploring how AI and deep learning can be used to solve the problem of role matching contract staff.
HR GO plc is a family owned group of companies, split into the Recruitment and Digital sectors. Their recruitment group is a 200-person strong operation turning over £80m and operating across the market in temporary, contract and permanent recruitment.
The main aim of this project was to produce recommendations on a method by which HR GO can quickly segment and categorise candidates, to better match candidates to jobs and to improve candidates’ experiences. Specifically these recommendations should cover the use of machine learning to calculate a “suitability index” for a candidate for a given job.
The Challenge
HR GO manage over 5000 contractors a week and have a lot of data on how people performed in the various contract roles. This data would be used to create a model related to each candidates’ applications, CV content data and role description matching. The goal would be to score candidates with a “suitability index” or likely success probability, for roles. This offered potential improvements in forecasting success and candidate engagement through improved feedback and candidate matching.
There was also potential for candidate personal development by providing an indication of training the candidates may require to improve their success rate with potential employers. Care needed to be taken with regards to the treatment of the data investigated to avoid any bias or other ethical issues surrounding the analysis of personal data.
The Approach
Dr Anna Jordanous from the University of Kent’s School of Computing is an expert in the field of AI with specific interests in computational creativity and machine learning.
The main tasks were to investigate and report findings on relevant data-driven machine learning tools and techniques within the context of candidate data, role matching and candidate support. Various dimensionality reduction techniques (to make the data more tractable) and methods for dealing with missing data (e.g. incomplete profiles) needed to be considered. The final aim was to produce a research based, investigative report on the pros and cons of using various machine learning approaches on the data. For example, a comparison was made between the use of various techniques for regression and clustering. Ethical issues related to this use of AI were also investigated in depth.
Machine learning can be described as algorithms that improve by being “tuned” through exposure to data. Machine learning is a technique of creating software “models” that are trained from huge volumes of data and then used to predict certain patterns, trends, and outcomes. [Fig 1-2].
Regression is a machine learning technique which uses the data provided during training to improve the accuracy of the algorithm at making predictions of some anticipated outcome. Regression takes data as input and calculates some numeric value – that value represents the machine learning algorithm’s prediction (probability) of something e.g. likelihood of being suitable for a particular job.
Classification is a technique which is used to classify data into different categories or groups based on the relationship between the attributes (features) in the data and the identified (labelled) output. For instance, individuals in the training data could be classed with an associated output / outcome based on other attributes present in the data which determine natural groupings and patterns in the dataset and can be used to predict the grouping classifications e.g. classification by types of job related to qualifications, experience, age or gender.
For this project, the regression technique was the most appropriate method to use.
Figure 2-2 illustrates the general flow of creating new prediction models based on the use of supervised learning along with known input data elements and known outcomes to create an entirely new prediction model. A supervised learning algorithm analyses the known inputs and known outcomes from training data. It then produces a prediction model based on applying algorithms that are capable of making inferences about the data (Figure 2-3).
“This is a problem of scale: the data are big, rich and diverse – this causes lots of interesting problems for machine learning! HR GO have a large amount of unique data, offering lots of information which existing off-the-shelf solutions for role matching are unlikely to incorporate.” – Dr Anna Jordanous, University of Kent
The Outcome
The final output of the project was a comprehensive report and review of the HR GO data and potential machine learning solutions which could be deployed to automate contractor candidate role matching.
Progress was affected by a couple of factors, for example, more emphasis needed to be placed on the importance of the job data as well as the candidate data for role matching, so more follow-up data examples were requested and provided.
There were issues with the job data not being as rich for machine learning as had originally been anticipated, so strategies progressed from the original intended range of options to making two sets of recommendations.
John Parkinson, Group CIO at HR GO had this to say about how the project has benefitted the business:
“The experience of working with Anna Jordanous and Don Shaw-Case from The University of Kent has been amazing. Both the management of the EIRA application process and the work with Anna thereafter has been simple, efficient, productive and exceptionally valuable. The project has been very focused on how technology and innovation can be applied to the commercial realities of our business. Anna and Don have taken real efforts to understand our business model, nomenclature and quirks which has yielded a clear, relevant, actionable and commercial set of recommendations from Anna in her report. The project has opened communication between us and The University of Kent. We are now in discussions about how we can move forward with other projects benefitting both sides.
It is wonderful that there are opportunities for businessess to obtain insight and guidance from such experts.
At HR GO we receive hundreds of thousands of applications for roles each year. At the core of our business we are, and always will be, a people business: our consultants are personal and strategic advisors to both candidates and clients who understand the market. We care most about finding the right people for the right job. Therefore, it’s so important to us that we make the most out of every contact we receive from a candidate and that is where Anna has focused her attention.
She has taken a deep look at our business and data and made recommendations about how we could use Machine Learning to improve the experience of our candidates by delivering some automated feedback, clustering and matching. Yet, in keeping with our culture, she has taken real care to note all of the ethical points, both positive and negative, that we must consider when implementing ML in the recruitment process.
We have taken Anna’s recommendations, which are both things we can immediately action and more strategic opportunities, and will be implementing them within our business.” – John Parkinson, Group CIO, HR GO
Next Steps
HR GO are a large and diverse national business and have expressed a desire to continue to work with the academic community. They are taking the recommendations from this project and seeking to turn these into a larger company-wide project to aid in automating the contractor candidate matching and suitability indexing.
References
- Microsoft Azure Machine Learning: Essentials; Jeff Barnes ; Microsoft Press 2015; ISBN: 978-0-7356-9817-8
Photo by Jason Leung on Unsplash
