Every week, we talk about important data and analytics topics with data science leaders from around the world on Facebook Live. You can subscribe to the DataTalk podcast on iTunes, Google Play, Stitcher, SoundCloud and Spotify.
This data science video and podcast series is part of Experian’s effort to help people understand how data-powered decisions can help organizations develop innovative solutions and drive more business.
To keep up with upcoming events, join our Data Science Community on Facebook or check out the archive of recent data science videos. To suggest future data science topics or guests, please contact Mike Delgado.
Here’s a full transcript:
Mike Delgado: Hello and welcome to our weekly Data Talk, a show featuring data science leaders from around the world. This is going to be a very exciting chat because we’re talking about AI and how it’s helping recruiters find the right candidates for jobs, and it’s actually a fascinating field. Has to do with prediction, has to do with artificial intelligence, and I’m very excited to talk to Kristen Hamilton, who is the CEO and co-founder of a company called Koru, based in Seattle, and Kristen, great to have you on our show today.
Kristen: Michael, it’s really great to be here, thanks.
Mike: So Kristen, can you kind of tell us about your journey that brought you into working in data science?
Kristen: Sure. I think every entrepreneurial journey starts with a problem that you’ve become impassioned with, that you really can’t sleep well at night until you figure out how to solve. And the challenge that we initially started facing was actually the gap between education and employment. So post-college, 53% of people were underemployed or unemployed, and there were five million open jobs in the country, so we really didn’t understand what was going on there, other than to hear that there was this skills gap. That was the problem we set out to really address and solv
As we dug in, we learned that there were two sides to this problem. There were the job seekers and applicants, and there were the employers, who, of course, were the customers of the talent, if you will. So, that was the business context. Now, you asked how we got into data science. Something I think about data science is, at some point, it’s like saying the use of technology, or in the old days, the use of a pen. It is a tool to help you solve a problem. It’s a very, very effective tool that has become much more fine in its ability to solve problems with both, large data sets and also small data sets.
The advancement of machine learning modeling techniques has been very significant in terms of our ability to solve many problems, and that was the case for us. We ultimately spent a fair bit of time looking at existing research and doing research of our own, to identify the skills that drive performance in today’s economy; so the innovation economy that we live in. The world is different than it was 15, 20 years ago, and the skills required to be effective have changed significantly. Obviously, data being one of those key drivers. The world is surrounded with data, how do we use that data to make good decisions? So, it’s more important to know what questions to ask, for instance, than what the answers are and content and all of that is easy to get by doing Google searches, but it’s really higher order thinking that’s so critical.
We sort of identified those drivers of success and we created, ultimately, a list of seven competencies, including things like grit and rigor, analytical rigor, teamwork, curiosity, ownership, polish. And they have a set of sub-competencies that are research based, but the part where the data science really came in and we ended up deciding to use this tool to be effective in solving our problem, was that we needed to define and identify empirically, which of those factors, which are generally predictive of performance in work, were specifically predictive of performance in a given role.
We work with them in the US on their analyst hires. We published a case study that was in LinkedIn Mobile Recruiting Trends 2018 about this, but we work with their investment bank and their corporate bank, and we needed to identify, what are the drivers of performance in those roles? An analyst role in the investment bank in the US.
Since we’ve been doing this awhile, I’ll tell you the punchline. We’ve learned through machine learning modeling that, the drivers of performance in one job, the same job, in one company, are different than the same job in a different company, even if they’re in the same industry and do relatively the same thing.
Mike: Hmm, that’s interesting.
Kristen: Yeah. So investment banking analysts in Company A, what it takes to succeed is different than in Company B. Same in sales. And you hear these stories anecdotally like, “This person crushed it at Oracle as a salesperson in enterprise and we hired them at my growing startup and they really [inaudible 00:04:46] work for them.” Because the drivers of success differ by organization based on things like what your product is, what your leadership style is, what your culture is, et cetera.
So the data science helped us to very specifically identify those drivers that were statistically significant in causation and correlation of performance as the outcome in this case. And how we did that is we have a data collection mechanism that collects about 450 data points from each applicant.
Mike: Oh wow.
Kristen: Or each incumbent employee. So we start building the models using existing employees and we collect data from a set of employees. We have about 450 data points per employee. And then we actually get performance data from the employer. What that allows us to do is to separate two groups which is key in the data. So we can separate the high performers from the other.
And then we use four to five machine learning models to see which drives the best signal in the key differences between high performers and low performers. And typically what emerges from those potential 450 drivers of performance is five to ten things that are significant correlated with high performers and not with low performers. And then collectively the entire set of those things, so let’s say the five to ten things, becomes the predictive model. Or, you know, sometimes we refer to it as the, “Fingerprint for success,” in that role.
And so again, through machine learning modeling we have now been able to take what has largely been guess work, historically. And frankly it has been the cause of challenge that it’s guess work because what you think drives performance in the sales hiring we’re collectively making is different than what I think and we’re both interviewing this person. We both have influence over who to hire. And we do survey existing influencers and hiring managers and recruiters and we typically find that there is not alignment because they don’t have agreement and a database picture of what drives performance.
So anyway, that’s kind of how it works in a snapshot and we have leaned quite heavily on these modeling techniques as well as the matching techniques which are … You know technically it’s AI when you take an applicant and then you compare that applicant to this predictive model and we assign them a fit score. And we love that word “fit”, you know fit is really the prediction of performance of a person in a role. And fit is two sided and so ultimately “fit” works very well for the applicant because if they are a good fit they will retain longer, they will perform better, it’s also of course good for the employer because they drive better results.
Mike: Wow that is fascinating. So tell me about, you know, you start working with a brand new company, maybe it’s a start up, and you kind of have a model already in place to kind of … You know you wanna have 450 data points on current employees to figure out and then divide them into successful employees and those who are maybe not as successful. What are some of those data points you guys are looking at?
Kristen: Yes, we have three different categories of data points that we look at. We only look at things that are known to be predictive of performance. I’ll give you something that’s not known to be predictive of performance: introversion versus extroversion. If someone is introverted or extroverted has not been seen to be a good measure of performance. It’s super interesting to know about somebody post-hire so that we know how to work well together with some of those more personality traits.
Kristen: But the key [inaudible 00:08:32] of things that drive performance are related to first of all your raw horse power in terms of IQ, that is a key. The other is the skills to do the job, so if you have practice and the skills to do the job. If you’re gonna write code, you have to know how to write code. And the third is this set of soft skills which are really getting a tremendous amount of attention now.
So we measure components that fit into those categories. We measure the Koru seven, so grit, rigor, impact, teamwork, curiosity, ownership, and polish. And some of those are self-explanatory but you know teamwork is not about having played four years of college sports. Sometimes people think, “Oh, he’s gonna be a great team player,” or, “She’s gonna be a great team player she played a bunch of sports, she’s been on teams.” You know it really is about recognizing emotions in others and being able to draw out the best from a team and that’s how you unlock power diversity. So it’s pretty exciting to understand these definitions and be able to then measure them in a very specific way.
And so what’s super fun is that these things used to be intangible and now we’ve found and we have this assessment so we also lean deeply on assessment science in addition to data science to be able to effectively, in a validated and unbiased way, measure something like grit and teamwork. So that’s one category and importantly we have labels for these things and actually there are sub-competencies so there are 21 and not 7, but some companies already have their definitions of competencies. They might say, “We call it being achieving-oriented,” “We call it something catchy like ‘winning is fun,’ and that’s one of our company values.” What we’ll do is we’ll actually help to first understand what they mean by that and then we’ll map the sub-competencies so they can use their own language. But it’s the same outcome, it’s this sort of, “How do we measure these work traits that are soft skills that have heretofore been probably considered less important and not able to be measured in a tangible way?” So that’s step one.
The other thing we look at, though, is transferable skills. So the research says that if you have practiced something, you’ve developed the muscle and you’ve done it many times, that you have a competency in that thing. So we ask people, even if they have limited work experience, I mean you can look at work and non-work experience. Maybe it’s in a political campaign, maybe it was in a club indoor team: “What are the skills you’ve developed?” And it’s from a very specific structured pick list. So you’ve developed the ability to teach or coach others or resolve conflict or do data analysis significantly, that could’ve been in your biology work, right? It may not have to do with business. We identify those areas where they actually have had a tremendous amount of practice, but we put it in the terms of business skills so that there is consistency and there’s structured data and the skills are transferable from one domain to the other.
And then we look also at biodata related to preferences and experience. So I’ll give you an example: sometimes we see predictors such as having worked a minimum of 10 hours per week during college or having previous experience in the retail sector being a better driver of performance for a sales job than having previous experience in sales. Because you can imagine when you learn this as an employer or a talent acquisition leader or CHRO, or even a CEO or a recruiter, everybody gets a sense of, “Wow, we thought this and now we know that.” And it’s a data-driven answer. So we have the most success in companies who are very willing to and hungry to be informed by data. And we see the number of those companies increasing, you know we’ve seen this revolution in data-driven operational approaches in sales and in marketing and in operations and now we’re finally starting to see it in recruiting.
Mike: You mentioned the differences between extroverts and introverts. Can you kind of elaborate on that?
Kristen: Yeah well importantly that is a personality trait. There used to be an approach to assessment that measured what we call the Big Five and introversion/extroversion is one of them. And we have looked at correlation coefficients between those Big Five traits and performance, because we have data to be able to do that, compared to the correlation between something like grit. And we have identified that the correlation coefficient between being an introvert or an extrovert and performance is extremely low.
Kristen: And actually is true for a significant for not all, but a lot of the Big Five, actually. And so that has caused us to understand that things really have changed or we learned more since we developed that list. And that’s really helped us to choose what should be the Koru Seven. So we empirically looked at, “Okay, here’s how we define grit. Let’s measure grit in a lot of people and then let’s look at people with high grit scores and their correlation with performance.” The correlation with performance for something like grit is much much higher than something like introversion or extroversion, so that’s sort of a trait that’s interesting to know about people and affects how they work. It does not affect their performance.
Mike: Hmm. I read Angela Duckworth’s book on grit, which is excellent. And she has a survey in the book where you can take it to figure out like how “gritty” you are. I’m curious how you are determining the “grittiness factor” of somebody.
Kristen: So we have tremendous respect for Angela, we’ve had breakfast with her, my co-founder and I, a couple of times.
She historically was funded by the Gates Foundation and my co-founder was at the Gates Foundation during that time and so we’ve been following her work quite closely and been, in some extent, involved in her work historically in the case of Josh from Gates. And she is the expert on grit, there’s no doubt about it, because of her work and researching how to define it in particular. She self-admittedly hasn’t really been focused on the assessment of it. Clearly in any assessment one of the key things is to ensure that your assessment is not gameable. And so if you truthfully answer her questions you’ll get a measure of grit. But it’s easy to not truthfully answer questions.
Mike: Mm-hmm (affirmative).
Kristen: “Yes, I always persevere.” So we follow, though, her definition and all of the research that she’s done around grit. And we have found ways to use assessment science to measure it in a way that it can’t be gamed. So specifically, the measure of grit: one way to think about measuring grit is how long you persevered in an activity or a job, but in particular it could be in an activity even, and the highest role obtained in that job. So let’s just say you were in the debate society: how long did you do that for? A few months? Or did you do that for four years? And did you become the chair of the club or were you a member of the club? So points are assigned for these things. And we can pull some of that data from LinkedIn profiles and resumes, so there’s ways that we can check that they’re being truthful.
And that’s an example of how we’ve taken research that’s been done in things like grit and also growth mindset and also things like organizational citizenship behavior and [inaudible 00:16:14]. There’s research that has been done by others in various areas you focus on. And taking what they have learned but then implemented that in such a way that it is an engaging assessment and that it is not gameable.
Mike: I think what’s cool about AI in helping with finding the correct applicants is it helps to remove human bias. Now, obviously there could be bias within AI and that’s something that needs to be worked on, but I think about how oftentimes you know you hear the stories of, “Oh, well they’re an alumni of the same school I went to, therefore they should be hired …” you know, “I’m gonna hire more USC graduates because that’s where I went to school and I wanna be faithful to my school and my alumni.” But that’s a college bias. And I think with AI, hopefully, we can help to remove some of that bias so that we’re bringing in the right talent. But I’m curious, from your perspective are there problems of bias within AI that you’re currently working on?
Kristen: It’s such an important question and I think we capture the theme of how we’re trying to help catalyze the change in this area of biasing. We want to support the grit over grades revolution.
Mike: I like that.
Kristen: Because the way things are done today, let’s talk about the way things are done today. The example you gave is quite common. It may or may not be related to you as an alumni in that institution, but typically people choose a small number of colleges. They tend to be top ranked colleges and they say, “That’s where we recruit and we’ll do a cut-off at GPA.” So that is bias for certain and indeed is also not getting employers what they need for a variety of reason. Number one, you take investment banks and consulting firms are now competing with tech companies for those same students. Everyone started focusing on the same group of people. Secondly, we’ve seen that top colleges mostly correlated with short tenure. So people get recruited away for the same reason so we’ve actually seen data that the higher ranked the college, the less likely they are to stay. So it can be counterintuitive.
Google did their study that was published a couple years ago in the New York Times with Tom Friedman and Laszlo Bock, this research that said, “Let’s look at everybody who’s ever applied to Google and how they performed and so forth,” and they were very focused on EQ, IQ in particular. Sorry, they were focused on people who were just super smart and went to top colleges and high GPAs and they found no correlation. So they realized, “Wow, it’s actually these other factors like, ‘Can you work well with people?’ ‘Are you able to influence, etc?” So it was the soft skills, that’s what they came up with. So the bias is not helping; there is bias and it’s not helping.
So I think there’s a couple different ways to be biased. One is that, you know, selection bias based on your selection approach. The other is unknown and is harder to identify which is simply we tend to do confirmation bias when we need somebody, if they’re like us, we tend to naturally like them more after you’ve had your first bias of selection. And so we have looked at the data in a couple of cases and understood that.
So one company where we do ask in our assessment for diversity information so we know what their backgrounds are. And there were groups of underrepresented minorities that were scoring very well in terms of their fit score. So we knew they were quality candidates by definition with respect to the job at-hand.
And the percentage of applicants was reasonably high, but the percentage of hires was much lower than the applicant percentage. So they were dropping off somewhere in the funnel. And we realized they were dropping off at first interview.
Kristen: it was really cool to be able to use the data to say, “First of all we’re not lowering the bar in terms of quality, these are quality applicants, but something’s going wrong. There’s bias in the system somewhere.” And rather than trying to identify that very complicated thing with the humans interviewing, we actually recognized that you could intervene by simply giving interview training to this group of applicants before they interviewed and helping them with the skill of interviewing and comfort in the environment. Because the job is actually not about how to interview well, but the [crosstalk 00:20:50] need to get the job, ironically. So we did that training because it’s actually part of what Koru does is to develop these skills.
Mike: That’s wonderful.
Kristen: So when we did that we saw the hire rate for this underrepresented minority group go up to twice that of white males.
Mike: Oh wow.
Kristen: Super exciting to see in the first place.
Mike: That’s awesome.
Kristen: Number one, using machine learning modeling to identify the drivers of performance so we could score their predictive performance so they were high quality and then actually being able to use the data in the funnel that said, “Hey, they’re dropping off at this stage.” And so we can intervene and actually really drive a positive outcome.
Mike: I think that’s awesome, especially for those who struggle with the interviews. Because there are so many amazing people that could do the job well, but when they go to the interview they get so nervous. Their answers are shorter, they’re not sure how to answer properly, and so they get caught up. But it’s great to see how your company is helping with preparing with the interview. And so to see that it’s been double the rate to help bring in more capable people into the organization and also increase diversity, which is another plus.
Kristen: Mm-hmm (affirmative). Absolutely. Yeah, it’s pretty amazing. It’s really exciting to think about how you can use something that seems impersonal, in some ways, at initial first [inaudible 00:22:17] to say, “Oh, machines are going to be making decisions or making recommendations and so forth,” and actually have the outcome be that the process and experience is more personal and the outcomes are actually better diversity. And a couple things are important there.
I think the first piece is our approach to AI and not just [inaudible 00:22:39], but many these days, is that the data and the recommendations help humans be better deciders. So it really is augmenting the work in day-to-day time spent by recruiters. Instead of looking at resumes for six seconds each and saying, “Yes,” “No,” “Yes,” “No,” based on [crosstalk 00:22:58] factored, not very data-driven, you’re actually looking at a stack-ranked list of recommended applicants in a really beautiful, consumer-friendly tool that explains to you why this person is rated a 98 and this person is rated a 35. And by the way, that 35 may be a really good fit over here in this other company. So we can compare applicants to different hiring profiles and different predictive models and say, “Hey, you’re not a fit at Citi, but you are a great fit for ZipRecruiter or you’re a great fit for Oracle,” or something like that.
So it’s helping those applicants in the same way that marketing professionals were guided and aided by the tool sets that they were given in technology stacks for marketing several years ago. It’s moving the recruiter into that same realm of being a professional and actually probably someone who can more effectively use data. Because they don’t need to be a data scientist themselves, but they simply need to be able to understand how the tool is serving their needs and then being able to leverage it effectively.
Mike: There’s a lot of data scientists in our community on Facebook and one of the top questions we get all the time is, “I’m currently a graduate student, I’m in computer science or statistics, gonna be graduating soon, looking to get my first job,” and it’s obviously a struggle for many of them because so many jobs for data science positions require X number of years experience. And that is hard to get when you never had a job. So how can people who are looking to get their first job kind of prepare themselves to help improve their kind of fit score, if the company was using Koru, to help them with maybe being seen or being more visible to a recruiter?
Kristen: Yeah, really good question. So I think the first thing is if you go to Koru’s website, so it’s joinkoru.com, the definitions of the Koru Seven are very well articulated. And the first step is self-awareness. So we do this when we teach people about the Koru Seven, but we’ll actually help them to understand what each of these things means and to identify which of those are strengths of theirs and which of them are areas of opportunity. And it’s just like anything; you need to get reps, right?
So knowing that … If you’re coming straight out of college these days, you’re in a challenging position because you both need to have grit and grades, you know you need to have … At this point the degree is so meaningful but you also need to have a hard skill and you need to have the ability to kind of position yourself as useful. And so thinking about translating the experiences that you’ve had; whether it’s on an athletic team, whether it’s in your thesis and your biology work, whether it’s something that you did working for a political campaign – looking through the lens of the Koru Seven. And so that when you positioned yourself when you’re applying that you talk about those things in that language, but importantly that you actually develop those skills. So if you were to take the Koru Seven assessment, you’ll be asked about your experience and you’ll be asked about your preferences. And the more you have actually done things to develop these skills, the more effective you will be in the workplace.
The part of it is really awareness and we’re actually even working, in some cases, with high schools to help people who are coming out of high school, and before they even enter university, to understand these skills and traits and then they are more able to develop them and also express them as they’re looking for work.
The world’s just changed so much I mean it’s not enough to be a top student from a top school and to have had that internship at some great company because, “What can you do for me today?” is the question employers are asking themselves. And because the average tenure of someone in their 20’s is 18 months … So if you can’t ramp to productivity faster than that, you’re costing them more than you’re giving them in a way. So you wanna be able to demonstrate your ability to effectively operate in an ambiguous environment where there’s no syllabus, the answers are not always there, you have to go find them, and you also need to know how to spend your time effectively to drive impact.
Mike: So for the data scientist who is just getting out of college, you know, they have the hard skills: they have the maths, the sciences behind them, they know how to program. So let’s talk briefly about the soft skills because you’ve touched on that quite a bit and the importance of the soft skills. What would you say is probably the most important soft skills that they can probably work on and help to show the recruiter that they have those skills?
Kristen: Yeah I think, assuming they have a hard degree, a [inaudible 00:28:41] degree, like a data science degree, then presumably they’re going to score very high on rigor and that’ll be a box that is checked effectively. So you’ve got the ability to analyze data and that’s great. The next way of thinking about it I think is … And the way to develop these skills, by the way, I think is like taking a model in the wild yourself. So find a way you can take the unpaid way, volunteer way, some personal research way, to actually build a model and take it into the wild and try and implement it. Because the things that you will learn will be tremendous.
Partially, I think that the teamwork skills in general are wildly important because there are no dark corners in organizations anymore. Everybody needs to collaborate. It is true of people writing code and peer programming models, it is true of people who are building data science models. The ability to communicate the challenges of a given model and the trade-offs is incredibly important.
So those teamwork skills are critical and if you’re somebody who prefers to be independent or alone, the news for you is that you can’t always be independent and work alone. And the most effective skill around teamwork is the ability to draw out the best from others; so good questions and good listening skills and emotional intelligence and awareness of a room actually are the drivers of performance and teamwork. So instead of saying, “I know this, I’m pushing my answer,” actually being observant of other people who could help you is really critical.
And then I think the other piece that comes up a lot with people on hard skills is the communication skills. So “polish” is in Koru Seven and polish is probably not the best word for it, it’s really about effective and authentic communication. And so the ability to clearly explain and authentically explain most important components of your work in all formats: written, oral, large presentation, individual one-on-one meetings, is very important. For a lot of younger people these days, they have been interacting more with technology than with humans in some cases than in previous generations and so that comfort level of articulating that in life settings goes way up when you’re in the workplace.
And the final thing I think I would say is impact. So impact is about driving business impact and knowing how to prioritize your time and treasure to drive business and backdoor to achieve goals. So the first question is, “What are the goals of what we’re trying to achieve here?” And it seems straightforward, but a lot of people don’t stop and ask that question, “Why are we building this model? What’s the outcome we seek to achieve?” And that’s the first step of impact. The next thing is to keep that in mind with every single action. And so when you think about whether you are analyzing data, whether you are putting together a story that the data is telling you or could tell, you can use the data to tell multiple stories, typically. So you wanna really make sure that you are focusing it on the problem at hand and the [inaudible 00:31:51] best use of your time and also the data that you have at hand.
Mike: For those listening to the podcast, we’re talking to Kristen Hamilton, she is the co-founder and CEO of Koru. You gotta check out her website, joinkoru.com, that’s K-O-R-U.com. If you’d like to see the transcript, get links to connect with Kristen, the blog on the Experian website is just ex.pn/datatalk45, again the URL is just ex.pn/datatalk45. That’ll bring you over to the video that we had on Facebook Live along with a full transcript and links to resources that Kristen has mentioned.
I wish we had a full hour because this has been fascinating talking to Kristen about this topic. This is definitely the future and it’s so cool to talk to somebody who is leading the way in HR and artificial intelligence. But before we go, Kristen, what excites you about the future of AI in HR?
Kristen: I think that it has the potential to stop the insanity of doing the same thing over and over again and recruiting and expecting a different result. So anyone who works somewhere and they have a relationship with their recruiting team or with the head of sales, head of HR, head of talent acquisition, and leave some data. There’s a real opportunity for us to collaborate, so I get really excited about that opportunity. And ultimately, for applicants, it levels the playing field because you get opportunities based on what you can do as opposed to based on pedigree or where you were lucky enough to get into school when you were 17.
Mike: Awesome. And for those that are interested in learning more, check out joinkoru.com. Kristen, what will people find when they get there, even if they’re not like a recruiter but they just wanna figure out more about assessments and those types of things?
Kristen: If you go to resource section of our website there is a tremendous amount of depth in terms of background research that supports the work that we do, case studies of examples where it’s been used, and tools not only for recruiters but also for applicants to understand how they could use these trades and skills and this research in interviewing and in talent acquisition. So I would love to hear your feedback and comments to any of that content and hopefully it is useful to you all. And if you have more interest there’s also the ability to see the product and how we do the assessment piece and how we do the stack ranking and rating of applicants, so it’ll give you a real feel of the product. And of course we’re happy to provide a demo of the product if that would be useful as well.
Mike: Wonderful. Well Kristen, thank you so much for your time, it was a pleasure talking with you. I feel like we could talk for hours, this is just a fascinating topic. Thank you for all your work that you’re doing to help level the playing field for applicants, for sharing the main attributes you guys are looking at, and just exciting to hear about what you guys are gonna be doing in the future. So I’ll definitely be checking out joinkoru.com more often to see about new products and services you guys are providing.
Kristen: Thanks for bringing the data science community together, Michael, it’s really a great service that you’re doing to all of us as well.
Mike: Thank you Kristen and have a wonderful day.
Kristen: Likewise. Take care.
As a leading technology entrepreneur responsible for raising over $300M and a passion for impact, Kristen Hamilton has a successful track record driving value for customers and investors. The first company she co-founded, e-commerce pioneer Onvia, she took public in 2000.
Kristen’s professional tenure as a leading expert in predictive hiring and the competencies that drive performance in the innovation economy she co-founded Koru, the leader in predictive hiring based on proven soft skills that really drives performance.
Backed by rapper, Nas, Koru provides actionable insights to help employers navigate their journey towards using data and analytics to predict quality of hire in their applicants.