Why Most AI Projects Fail w/ Ben Taylor at Ziff @BenTaylorData (Episode 33) #DataTalk

Listen to the podcast:

Every week, we talk about important data and analytics topics with data science leaders from around the world on Facebook Live.  You can subscribe to the DataTalk podcast on iTunes, Google Play, Stitcher, SoundCloud and Spotify.

This data science video and podcast series is part of Experian’s effort to help people understand how data-powered decisions can help organizations develop innovative solutions and drive more business.

To keep up with upcoming events, join our Data Science Community on Facebook or check out the archive of recent data science videos. To suggest future data science topics or guests, please contact Mike Delgado.

In this #DataTalk, we talked about the challenges of launching and completing A.I. projects with Ben Taylor , Chief Data Officer and Co-Founder of ZIFF, Inc.

Here is a full transcript of the interview:

Mike Delgado: Hello and welcome to Experian’s weekly Data Talk, a show where we feature some of the smartest people working in data science today. Today’s topic is Why Most AI Projects Fail, and we’re super honored to talk with Dr. Ben Taylor, who has been working in machine learning for over 13 years. Just a little bit about his background. He is the Chief Data Officer and Co-founder of ZIFF, Inc. He was formerly the Chief Data Scientist at HireVue. He got his Bachelor of Science and Master of Science in chemical engineering and then went on to earn his Ph.D. in chemical engineering at the University of Utah. Just a spectacular background. Ben, welcome to Data Talk.

Ben Taylor: Thank you. A quick correction, too. I’m actually still in my Ph.D. …

Mike Delgado: Oh, okay.

Ben Taylor: … and I think the likelihood of me finishing … We’ll see what actually happens. So, yeah. Sorry about the confusion.

Mike Delgado: Oh, no worries. I mean, just the fact that you are in a Ph.D. program, chemical engineering, is super impressive. Yeah, not that you need to complete it. You’re already doing some amazing work. I thought we could start off by you just sharing your journey.

Ben Taylor: Yeah.

Mike Delgado: What led you into data science?

Ben Taylor: There was no formal program, and we come from diverse backgrounds. My background is chemical engineering. My co-founder came from ecology. I have a bias toward physics, which is funny. So if I’m hiring and someone says they have a physics background, I’m like, “Really? Tell me more.”

Funny story. My wife accidentally hit one of our neighbors’ cars. I think I am the world’s worst neighbor, because I don’t know anyone. If I lived next to you for a decade, I’d be like, “What’s your name?” I’m just so focused and busy and everything else I’m juggling. But I found out that it was a college kid and he was studying physics. As soon as I found out, I was like, “Oh, really. You’re studying physics, huh?” And at HireVue, even … as a quick aside, we’ve hired three Ph.D.s with physics, and the CTO and I, we decided the fourth one will not be physics. We made a decision. We still hired physics.

Mike Delgado: That’s funny.

Ben Taylor: And I hope that’s not discouraging to anyone because I have lots of examples, people with really diverse backgrounds. But I did chemical engineering. I didn’t like it. When I got to grad school, I was liking it more. And then I got a job at Intel and Micron. This is their Lehi fab, so this is a semiconductor. I worked in the fab for a year, and then I worked outside the fab for three. The big breakthrough for me for data science is … the thing I tell people is that they’ve actually had data scientists for decades. Legit data scientists, and those have been in finance. They’re called quants. They’re quantitative analysts. They program really well, and they know insane amounts of math. They essentially all have guns to their heads to work ridiculous amounts of time, but they also get insane compensation. $700,000 a year compensation for individual contributors … Those were already existing before the data science hype.

I worked at a hedge fund for almost a year. It was a very powerful experience. It was also extremely stressful. We built out a million dollar GPU cluster in the office. This was five years ago. Today, that’d be, “Wow, that’s ambitious.” But five years ago, that was insanity. So they hired me as their GPU AI expert, and then they went back to Intel and Micron, and I just wasn’t happy. The hedge fund, it was really, really, really smart people. It was a fire hose of information, but it wasn’t a family-friendly work environment.

And then going back to Intel is very family-friendly. So it’s kind of, “What do I want?” And I decided I didn’t want to stay there, so I went and started searching out the job market at the time. People who had actually built AI solutions were in high demand, because there weren’t any. So the job market was really hot, and I went into the job market, and the funny thing with HireVue is … someone told me about HireVue, and they said, “Oh, there’s this HR company. You should take a look.” And as soon as you said the word HR, that’s an obvious disqualifier. It’s my experience with HR from Intel; we literally had people with master’s degrees who were throwing temper tantrums because they didn’t get a promotion to a mentor position. And it’s not their fault. These are people who have never had a job; they’ve never had things not go their way, and they actually have to get sent home for a temper tantrum. “Really?” A white-collar job and someone’s acting like that. And HR deals with that. So I saw HR as a place where people would babysit humans.

But I found out they were raising a round from Sequoia. And I think for California, Texas, New York and other areas. That’s probably not that special. But for Utah at the time, that was unique. I think we’ve had more, but at the time, there had only been three companies that had raised growth rounds from Sequoia. So I took a close look. I saw that they had the videos, and I got really excited about … You have videos with outcomes, and you should be able to do something pretty interesting there. So we built out interview prediction, which is a … it’s a hard problem. It’s also a fun problem because it’s very complicated. It’s one of the few problems where you do care about racism and sexism.

So if you name an industry, most likely that’s not actually a concern. The concern is prediction quality. “Ben, give me the highest accurate prediction that you can, or the hedge fund, “Go chase alpha.” But at HireVue, we can’t just do that, because that would be a recipe for disaster. And then … I love HireVue, fantastic job. Loved working there, and I had a lot of internal pressure to go do this startup because I saw a big opportunity.

Mike Delgado: So, I loved hearing your background, going from the hedge fund, high stress, smart people, but the work/life balance wasn’t there. And then, moving over to HireVue, which was… Now you’re working with new sets of data, and …

Ben Taylor: Yeah.

Mike Delgado: You’re making sure that gender and racism and that type of data set is … you’re able to work with that?

Ben Taylor: Yeah. It’s funny what makes a good job, because I’d argue most people actually don’t know. And it’s just like buying a house. If you’re buying a house for the first time, you actually don’t know what you want. You think you know, but it’s not until you go look at 50 homes that you realize this is a thing and I actually care about this. And so for me, I was arguing with, I had a flash where I really wanted to wear sandals. So that’s the Intel Micro group. I wanted to wear flip-flops, and that was important to me. And it was important enough to me, I was willing to dock pay over it. Like, I …

Mike Delgado: Are you … really?

Ben Taylor: Yeah. You have a decision matrix, and me wearing flip-flops, that was a thing and it was … If you’re willing to let me do that, then you can dock my pay. But if you’re not, then you’re going to pay me more.

And the other thing that was frustrating is … I wanted a laptop. A $3,000 laptop. A good Mac to do stuff. But at the organization, it’s one of those things where, to get that, they had to go up to three layers of management approval, where everyone gets the same $600 laptop no matter what. There are no exceptions.

So it was them not realizing that there needed to be exceptions for data science. Data science is a new, emerging field. Where now, they’ve completely changed. They are now caught up. They’re forward-looking. So when I interviewed HireVue, the engineer interviewing me was actually barefoot. And I joked with the CTO that that was important to me. So he’s barefoot during the interview.

And the other thing that surprised me with these startups is they told me the official time-off policy is no policy. I said, “No policy. What do you mean?” They’re like, “There’s no policy. No one tracks it.” Where at Intel, I had … I think once I showed up 35 minutes late, and they’re like, “Well, that’s 5 minutes past 30 minutes, and therefore we have to write that up.”

Mike Delgado: Oh my gosh.

Ben Taylor: Yeah.

Mike Delgado: Weird.

Ben Taylor: You can’t blame them, because it’s a manufacturing facility. They need 24-7 support, and they’ve got a lot of entry-level employees, technicians and engineers, and they can’t babysit people. They have to rely on you to keep that going. So that was … It’s funny what matters. Sometimes you have no idea until you do a few jobs.

Mike Delgado: So tell us now about ZIFF AI, the company you co-founded and are leading AI there.

Ben Taylor: It’s interesting that this even happened, because at HireVue … HireVue is my perfect job. We had had some major wins on the AI. We had an AI product. HireVue was my brand, really exploded. I think I went from having 400 connections and I’ve got 13,000, 15,000 followers. And I was speaking a lot, so I was traveling West Coast, East Coast. I speak in Sydney. And HireVue is doing a really good job with that. And I love my boss. And with the no time-off tracking policy, I was skiing an amazing amount of time. Like, I think I was averaging two days a week. And this isn’t all day, but this is me skiing…

Mike Delgado: That’s nice.

Ben Taylor: … two days, in the morning, just for these short ski runs. And they’re really generous. They were generous on pay. I had a team that I loved. All the things where you’d say, “Okay, why would you leave?” Like, “Why would anyone leave?” My wife was really happy. Everything was good.

But I think the thing that made this happen is when I looked at AI, just AI in general, and the different companies out there that attempt to provide it, I felt like the barrier to entry was way too high. You look at the Microsofts, the Googles, the Amazons, the IBMs, and I felt like, “Man, this should not be this hard. Why is it so hard to take a data set and actually provide value?” And for deep learning, it was even harder. The realization was this is just too big of an opportunity. If I stay at HireVue, I’ll be happy, but I may regret not taking this opportunity. And I think for me I’d rather attempt to launch a rocket and watch it explode and not … Because I can go back to the job market, and I won’t have any regret having attempted the startup. But if I stay at a cushy job for the next four years and then wonder what might have been, I’ll have regret. Failure doesn’t discourage me.

I think the realization doing a startup is, it’s 100 times … I’d like to say it’s 10 times harder than you thought it would be, but it really is 100 times. And it’s not fun, but it’s also not … Knowing what I know today, I would still go and do it. And I think Elon Musk has that quote where he says, “It’s like chewing glass and staring into the abyss.” It’s like juggling knives on a daily basis.

The point there is that HireVue, we actually had examples where we had dedicated resources for two months on a project that was a failed project. You’re like, “Oh, that didn’t work” or “Oh, the customer, they weren’t willing to pay for it. Maybe we shouldn’t have done that.” But on a startup, especially a bootstrap startup, that tightens the screws a little bit. Your mistakes are incredibly painful, and your time resolution is almost … it’s almost down to a day. So if I don’t work on the right thing today, that’ll hurt. And if I work on the wrong thing for a week, it’s like a fork in the eye. It’s just unbelievably painful.

Mike Delgado: Wow. And then that leads us to a blog post you wrote last year. I think it was in October. “Why Most AI Projects Fail.” And I loved reading your article and wanted to ask you a couple of questions about it. Very provocative headline, but you talked about science project sharks, and I was wondering if you can break that down. What do you mean by that? And how do the sharks steer data scientists off course?

Ben Taylor: Yeah. This happens a lot. A lot more often than you think. You think the business side would save you from this. I can understand a technical person saying, “Wouldn’t it be cool if we did this ridiculous project that’s very technical?” But we’ve heard really bad ideas come from managers and executives that are not technical, and they’re distractions. And so the joke we have is if you’re starting a project suggestion with “Wouldn’t it be cool if …,” the answer’s “No, that would not be cool. That’d be stupid and that would be a waste of time and resources.” And the thing that’s really scary is a lot of times this is their first AI project, which puts that much more pressure on it to be a success. Because if your first AI project is a failure, I would suggest not doing one this year. Don’t burn through that political capital or those resources.

Mike Delgado: Can you give me an example of some things you’ve heard about … Like, “Wouldn’t it be cool if …” What are some examples of things you’ve heard or things people say.

Ben Taylor: So one company we were talking to was fascinated with using deep learning for branding. The photo that’s coming into our system doesn’t match our branding because the photos are coming from somewhere that might not relate. You upload images from everywhere, and do the images match our branding? And can deep learning do that? Yeah, I’m sure. I’m sure deep learning can do that. Then what we found out is most of their success came from some outbound marketing campaign where everyone saw the same message. So there’s no targeted demographics, there’s no personas, and there was so much revenue attached to that … And so the thing we’d tell people is, “It’s great to have a lot of good ideas. Let’s come up with 10 really exciting AI projects and put them on the board. Then let’s rank them based on revenue. And I don’t care if you can’t come up with the calculations.” Maybe there’s so many assumptions where you’re like, “Geez, I know this is worth a lot of money. I can’t tell you how much.” Just write a number. Spend five or 10 seconds, write how many millions of dollars this project will save.

A good rule of thumb is if you don’t have any AI, you’re trying to just make up numbers. Maybe a conservative estimate could be a 10 percent improvement. That’s not going to be true for all … You can think of lots of use cases where you’re lucky to get a fraction of a percent improvement. But just in general, if I’m doing churn, if I’m doing lead analysis, if I’m doing some deep learning augmentation, just throw a 10 percent improvement on that problem, and then maybe you could bound it with like a 30 percent improvement. So you know this is my lower and upper. And if this happens, do I make a lot of money? And the data science project sharks are … They’re out there to distract you, and they can be really scary because we’ve seen lots of companies work on things they should not have ever started.

Mike Delgado: Yeah. I think, to your point, you’re saying that people are very curious. They have a lot of good ideas. They have maybe strong business backgrounds, but they never took courses in data science. I’m curious about … Because that can cause a serious communication problem.

Ben Taylor: Yeah, it can. And communication breakdown was one of the points I had for why projects fail, because the majority of data scientists, they’re very technical, but they’re actually … So, the joke I say is, “They’re geniuses when it comes to data and programming, but they’re morons when it comes to communicating to executives, or just to a normal person, and explaining what they do.”

The other thing I’ve been surprised by talking to executives, is … David, my co-founder, he’s done a lot more consulting. So he has more experience dealing with really dumb questions and kind of shockingly bad assumptions coming from the customer. One example I’ll give, I had a company that told us they wanted to do some task and I think it was lead … This is not something we would even do today because we only do deep learning. But years ago, we had a company reach out that had budget … This is coming from the executive, and they want to do some type of lead scoring to improve lead quality in it, and it backs up for them. So you’re getting executive buy-in, they’ve got budget, and the project is actually tied to revenue. So, the thing’s like, “Oh, this is a great project. Or it should be, hopefully.” And so I go and meet with them, and they pull up their data set for features. Okay, what are we going to be using for predictions? And it was literally first name, last name, address. There are ways to scratch at that, but if that’s all you have, no. Nope. The gut check for someone like an executive is, you can say, “If you stared at that screen all day, as a human, would you be able to predict that?”

Structured data sets can be hard when they get complicated, but usually, at least with some variables, there’s a gut check. And to be even darker, the only thing you may be able to accomplish is building a racist model. This is your name and you live in this part of … this is your ZIP Code. Where the data’s garbage. We tell people, “Look at the image, listen to the audio, look at the text. If you think you could predict that outcome, that’s encouraging. If you can’t, then there’s a good chance it’s a nonstarter.” It doesn’t mean you can, because there are examples of … like in manufacturing, you’ve got audio where you could predict a default on a tool or a fault. So if I’m in a big manufacturing facility, just listening to the sound on the mechanical machine, a human may not be able to differentiate, but a computer definitely could. The computer could predict you’re about to have a failure on this machine.

Mike Delgado: Wow.

Ben Taylor: So that’s an example of the human can’t, but the flip side there is you’re giving me an unstructured data set. You’re giving me a rich audio sample that goes into a spectrogram, and I don’t expect a human to compete with me there. But with structured data and text, you better be able to see something.

Mike Delgado: Wow, that’s amazing. I had no idea about what you just talked about, the tools, being able to listen to tools. That’s fascinating.

Ben Taylor: Yeah, it’s interesting. Some people joke that … Sometimes when I speak, I come across as being antihuman. And people say, “Ben, you’re a human. You know that, right?” Because I have so many examples of humans being replaced. And these aren’t just humans. These are Ivy League–educated … dedicated their entire career on linguistics, emotion, understanding personality, personas, speech phonemes, speech elements. There are so many examples where you’re looking at an Ivy League expert who dedicated decades of intense focus on a topic, and they’re being made irrelevant from AI.

We’ve talked a lot about this internally. There are some examples that are pretty dark, where there is no opportunity for them. One of the things we do see that is maybe a little bit more optimistic is some of these people should be involved with this transition where they’re involved with curating the data to train the AI. If you’re a linguistics expert, your ability to pull out topics and create value for downstream predictions is over. It’s game over. You can’t work to generate a new topic. But the thing you can do is work with AI to see what topics the AI is discovering and you can assign human-understandable concepts to those topics. And you can do quality checks. That’s how I see these experts being involved with some of the stuff.

There’s lots of examples where … And this is where some of the job loss will come from, so head-to-head, there are so many examples where superhuman accuracy is expected. And if your job is to compete against the computer, superhuman accuracy, then … One of the ones we’re excited about is home appraisal. We have some ridiculous home appraisal deep nets right now that are getting shockingly good home estimates …

Mike Delgado: Really?

Ben Taylor: Yeah, using images. They don’t just use number of beds, number of baths. One is your home bill; let’s do the comps. It’s all of that, all of the text, anything the realtor might have possibly said, and then all photos from your home. In some cases, that can be a lot. It can be over a hundred photos. And we love this because … When I’ve presented, I’ve said, “How many people have had their home appraised?” Anyone who’s over 20 or 30, all their hands go up, and you say, “How many people were happy?” Half the hands go up. “How many people were disappointed?” Half the hands up. And if you say, “How many people feel like they paid too much for the value of the appraisal?” That’s where all the hands go up. I feel like there’s no way this was worth $500 for you to look up my own price on Zillow. And I’m sure you didn’t just do that. You did the home comp, but …

So there’s an example of potential massive, massive disruption, but it’s not anything that’s going to happen this year, because there are banking regulations and there’s educating the market. But 10 years from now, or five years, you’re not going to have a home appraiser appraising my house. That would just be a huge waste of time. But you may have really good home appraisers who are curating the data. They’re checking the predictions where we’re not confident, and they’re looking at the mistakes where a realtor or someone says, “This appraisal is off.” It goes into a queue, and a human appraising expert reviews the mistake.

Mike Delgado: And banks would love that.

Ben Taylor: Oh yeah, banks would love that because the accuracy … I’m convinced today, the accuracy is higher. And there’s also some things where they’re considered to be subjective, and the appraiser has to not consider them. And so one of the ones I’ve heard about that is going to be pretty painful for people is landscaping. In Utah, if you spent $40,000 on landscaping, when they come to appraise your house, not only can I not consider the full value, I actually can’t consider it at all. So you get zero credit for that $40,000 landscape spend because they’re not equipped to grade you there.

Mike Delgado: That’s amazing. I know we’re coming close to the 30-minute mark, and I want to ask you a couple quick questions. The first one is, what do you look for when you are hiring a data scientist?

Ben Taylor: I’ve talked about this too and just written different blogs and posts. For me, the thing that I settle on is passion. And I might turn that up a little bit more and say obsession. So the thing that we … The job that I’m hiring you for, you’re actually not qualified to do a year from now. And I wouldn’t say, “You’re not qualified.” I’m actually not qualified. I’m not qualified to do your job a year from now, based on the skill set I have today. The biggest thing for me is hiring people who have a passion and an obsession, where I don’t have to hope that they stay up to date. They just naturally stay up to date. And I think for people to be really successful in this space, they … If it’s a passion, they’ll be just fine. Because they’ll work outside of work. They may do Kaggle competitions, they may do some side consulting. I’m not saying you have to work every weekend doing data science like I do, but if you’re passionate and you’re obsessed, that’s probably a thing. In the evening or on the weekend, maybe you went and bought a GPU gaming tower so you can do some deep learning. Those are the people I look for to hire, and they’re really hard to find.

Mike Delgado: Now, you mentioned in the very beginning of this broadcast how you’re drawn to somebody with a physics background.

Ben Taylor: Yeah.

Mike Delgado: What is it about the physics background that intrigues you and makes you …

Ben Taylor: So the thing that kind of makes me … it makes me a little bit of a hypocrite. I’ve told people, “I don’t care about your background,” and I don’t care, I don’t … So I have hired a college dropout. The story is, the high school dropout, college dropout, that’s fine. I don’t care. But what we’ve seen with physics is when it comes to the interview, they stand out with the technical …

So if I’m comparing people, technically, on how well they did on the interview … The example with one of our first physics hires, one of the questions was, “Tell me about your experience with natural language processing.” And his response was, “Well, I actually don’t have any experience with natural language processing, but I am going to tell you what I know about it.” And his response was more technical than all the people who said they had actually done work and provided value. And we’re like, “Let’s bring this guy in and talk to him more. And I would throw myself under the bus and say that humans are actually terrible at hiring. So the idea that physics is like a secret weapon. How many people have I hired? Not enough to generalize that and … But I am glad that we have hired a college dropout who is working out quite well. Which is kind of a … it’s a pretty wide extreme. I’ve hired Ph.D.s and college dropouts, but I really … I’d hire anyone if they had that passion and obsession.

Mike Delgado: That’s cool. Last question. What advice would you have for those people who are interested in pursuing careers in data science? Maybe they’re in college right now or high school, and they see this obviously isn’t a … this field’s blowing up.

Ben Taylor: Yeah.

Mike Delgado: What path would you want them to take?

Ben Taylor: A lot of computer science courses cater to that now. They’ll have a data science track. If I had to do it over, I maybe would do computer science to make sure I can understand how to use the latest and greatest available to us with writing good software. The other thing that I think is maybe discouraging for people in school is these universities are having a really hard time keeping up with industry. So the stuff they’re teaching, there’s a disconnect between what industry wants, and that’s a risk you take being … relying on the school to deliver the curriculum needed for you to get a job. That’s not something you should do. Going to school and learning the building blocks and the basic information you need to be successful, that’s good. So the feedback I’d give to people is definitely study data science if possible, because it’s not a market that’s going to go away. Like AI, you’re looking at decades and decades of really interesting jobs, really exciting problems.

Networking is important, and I’d say networking with people involved in the business. Going to meetups, meeting with people who work here full-time. Not just me. People reach out to us all the time for recommendations. Someone reached out a couple weeks ago saying, “Hey, I need to hire five deep learning data scientists,” which is insane. Hiring data scientists is hard enough, but hiring five with deep learning experience is like, “Well, good luck hiring one right now, but …”

So there are always these requests for recommendations. If you’re networked in with the right people, if I get a request for someone, what happens is you went from being part of hundreds in an applicant pool to being considered in the top three, top five. It allows you to just short-circuit that process to be considered. So networking is really important for these people who want to get jobs, and it aligns them with what businesses want. Because what universities teach and what businesses want, in all cases they’re different, and in some cases they’re sadly very, very different.

Mike Delgado: Great advice. I want to thank everyone who is watching live, as well as those listening to the podcast. For those listening to the podcast, if you’d like to read the full transcript and links to where you can follow Ben Taylor on LinkedIn, the short URL is just ex.pn/bentaylor. That’ll bring you over to the Experian blog where you can learn more about Ben, his work, his company. And I want to encourage everyone to follow him on LinkedIn. He’s leading a lot of great discussions there. He’s very, very active. Ben, thank you for all your work, all your insights and helping to build up the data science community.

Ben Taylor: Yeah. Thank you, Michael. Glad to connect.

Mike Delgado: Great. And for those who don’t know, we have a Data Talk every single week, where we bring on different data scientists and talk about all things machine learning, deep learning, data science. And I’m really grateful to have Dr. Ben Taylor with us today. We’ll see you all next week. Thanks again, Ben.

Ben Taylor: Yep. Thank you.

About Ben Taylor

Ben Taylor earned his Bachelor of Science degree in Chemical Engineering from the University of Nevada and his Masters of Science and is on leave from his PhD program in Chemical Engineering from the University of Utah. He is the Chief Data Officer and Co-Founder of ZIFF, Inc. and previously served as the Chief Data Scientist at HireVue.

Check out our upcoming live video big data chats.