Wondering what the job market looks like after a PhD program? What it takes to get a data scientist role coming from academia?

Our close friend Jeremy Karp just obtained his PhD and will soon start a new role as a Data Scientist at Lyft.

Topics of our conversation include:

  • what pros and cons Jeremy weighed in considering a PhD program
  • how Jeremy prepared for his most recent job interviews
  • which interview formats and questions are top Data Scientist employers utilizing to evaluate candidates

For the full text transcript see below the fold:

Jeremy Karp, Data Scientist

Audio:

Play in new window || Download

Video:

Max: Welcome all!

Max of the Accidental Engineer here–today we are joined by Jeremy Karp. Jeremy will soon be joining Lyft as a Data Scientist.

Jeremy, do you mind introducing yourself a little bit and what your most recent background looks like?

Jeremy: Sure. So I’m Jeremy, your most recent guest! I just finished a PhD at Carnegie Mellon, where I was in the Algorithms, Combinatorics and Optimization program. There I worked in interdisciplinary research ranging from theoretical algorithms to applied optimization in machine learning and the connections between those fields. That was the last five years or so. Yeah, so, I spent five years getting my PhD there.

Max: I think you’re a super awesome guest to have on the show for the reason that you’re definitely an “accidental engineer” in the way that your undergraduate degree was not in computer science.

Your first job out of college was not as a programmer or software engineer or data scientist–which might not have existed at the time, as a job title back in 2010-ish.

Jeremy: Yeah.

Max: Do you mind sharing for our audience how you found your way into the PhD program? How you made the call that you wanted to go back into grad school?

Jeremy: Totally, yeah. So, I majored in economics in college and I graduated in 2010 and took a job as an economic consulting analyst at a company called Cornerstone Research. I worked there for two years before going back to get my PhD, and I guess I had some intention of getting a PhD starting at around junior year of college. But I had also been doing internships and setting myself up for getting a consulting job and I wanted to take a break and try that out.

Max: What was your undergraduate major again?

Jeremy: In economics and I minored in math, so I was one class away from a math major and that was kind of…as I started taking economics classes, I realized it was the more theoretical mathematical modeling side of the field that I was interested in. So I started taking math and CS classes too, and continued a little bit after graduation as well to prepare for grad school.

Max: I think we’ve had one or two PhD folks on the show so far, and all of them have mentioned a little bit about what their grad programs prepared them for in the job market—whether they did internships throughout their grad program or not, and ultimately how their grad degree help them in the job market.

So, my question to you is, did you do any internships through your PhD? At what point were you more or less sure about whether you’d stick with academia or go into private industry?

Jeremy: Yeah, totally. My program is an interdisciplinary one so it’s joint with operations research, math, and computer science. And then in terms of research–as long as we found an advisor who was supportive of that direction—we were able to do research in anything in between any of those worlds.

So it was a lot of opportunity; you also have to know what you’re doing somewhat to find the right project.

Max: Is this something that’s unique to Carnegie Mellon?

Jeremy: It’s…so, the same program, more or less, exists at Georgia Tech. I think someone who graduated from the CMU program became faculty there and thought it was a good idea and started that program. I think University of Waterloo also has a similar program.

A lot of people are aware that there are connections between these fields and so there are a number of ways…I think Cal Tech has a program with a different name but it’s a similar program between applied math and computer science.

At many universities you’d have to pick a department and be Operations Research or Computer Science or Applied Math, and then wade out on your own into other departments.

Max: So your PhD advisor was okay with you not being necessarily committed to going into professorship after your PhD?

Jeremy: I think that any professor, their first choice for their students would be to become an academic but my advisor, in particular, he does a lot of applied work as well as very theoretical work and he likes having a network of people in industries.

I think I got some nudges at the end of [my program]: “Let’s stay in touch. Maybe you can share some datasets with my future students.”

I’ll have to see how that works out with my future employers, but I mean I’m also…that something I’ll remain interested in, is research and contributing back.

Ultimately they’re okay with it–between a third and half of the students in my program end up going into industry and then half to two-thirds end up staying in academia is roughly how it shakes out.

Max: As someone who’d never held a job in a technical engineering, programming, coding: how did you get your first technical job after joining the PhD program? I know you did some internships.

Jeremy: Yeah, that’s right. I think it was after my third year and fourth year that I interned at Amazon in Seattle and I guess networking is always a good thing. I wasn’t even particularly ambitious about this but an alum from my program who works at Amazon came back to give a technical talk.

He was clearly there with the intention to recruit people and I talked to him afterwards, and I guess my program is quite small so a perk of that is when you meet someone who’s an alum from your very specific program there’s instantly a strong connection there.

The downside is that your alumni network in aggregate is smaller than others. But he clearly was interested in talking to me as opposed to people in other different programs, and so we sat down and chatted for 20 or 30 minutes, and then the next week I got an email from an Amazon HR person saying, “Would you like to submit your resume?”

At the time I was open to industry-type work but I think I was still leaning more towards academia but I figured, you know, there’s no harm in interviewing for this position. So I interviewed, I got the job and then once I had it, it seemed like a good opportunity that I shouldn’t turn down and I actually had a really great first summer there working in the supply chain research group at Amazon Seattle.

That both inspired me to come back the next summer and continue that work there and also made me realize that there are great opportunities for people like me in industrial positions and ultimately, I’m going to work at Lyft. But I saw what I think is a similar role at Lyft that I held at Amazon and that experience both filled in gaps in terms of skills that I wouldn’t have learned in the PhD program alone and also got me more interest in those types of jobs.

Max: In contrast to some of your classmates in PhD programs at Carnegie Mellon and that you’ve probably met elsewhere at other universities, did you feel like you were advantaged in your job search for an internship and then a full-time role afterwards due to the fact that you had held a full-time job in between undergrad and starting your PhD?

Jeremy: I think people liked it but ultimately…I think the school helps. Carnegie Mellon has a great reputation and I was lucky to get to go there. But I felt the job market sort of comes to you when you’re in a program like that.

Ultimately employers do care that you aren’t just someone with your head in the clouds and so they want to see either that you worked on some more applied-type projects throughout your research there or you have a background as some sort of analyst or engineer or other ways of demonstrating that.

I think 85-90% of it is that if you’re in a strong grad program–that takes a lot of the burden off of the person doing the job search. I felt really grateful to be in a really strong position there.

Max: We can’t talk about some of the more specific things that you worked on in your internships and that you will be working on at Lyft. But for people who are currently in PhD programs like the one that you were in, there’s operations research, computer science, machine learning.

How much alignment was there between what you’d done in your first three years of your PhD before starting your first summer internship with Amazon and the work you were doing in that internship?

Jeremy: I think there was a lot of alignment in the sense that I don’t think I would have been able to do that job successfully without having at least done the masters curriculum of coursework that I’d done. And I guess probably the coursework more than my specific research was useful.

I think machine learning is becoming useful in all areas. Even if the role is more traditional operations research there are opportunities to incorporate machine learning into that. My machine learning background before that internship was somewhat…I guess I’d taken the graduate machine learning class where you implement the basic algorithms and do a project and I’d done a couple personal projects that were related to applying machine learning to data since I was interested in it.

That was enough to get my foot in the door there.

In Amazon specifically, they care a lot about optimization and use a lot of fancy optimization models–that’s something that I was really well-positioned to do.

You might have to beef up on linear programming, integer programming but that’s something that was very thoroughly covered in my PhD program. So that was a natural fit for what I wanted to be doing.

Max: Are there any problems in applied math or otherwise that you’re particularly excited about working on over the next 10 years?

Jeremy: Yeah. If I were to stay in academia and build a research program and also connected to what I saw at Amazon, is that in operations research typically you construct some sort of model.

For example, the famous problem is the diet model where you want to consume a certain number of calories and have that break down to fat, protein, and carbs that’s balanced. And you also want to minimize your costs. So you have a portfolio of foods you can buy at the grocery store and you want to select the portfolio that meets your dietary restrictions and also is the cheapest.

This is an abstracted model and there are all kinds of models like that in operations research where, if you know the full information, then you can enter it into a linear program or something like that and solve it exactly.

But when you’re dealing with a company like Amazon or Lyft or Google or whatever, oftentimes what you have is virtually infinite amounts of data. And so the question becomes, “How do you translate, how do you extract from data the exact coefficients that go into these models?”

What I’m interested in specifically is: are there end-to-end frameworks of taking data and converting that into discrete decisions that affect future decisions? So, you know, “intent models of discrete sequential decision-making.”

This is something that I saw at one of the projects I worked on at Amazon, which involved truck scheduling and they were basing this off of heaps of data, but ultimately trying to solve an optimization model.

This is something that I think appears widely and was part of the research I did in my PhD, part of the work I did with Amazon, and I think it’s still largely an unsolved problem. Every company probably has some approximate or heuristic way of dealing with it.

But something more fundamentally driven but also works in practice would be the “holy grail” and for now I’ll be approaching that from the more applied side of things. But I think it’s an unsolved problem that greatly affects all kinds of businesses.

Max: I know I’ve thrown some shade at PhD programs in subtle ways in previous interviews that I’ve done.

From a career and skills development perspective I often get people who are curious about “What programming language do you learn first?”

My answer for those people is to get a job, find out what they’re using and learn it because that’ll be the best feedback mechanism above all else, potentially.

However, having a recently furnished PhD in hand, I want to hear from you and our audience probably also wants to hear: what were the pros and cons of doing a five-year PhD program as opposed to trying to find a job in machine learning or data science straight after deciding that management consulting was not the ultimate path that you wanted to go down?

Jeremy: Sure. I think for me the PhD was very much a positive decision but it’s probably also not for everybody. You know, it’s a very personal thing.

And especially towards the very end I realized how much of a strong, deep personal endeavor it was for me that I feel very happy to have been able to do, but not everyone feels the same way.

In terms of career progression and things like that, I wouldn’t say I had this premeditated plan of I’m going to enter a PhD and then five years later I’ll be a data scientist (which is a job that barely existed when I entered at Lyft which is a company that didn’t exist when I entered!).

But I saw my options as I was working at this economic consulting firm–they’re a very good firm, a positive experience if you’re an undergrad in economics, I would still say that Cornerstone Research is a great place to work.

But when I had looked at my options, it was essentially:

  • try and continue to progress at the firm,
  • go to business school,
  • go get a PhD,
  • go to law school or
  • try and laterally pivot and move into some sort of product role at some company or analyst role.

I think my assessment at the time was pretty accurate as most of the people I joined…a few of them have become software engineers or are data scientist now through slowly moving through a sequence of two or three jobs that pushed them in that direction.

A couple people are still with the firm and are associates or higher up.

There are people who have done MBAs and now are in the typical post-MBA type jobs.

And there are some people in PhD programs, and I guess for me when I saw my own skills, my own interests, I thought I had always been interested in academia and research and theoretical math and things like that.

I said, “I’ll probably always have some regrets if I never try and go down that path.” I also saw that when you look at the PhD programs I was interested and the recent graduates, the ones who were in industry: they were doing things in tech and finance and government that all sounded more exciting to me than the types of exits that I had from the other paths.

it made sense to me from that perspective, and a good PhD program pays for you to go there. So even if you dropped out with a master’s degree, you get a free technical master’s degree or paid for one and that seems like a good deal if that’s the right degree for you.

That was more or less my experience, so I feel very good about it.

Max: There’s a couple other upsides that I consider being really compelling things about choosing to do a PhD program and that I think were tremendously beneficial for your career and for anyone doing a PhD:

One is that hen you’re holding down a full-time job or, God forbid unemployed, you’re operating under a lot of duress so you aren’t able to fully explore the subjects and topics that you really want to explore. Even when you’re doing it as a full-time employee somewhere, it’s often hard to align that with your full-time job. Often when you want to change careers or change industries.

The second thing that I want to plug about a pro–a huge pro–to doing a PhD and true any grad program is that–like you alluded to earlier–employers come to you when you have a natural ending to a graduate program. So universities and grad programs, in particular, are a watering hole for employers to go to and they know that they will find people who are interested in entering industry.

Jeremy: Absolutely.

Max: So those are two additional huge upsides that I should mention and retract some of the shade I’ve thrown at PhD programs.

Jeremy: Yeah, it’s a long time to spend in school. I think people sometimes attribute that this person has a PhD from wherever so they know everything about “X”, but really what you know is that anyone can learn anything–what you know is what you spent that time studying.

So a PhD gives you great freedom to learn about pretty much anything that you want to. At least within the reins that your advisor gives you. Both from courses and research I learned a ton of things.

And also, I took some times where for two weeks I’d spend two-thirds of my time reading about something that I thought was interesting. Maybe it was to apply it to research, sometimes it was just because I was interested in. And you’re getting paid the cost of living expenses while you’re doing that.

For me that was a great opportunity. I’m sure there are some whiz kids somewhere who know everything I know and are dropping out of college to take my job.

But for me, personally, I know a lot of skills and deep knowledge about topics that there’s just no way I would have that knowledge if I didn’t have the opportunity to do a PhD. That meant a lot to me and I value it a lot.

Max: One of the particularly interesting things about your background and doing a PhD is being at Carnegie Mellon, being in Pittsburgh: ground-zero for a lot of robotics research, self-driving car research in your tenure as a PhD student over there. Uber virtually bought the robotics department at Carnegie Mellon, correct me if I’m wrong here.

Jeremy: That’s kind of true, I think it was a little bit oversold…there’s a separate institute NREC that is affiliated with the university but is not on the campus. I think a lot of the people Uber hired were research engineers—many of whom had PhDs or Master’s degrees who were working at this institute.

That said, I think it certainly did disrupt a lot of projects going on at this institute. And they approached a couple of faculty members and many Master’s students and I think some PhD students dropped out to go work for Uber. So, you know, on one hand, it probably makes it more attractive for grad students because it shows that if you go to Carnegie Mellon–especially, if you do robotics–there’s great opportunities. I’m not sure that the PIs running research labs were as thrilled about it, so it’s kind of a mixed bag.

Max: I mean, enthusiasm about this type of private industry involvement in academia is something I’m really unfamiliar with. Sounds like there’s mixed feelings.

Of your classmates in your PhD program who are sticking with academia, is the dominant form of funding for their research still coming from the US federal government in the form of grants? Or do academics in the area you were doing your PhD finding other funding sources for their research?

Jeremy: It’s a mix, I think probably NSF grants is the most common. There’s also military grants and things like that. Department of Defense grants, I know a lot of individual students who have Google or Facebook fellowships. Maybe not a lot but a few who got those fellowships, but I think NSF is the number one source.

In terms of the faculty I worked with, my program does skew more theoretical so I think that probably means that more of the funding comes from more fundamental research sources like the government.

But big tech companies like Google, Microsoft, Facebook, are all very serious about influencing the direction of academic research mostly for better but that’s definitely something that’s in play. For sure.

Max: So most recently you found this job with Lyft and you went through the job search and some ways it was…I’d imagine it would be pretty awesome being a grad student while doing that search.

Although I understand you were also finishing your dissertation and having to prepare to defend it, so that was probably stressful.

For our audience that’s curious about job searching as a PhD student, what are the recommendations you have for them about what employers are the most earnest about hiring? What’s the interview format and timeline look like for different employers, if you don’t mind being specific?

Jeremy: Yeah. I might not want to be specific about the employers because you sign a lot of NDAs with these things and I don’t know exactly what’s covered.

I was pretty focused on my interviews because I had a return offer to go to Amazon, so my thinking was to only apply to places that at least would have a serious chance of being more compelling than that offer.

And so, you know, if I was just applying broadly, I probably would have cast a wider net. But I got interviews through a mix of…I sent some cold applications, I also got some recruiters directly contacting me when they’re monitoring the top schools and they knew I was…I’d been there for, you know, for years and so…

Max: The watering hole affect.

Jeremy: Right. So people start reaching out, so I got a few interviews that way. And then also I talked to friends, people from grad school, people I knew from all walks of life who are involved in engineering and technology and got some referrals that way too.

All of those were strong pathways for getting interviews.

In terms of the structure and content, most places have a phone screen and then nowadays it seems to be quite popular to send take-home challenges which, as a PhD student, I guess is a little less inconvenient probably then if you’re a full-time person because I had the flexibility to take a day off and spend it doing some take-home machine learning challenge.

I could see that if you had a full-time job it would be a little harder to find the time.

Some of them were kind of interesting but it does take up at least a day for each company you’d want to do that for.

Then after you submit the take-home challenge, then after that usually came a flyout and I’d go to the office and do a full-day of interviews. There’s some variety but usually there it’s four to six 30-60 minute interviews plus a lunch.

For the positions I was applying for, I was a little surprised the interviews were 90 to 100% technical.

The least technical part of it was asking me to explain my thesis research and the rest of it was very, you know: here’s the scenario, here’s a problem, here’s a model, here’s an equation, code this thing.

And so, you know, I guess there’s a specific skill of doing these technical interviews that…there are quite similar to the “Cracking the Coding Interview” type data structures and algorithms interviews but more about statistics in machine learning and optimization for the positions I was interested in.

Max: Preparation-wise, was there much preparation that you did before your first interview and did it help very much?

Jeremy: So, I probably could have done more preparation. The first interview I did not prepare very much for and then I didn’t do so great on it. And then I realized that I should probably study up a little bit more for the remaining ones.

I think it’s, in general, it’s hard to…there’s such a wide variety of things they could ask you about.

The impression I got at most of the on-sites I did was that they’re looking for you to really excel at the areas that you are good at and then to be okay at the remaining things.

I don’t think they expect people to be an expert at every topic because nobody is. I guess that before the interviews that I was more serious about, I tried to think about what they would care about. I’d Google the people that I thought I might be likely to interview with and from their backgrounds, guess what they might ask me about.

Around that time, I went pretty deep into this book “Elements of Statistical Learning,” which is a really great theoretical statistical learning book.

Something that’s become clear to me over time is, if you start diving deep into the next level of a technical subject it makes everything at the lower level of the pyramid become intuitive and easy.

So, like, once you’re learning real analysis and higher-level math like that, something like calculus or linear algebra starts to become intuitive and you can connect the dots yourself on a concept.

I thought diving deeper into statistical learning and things like that would make it so that a more pedestrian machine learning question would be easier to wrap my head around or at least directionally know what the right way to go with that is.

Whether that was the best strategy or not, I don’t really know. I think there isn’t really a book like “Cracking the Coding Interview” for data science interviews yet and they all were kind of different, so probably the best preparation is a wide range of experiences and then getting a good night’s sleep before the interviews.

Max: Besides prep beforehand, I think a lot of people have a suspicion that there is a very finite number of employers out there for jobs that they would be qualified for, and that if they do that first interview with Lyft, for example, or Amazon or Google. That once they bomb that first interview, that they won’t be able to get a job at any of those places anymore.

So I’m wondering if you can speak to our audience who have that fear about how the job market is not that finite and that you should by-all-means interview for the jobs and prepare for them, but also that there are many employers out there. Did you find that to be the case?

Jeremy: Well, I think it depends. I guess it depends how specific and targeted you are. I actually, I tend to agree with that for my own personal situation.

I knew that I really wanted to be in a job where I felt like the specific area of expertise I have in terms of this discrete sequential stochastic decision-making–I could see ways where I can directly apply that to those questions. I felt like I did have a somewhat small group of companies that were sort of the “first tier” of what I’d want to target and so in that sense the stakes were high.

But would it have been the end of the world if it hadn’t worked out? If you’re willing to be flexible, there’s a huge data science industry in San Francisco right now.

So I guess you do have to be strategic about, I suppose you’d want to think to yourself, “what is the niche that I’m most interested in and then how specific is that and how important is it to you to work in that specific area?”

If it is very important it could be wise to think carefully about the order that you’re applying to jobs. It’s probably not very ethical to apply to jobs you have no interest in but you might not want to interview first at the job that is your number one priority. Maybe interview at your number three priority first or something like that.

No matter what your circumstances, doing 20 on-sites is a whole lot, and so you might want to think about knowing what you know of yourself, of your performance under stress, what you think is the uncertainty in the job market.

The right way to order that is a good operations research paper, so one to think about carefully.

Max: Wow, a lot of stochasticity there ;)

Well, Jeremy, this has been freaking awesome. I’m sure we’ll have an opportunity to do this again a little ways down the road once you’ve had some time at Lyft, and while we might not be able to talk about the specifics of what you’re working on once you start your new job, I hope we have another chance to talk soon!

Jeremy: Absolutely, thanks for having me!