Machine Learning for Content: Jose Moreno

Jose Moreno, Senior Software Architect at Netflix joins us for a conversation about how he became a software engineer and applies machine learning to solve problems in the entertainment business.

Topics of this episode include:

• how Jose ended up at Netflix, working on machine learning problems in the Content Algorithms team
• what were some of Jose’s formative computer science experiences and his advice for current students
• who was Jose’s favorite professor at Harvey Mudd College, his alma mater

For the full text transcript see below the fold:

Audio:

Play in new window || Download

Video:

Max: Welcome, all. Max of The Accidental Engineer here.

Today, we’re joined by Jose Moreno, Senior Software Architect at Netflix. Jose, do you mind sharing for our audience a little bit about what it is that you have done in the several years you’ve been at Netflix as a software engineer?

Jose: I’ve been on two teams in Netflix so far. I joined the Globalization Technical Infrastructure team (GTI) initially. And then I recently moved over to the Content Algorithms team about a year ago.

Max: For our audience that doesn’t know Jose, Jose had spent over six years at Microsoft out of college, before joining Netflix.

So going back to how you got into software engineering, a lot of our audience is curious about first jobs that people got out of college or otherwise. Do you mind sharing how you ended up at Microsoft?

Jose: Yeah, so after my junior year I decided that I wanted to do an internship. And it was actually a rough time because that was 2008, so it was pretty hard to get a job that year and I got a lot of rejection. So I interviewed with quite a few different places and nobody was really hiring that much.

It was pretty tough. And a Microsoft recruiter just randomly sent out an email to Harvey Mudd. At this point, they weren’t actually recruiting at Harvey Mudd very much, but they were like, “Hey, if you’re interested in a position at Microsoft, send us your resume.”

So, I sent my resume, had a phone interview. It was kind of a whirlwind. All of a sudden I’m flying to Seattle. I was already facing a lot of rejection, so I was like, “I’m probably gonna get a rejected again,” but went through the interview and everything went really well.

They decided to offer me an internship and I took it. The internship was over the summer. I think it was 10 weeks, and at the end of the month, at the end of the summer, I guess I impressed them enough for a full-time offer.

Max: And then you spent the next six-plus years?

Jose: I spent six-plus years there, yeah.

Max: A lot of our audience did not get computer science degrees in undergrad–how did you end up knowing that you wanted to do computer science?

Jose: So growing up, I always thought I was gonna be doing something with math. I didn’t grow up in a technical family, right. My parents emigrated from El Salvador. They knew nothing about computers. But so but I loved math, and so I thought I was gonna be a mathematician. I think when I was really young I used to call it “mathemagician.”

Max: Isn’t he a Harvey Mudd professor?

Jose: There is, yeah, and he’s amazing, he can square numbers that are six-digits long in his head. I have no idea how he does it.

Max: Is he who had attracted you to Harvey Mudd?

Jose: He is not, no. That was also an interesting story. I was focusing on math and then my junior of high school they decided to teach the first computer science course. The description said, “If you love math and you love computers then come check out this course.” And I was like, “Hmm” I like math. I like computers. I like playing “Wheel of Fortune” on computers anyways.

Might as well check it out. And like, after taking that class, the rest was kind of history I fell in love with it, started taking some community college courses that summer. And then they offered an AP computer science class my senior year. Took that and from there, I just knew that I wanted to do it.

Max: So a lot of the hurdles that I think people early on in software engineering–whether they’ve even gotten their first job or not yet–is an issue encountering a lot of people is having the confidence to do software engineering and programming.

Sounds like early on, having a high school course that showed you what it was like was a pretty positive, reinforcing experience. When you got to Harvey Mudd–I’m not joking about it being MIT of the west–was it really intimidating to take your first CS classes?

Jose: Oh, it was crazy intimidating. When I went to Harvey Mudd and I majored in computer science, I just assumed that everybody else has been doing it since they were like five years old. That everybody has just been coding their entire lives. And I’ve been doing it for maybe one or two years and courses that were relatively simple, so I was definitely intimidated.

Max: So what were those first intro CS classes like–were your fears validated?

Jose: My fears were not validated. I actually realized that everybody was kind of around the same level. There were definitely some people who had parents who were in computer science and they were way ahead of the curve initially. But they didn’t have all of the technical background that you learn in college or your courses.

So it definitely leveled out after the intro courses. And I also learned that although there were other people that were doing computer science majors–they had not taken classes in high school.

They wanted to do CS but they were still new to it. And so I was actually ahead of the curve just with a couple of high school classes. So yeah, my fears were definitely unfounded.

Max: A lot of our audience never got their computer science undergrad. What curriculum-wise do you think or have found the most valuable 10 years out of college from your undergraduate education?

Jose: Which class?

Max: Which class, which specific subjects, concepts?

Jose: Yeah, data structures and algorithms.

Microsoft actually weren’t recruiting at Harvey Mudd when I got the internship. But then they did recruiting. I guess I impressed them a little bit and they started focusing on Harvey Mudd. They would fly me back every single year and I would talk to a lot of the students who were interested in their internships or full-time jobs and to prepare them for the interviews, I always said, “Go back and read your data structures and algorithms book.”

Because that’s what interviews are really focused on, right. You started talking about linked lists. You talk about dictionaries and just a whole bunch of algorithms in terms of how to optimize certain problems, how to think about it different ways.

So I think that that class is the most important class. The rest of them were just like, “Yup, here are some other subjects to learn.”

Max: What were some of the other subjects that you learned separate of data structures and algorithms that were the most interesting to you?

Jose: So I actually fell in love with machine learning while I was at Harvey Mudd as well. I took some machine learning classes with Professor Christine Alvarado who’s at UC San Diego now.

So machine learning was an awesome course. We had a “clinic” that was a senior long project right, with the actual company who came in and hired a team. That was pretty valuable as well. Although I was a program manager for that team as opposed to a software engineer, that was interesting.

Max: So you were not coding?

Jose: I was not coding, yeah.

Max: That was in contrast to your internship the summer before?

Jose: Yeah. I was going in as a software engineer for Microsoft in the following year. So the second semester, I was like, “Mm-mm, maybe I should be coding for this.”

By the time that I was like, “Yeah, I wanna do a little coding for this project.” Everybody was like, “Well, we’re kind of done already.” We were actually really well-prepared. I guess I saw the job as a program manager because we were ahead of the curve, yeah.

Max: Where was it, that you guys were tasked with doing for your senior project?

Jose: So the company was called genius.com if I remember correctly. Basically, they send out marketing emails for different companies.

And they had an issue where, basically, if I would send you an email, they didn’t really know if you forwarded that email to somebody else.

And they had metrics around it right, so if you clicked on the email they had metrics around that. But if you forwarded it to somebody else and then they clicked on it, they wouldn’t know if it was you or a third-party that was clicking it.

So they wanted us to look into a potential solution for how to figure out if it’s a third-party clicking the email or if it’s you.

Max: Got it.

Jose: We actually took a machine learning route to it. And at the end of the day we found that the most useful thing was to just put in a GUID cookie. And the first time somebody clicks into it right, because then we’re going to assume you’re the first party.

If the second time we see that it doesn’t exist or it’s a different GUID then it’s a third-party. We did so much analysis out of this thing and it was basically just like, “Okay, we’ll just put in a unique identifier in there.”

Max: Fair enough. So you spent six and a half-ish years at Microsoft, after graduating Harvey Mudd?

Jose: Yeah.

Max: What all teams were you working on there? What all projects were you working on there? Was it exclusively as a software engineer?

Jose: Software engineer throughout my time there. I was on four different teams I think. I think I have to think.

Max: Is that super abnormal at Microsoft?

Jose: So most of them no. I think it’s mostly pretty normal because you go through some reorgs while you’re at a company that big. I started off on Windows Live profiles. And then was reorged into social update platform. Then reorged into Live ID, which is now Microsoft Accounts.

Max: So a lot of our audience probably doesn’t know the phrase “reorg” as a verb.

Jose: Oh, yeah.

Max: So I’ll break it down really quick–a reorg is a reorganization of a company’s hierarchy or focus team-wise. And that happens often, at large companies when managers get promoted or managers leave as teams get reallocated or product priorities or service priorities change.

Jose: Yeah. That’s exactly it.

Max: It’s not so shocking that in six-plus years at Microsoft you ended up on four different teams.

Jose: Yeah. And sometimes managers just shift from one team to another. They reorg themselves maybe a couple of skipped levels ahead of you.

And they decide to bring over some team members with them. That’s part of reorgs as well. That time, I ended up on Microsoft ID or Microsoft Accounts by the end of it.

Max: So which were the more interesting teams to work on if you don’t mind being opinionated?

Jose: I loved the first two teams for a couple of reasons. I thought the subjects were really interesting. I think my teammates on those two teams were world-class. I owe a lot of what I learned my first few years to a lot of the things that I learned to them. They built some high scale systems.

Max: What were these teams specifically?

Jose: Windows Live profiles. So the idea for Windows Live profiles was to centralize your profile information for your Windows Live ID. This wasn’t true in 2009, but it is true now.

If you log into a Windows 10 machine and move to another Windows 10 machine, all your personal configurations and data comes with you. There was a big shift inside of Microsoft to say, “Okay, we need to centralize this information.” Windows Live profile was the first team that did that.

The second team was Social Update Platform. This does not exist anymore, unfortunately.

Max: Sure. This is also not uncommon at all.

Jose: Yeah, exactly. But when Windows Phone first launched, they had a very interesting idea in terms of integrating all your social networks into one location so you don’t have to jump to different apps to see your LinkedIn feed, your Facebook feed and your Twitter Feed.

What we did on the team was we grabbed all your different feeds, combined it into one location and then on your phone, you just go to that one spot and you get your feed. You’re not gonna jump around from app to app. I found that super interesting. And one of the things that I ended up owning there–probably my first time really owning a big feature–was being able to “pin.” Let’s say I’m interested in Max right, I could pin it in my phone.

And anytime you updated something, I would get a lifestyle notification on my phone, “Max: the status has updated,” or, “Oh, check it out, there’s a new podcast video.”

I would get it right on my home screen. And I don’t have look into seeing whether you put a new video up.

Max: So after your time at Microsoft, I know one of the big reasons that you decided to move to Netflix is that Seattle and where Microsoft is headquartered has a very specific climate.

Jose: Yeah.

Max: And so quality of life is a really important thing in life, right? :)

Jose: Yeah.

Max: Do you mind speaking to how you came to the decision to move on from Microsoft and that Netflix was the place to go to?

Jose: Absolutely. So I grew up in LA. I went to school in LA and then moved out to Seattle. When I grew up in the valley San Fernando Valley in Los Angeles the temperatures would get up to 110 in the summer.

And I played football in high school. So I would be in full pads. I would be dying in that heat, just dying. So when I first moved out to Seattle, I was like, “Okay.” I was super welcoming of the climate. 50, 60’s is just perfect, it’s amazing. But one of the things that you end up learning pretty quickly is that the sun does not come out in Seattle very often.

It’s not the rain. People mistake it for the rain. The rain is okay. It’s very light. So it’s not really that big of a problem but the sun not coming out for weeks at a time–for months at a time–that started getting to me.

You hear about seasonal affective disorder. Two years before I moved out of Seattle, I legitimately felt depressed for the first time in my life. And Carol, who’s my significant other, she’s like, “Oh, why don’t you try taking vitamin D pills?” And I was like, “Okay.” So I started taking them–a couple weeks later I’m dancing around and I was like, “Oh my God, this thing has legitimately got to me.” So I wanted to get back into a sunny place.

Max: Nice, in Los Gatos, California?

Jose: Yeah.

Max: Pretty sunny place?

Jose: It’s really nice out here, yeah. And it’s not too hot, so it still works itself out.

Max: You mentioned that in undergrad, machine learning was a topic you’re interested in and it sounds like it was only once you got to Netflix that you professionally got placed onto projects that used machine learning.

Jose: Yeah.

Max: I know we can’t really get too specific about what Netflix is doing with machine learning but our audience I think would be really stoked to hear about:

• specific tools that you guys are using
• whether you are particularly interested in classification or regression problems
• what different kinds of data sets are interesting to you personally?

Do you mind sharing a little bit about that?

Jose: Absolutely, so in terms of tools, we’ve used or I’ve used personally two big ones. One is TensorFlow, which seems to be pretty standard across the industry these days. And actually one that’s not heard of as much is DeepLearning4J.

Most of the code that’s written at Netflix is in Java. I found DeepLearning4J, which allows me to do machine learning models in Java, so I don’t have to worry about integrating models that were created in Python into our Java-based applications.

I was able to create them in Java and just easily adapt it. So I’ve decided to use DeepLearning4J and so far it’s been pretty nice in terms of allowing me to do a lot of what TensorFlow allows me to do, building models and using them in our Java services.

Max: Got it. Is much of the work that you do “exploratory” or is it more building up services that are higher level type of stuff?

Jose: So I have a unique position in that I do both. I think it works out really well for me because when you’re creating machine learning models it’s very exploratory, right? It may or may not work, and there have been times where it’s like a month of me looking into something trying to create a model. And it’s just not working. So you analyze the data, and you look at the data and it can be very dirty.

So after a month, maybe six weeks, you have nothing to show for what you’ve done.

Max: Dude, that’s brutal.

Jose: Yeah. And it completely happened earlier this year for me. So after the six weeks of that–of nothingness–I talked to my manager and I was like, “Maybe I should work on one of our services right now and actually code up something that I have,” and I have something to show for.

So I’m able to switch between creating services and some of the infrastructure that we use, and then going back into the exploratory data things.

Max: One of the guests we’ve had on previously–Charles Martin, a data science consultant–swears by Jupyter or IPython notebooks.

Jose: Oh, my God. I wanna start using Jupyter. I haven’t. I’ve seen it. I haven’t gotten the chance to do it yet. But I’m super excited to use those notebooks. They’re amazing.

Max: Yeah. They make exploratory analysis really easy and obviously reproducibility is one of their big things.

Jose: And you’re able to keep notes as you’re working through. And just have the history of how you’re progressing toward the end result. I honestly think… So I’ve seen them. I’ve used them a little bit but not really in my job. That’s definitely something that I will be using moving forward.

Max: Sweet. Yeah. This obviously isn’t a sales video for Jupyter or to sell open source software!

Jose: Yeah, definitely it’s not, and I wasn’t planning on talking about Jupyter notebooks!

Max: I guess the reason they rebranded from being called IPython Notebooks to being called “Jupyter” Notebooks is because the concept of a web UI, where you edit and run snippets of code and interspersed with documentation was originally done in communication with a Python backend.

Jupyter was a generalization of message-passing to a backend that wasn’t necessarily Python. So there are hypothetically Java backends now, where you can make a Jupyter Notebook and write your Java code that invokes Java functions. And instead of your Web UI communicating with a Python kernel they communicate with a JVM kernel of some kind. So this may be sooner down the pipe for you than…

Jose: I’ve only seen it in Python but yeah, I’ll definitely try that next time I’m creating Tensorflow models. I haven’t heard of the JVM integration part of it. That’s something I’ll look into.

Max: Yeah. There’s different kernels—there’s Julia language, there’s Ruby, there’s Haskell, up to wazoo.

Jose: Yeah, yeah.

Max: One of the other guests we’ve had on also has a very similar background to you. I think he spent five years at Microsoft before moving to San Francisco to join a smaller company than Netflix called TuneIn. Alex Andresen made a similar job switch from Seattle to the [San Francisco] bay area. When we chatted, we talked a little bit about the crossover from C# to a Java shop or how translatable a career at Microsoft is to a career in the Bay Area. Can you speak a little to how the learning curve was for you, coming from 6+ years at Microsoft to Netflix?

Jose: Yeah. So I did a little bit of Java my senior year of high school, and then maybe one other class in college but definitely spent six and a half, seven years without doing Java.

I found the jump from C# to Java to be relatively simple. They’re very similar languages especially around syntax. There are some things that are pretty different in terms of using Spring or Guice for dependency injection, which is what we do at Netflix. But aside from that, it was a pretty easy switch.

I think the biggest difference is that a lot of the tools that you use in C# are .NET-created, so created by Microsoft specifically. Whereas in Java you have so many more open source libraries available to you. It’s sometimes a little confusing when there are 50 String Util classes–like, “Okay, which one do I use for this?” But aside from that, I thought the jump was relatively simple.

Max: What were some of the on-boarding processes that Netflix had in place for people like yourself who were not coming from a Java background?

Jose: There generally is not a whole lot. Looking at somebody’s Java background and figuring out how well they’ll do in Java is part of the interviewing process at Netflix.

They want Java skills, and when we’re hiring somebody–or at least the teams that I’ve worked on–when we’re looking at somebody that doesn’t have a Java background then we try to figure out how easily it will be for them to jump over to Java. And if it’s not an easy transition then that’s enough for them to not get the job at Netflix.

Max: Is this pretty generally applicable to machine learning roles at Netflix–that Java is an important pre-requisite?

Jose: No, for machine learning it really depends on the job that you’re working on. I don’t think Java is necessarily a requirement if that’s the position that you’re interviewing for at Netflix.

Max: But for many software engineering roles?

Jose: Yeah, for software engineering. Generally, Netflix uses Java, so we look for Java skills.

Max: Got it.

On that topic, I think we should take a moment to plug jobs at Netflix.

Jose: Yup.

Max: One of the things that Jose told me to make the audience aware of is that Netflix primarily hires experienced hires, as opposed to recent grads.

Jose: Zero recent grads and no interns. So yeah it’s not a place to apply to coming out of college.

Max: However, for those who are interested and qualified or are curious if you’re qualified, you guys can check out software engineering roles, machine learning roles at jobs.netflix.com.

Jose: Yup. And read the culture manifesto that we have on the site! It really talks about the culture at Netflix, and gives you a better understanding of what you would expect there, also very important for Netflix employees.

Max: Oh, for sure, I’m glad you mentioned it.

Jose: Yeah, absolutely.

Max: Dude, Jose, it’s been a pleasure! Thanks for joining us.

Jose: Thank you for having me.