- how engineering and product managers effectively leverage the crowd to get their software built
- how to figure out what third party code to use
- why people don’t care about test-driven development (TDD)
For the full text transcript,
Max: Welcome guys. Max of the Accidental Engineer here. Today we’re joined by Anand Kulkarni, and for those of you guys who do not know who this is, do you mind introducing yourself real quick?
Anand: Sure, so two-time entrepreneur. Originally did a graduate program at Berkeley working on mathematics, computer science. Left in my last of year of a PhD, took a leave of absence to start a company called LeadGenius. Ran that for six years, now I’m working at a company called Crowdbotics, which I’ve been doing for the last six months or so.
Max: So I know crowdsourcing has been a very big portion of both the PhD program you were doing, of your last two startups, and the one that you’re on right now. Excuse me, I mis-phrased that. For the people who don’t know exactly what we mean when we talk about “crowdsourcing” do you mind giving a general, broad definition?
Anand: Yeah absolutely. So broadly speaking, crowdsourcing, or crowd computing as we like to call it if we’re being fancy is this question of how we can get strangers over the internet to our work for us in a way that’s predictable, reliable, consistent. Current enthusiasm in crowdsourcing came about in 2005, when Amazon released the Mechanical Turk platform, which let you write computer programs and make API calls to human beings.
The last 12 years have seen people coming up with new applications for these kinds of interactions. Places where you can stick a human being into a computational cycle to do something that machines couldn’t do before. In the early days a lot of this was spent on things like, “How do we annotate big data sets to help train machine learning systems?” Today we’re looking at really complicated, advanced coordination systems that combine big groups of people to do all kinds of unexpected things.
Some of those are real world things like Uber or Lyft, and some of them are totally virtual things. Like what you might find on Upwork.com.
Max: In your last startup LeadGenius there’s a type of crowd computing that was being done. Do you mind sharing how LeadGenius performed crowd computing?
Anand: Sure, so Lead Genius is a company that attempts to…automate parts of the top of the funnel sales process. Companies have this ongoing need to try and find effective information, accurate information about potential customers they’re gonna sell to. And historically the way this was done was by buying big lists of data from people, and then having your staff go through and manually cold-call them. So I got a big sales team, which could get pretty expensive. LeadGenius had the innovation that we could scrape big parts of the web, public data sources like government records, combine all this information together using some form of fancy editing resolution algorithms, what we called them. And then use human beings to go and clean up the data. So a lot of people would try to figure out ways to collect information automatically about companies and people in the past.
And it kinda worked, but to get really high-quality information, to get the best data, you have to use humans. So that was our application, we used human computation or crowd computing to try and clean up all the information as we collected it. And resulted in us being able to provide information that other people could not.
Max: So getting into the nitty-gritty of what a member of the crowd in that scenario was doing, it involved manually scraping the internet? Like going to different websites and copy-pasting data, or using tools to scrape the data like you just described? What is the nature of those kinds of manual tasks that historically have been manual and are becoming more automated or becoming more like API calls like you described?
Anand: Sure. So a typical example might be when there’s some change in information out there in real world, send a person to go and verify the information is correct. So let’s say you changed jobs, okay? First thing that happens is of course you fill out your employment paperwork, file a bunch of paperwork with the government. We can’t touch any of that data because it’s all private. But at some point you’re gonna update your LinkedIn profile. And you’re also gonna update the information on the company’s website to say, “Okay, here is this person on this company’s team page.”
When that information changes, it’s useful for us if we’re trying to sell something to your company to know that that change happened. We might invoke a person to go out and check the information on that company’s website. We might ask somebody to go and look at your Twitter profile, too see if you’ve changed your state of affiliation. And those are the kind of tasks that you can automate, or you can have a person go out and grab the information and check. Lead Genius was able to do that very large scale, across just about every company in the US was our goal.
Max: What’s the state of the market look like in terms of what are the most dominant businesses in the marketplace of offering crowdsourced or crowd computing as a service?
Anand: Yeah, it’s a good question. Back in 2005 there was one game in town, which was Amazon Mechanical Turk. But it kinda depends on if you take this sort of broad view of what constitutes crowd work, or if you take a really narrow view. So if you’re talking about this narrow view of doing really API tasks, API-level tasks that take a few seconds. Okay that’s Amazon Mechanical Turk, systems like CrowdFlower.com, and internationally you’ve got a few other ones. All of which allow you the ability to write programs that invoke a human being. But I think it’s more interesting to take a look at sort of the broader spectrum of who’s doing crowd work, right? What are crowds, but basically people that are on the internet being invoked by software to go and do things, right?
You know in that sense, Uber and Lyft are two gigantic crowd services providers, right? You’ve also got TaskRabbit, which is you know another one of these physical and real-world activities people do. You’ve got freelancer sites Freelancer.com or Upwork.com, which certainly allow you to have software-driven access to lots of people. And you’ve also got some interesting new startups in this space, right? Trying to productize certain slices of crowd work. So Scale.API is one that’s really interesting…
Max: Before we get too far, I wanna ask you about what it is you’re up to now. Because I know the business you most recently started, is in exactly this space, and with a very interesting twist.
Anand: Yeah. So the new company is called Crowdbotics. Crowdbotics is trying to automate parts of the software development life-cycle.
Max: For people who don’t know what the software development life-cycle is, maybe can we unpack what that means again?
Anand: Yeah, sure. The goal of software development is to turn specs into code, right? At a very, very high hand-waving level, right? We wanna have some description of some software that we want on one side, and want working software to come out on the other side. That’s a hard process, and of course there’s more to product development than that. You have to make sure the product actually meets somebody’s need, you have to make sure that the software actually works. But really what we want is efficient ways for us to convert descriptions of products into software that we all can use.
Historically the way we that we’ve done that has been to have teams of engineers who sit together in an office, they follow some software methodology that they like, like Waterfall or Agile. And…you know despite the best efforts of a lot of engineers, and a lot of teams thinking over a lot of time, it’s still very hard to build good software.
Max: I guess what’s the state of things today versus the state of things post-Crowdbotics? You were describing the current software development life-cycle of salaried employees who communicate with stakeholders in their company and get specifications about what’s the software they’re getting paid to build should do. And then they’re going away and just banging away on their keyboards and making it. So in a post-Crowdbotics presence in the marketplace, how does the software development life-cycle change?
Anand: What we’d like to do is be able to provide scalable on-demand software engineering talent to try to support software engineering teams in the process of building software. Ideally what that means is we’d like to be able to have a system where you give us specs and software comes out the other end. Behind the scenes we have a combination of code libraries and people. And the people are contract software engineers who we’ve screened and vetted. And, also a whole bunch of scraped software that we’ve pulled out of GitHub, Stack Overflow, that we’ve got some special sauce to snap together in efficient ways. We’d like to see a universe where a lot more people are able to build software by using systems like Crowdbotics. Where we do a lot of the heavy-lifting, and then use human beings to snap together the last few pieces.
Max: One of the developments in software development life-cycles over the last 50 years has been the introduction of test-driven development. And there’s this zealous minority within the software engineering community that believes that, or argues that your software doesn’t do what it says you do until you have some automated tests that verify that the behaviors you say it does, actually does it. That’s something that’s common in software development life-cycles in very high-risk enterprises. If you’re making a pacemaker, let’s say, or you’re making software around an airplane, having those tests in place is really valuable.
But, are you finding with Crowdbotics and the type of customers that you’re seeing, is this kind of radical change in software development life-cycles something that’s amenable to high-risk enterprises? Where you’re maybe building software for airplanes? I guess how do you prove that Crowdbotics does what it says it does?
Anand: I certainly don’t think that any approach you’re taking should be incompatible with test-driven development. Especially if you’re making big claims about being able to build software faster or cheaper, you definitely need tests all the more. In fact, when we first started Crowdbotics, I really thought the killer use case that people who were gonna find the most popular was going to be writing test cases to try to prove their software was correct. So far nobody has ever used Crowdbotics for that purpose.
I think despite the fact that we recognized the importance of correct code, and we really want provably correct code. Or at least if not provably correct, then…let’s say well-tested and well covered code being important. I think that the reality is that a lot of people that wanna build software are really concerned with time to market, and getting stuff out sab that’s good enough. And so, that’s really the place that we’re seeing people use Crowdbotics to get the most value.
And I encourage people to always budget time for test cases…
Max: For sure.
Anand: Writing test cases, things like that. But, really I think it’s engineers who’ve been burned a few times, who start asking for that. It’s less often that product owners really drive that discussion. And at least for Crowdbotics we see most of our users are actually coming from the product side as opposed to the engineering side.
Max: So I guess what are kinda the demographics of job titles that are maybe not engineers who would engage as Crowdbotics to perform software engineering tasks that they might otherwise in their business, have to go and talk to the CTO to get the resources allocated.
Anand: Okay, great question. So we see two types of customers at Crowdbotics broadly speaking, right?
Anand: We have highly technical teams. These are usually venture-backed startups, or maybe technologies groups inside a big company. They’re just trying to augment the resources to go faster, right? So they’re saying, “We build software all the time, but we have a lot of pressure to try to go faster than we can actually go today. We can’t headcount to hire five more engineers like we know we need to. So we’re gonna turn on Crowdbotics, which is this scalable resource, where we have pools of engineering talent that can drop in and pull out when we need to. And that’s sort of the conventional use case. These are some more common activities that we’re not so unfamiliar with, having worked with remote teams in the past.
But then you got users who are not technical, right? And these ones are more interesting, because for these folks, they’ve always had a battle to try and get engineering resources into their organization. And the engineers usually don’t wanna work on things that are not associated with the product they signed up to work on. So for example, this is the marketing team inside the company or the sales team inside the company. Who say, “We really need to revamp our marketing pages this month.” Or, “We really need to build this one-off mobile application to coordinate with our conference launch.” And these are still involved, highly technical activities that you need somebody who knows their way around an IDE to be able to put together. And, for them it’s kind of a lifeline, which is nice.
The third category, which is really unexpected, I didn’t think people were gonna do this six months ago, but today it’s probably our most popular category, are people who are building a full stack on Crowdbotics. So these are…yeah these are non-technical teams, right? In some cases they’re funded as well. And they’re smart, they have a real understanding of the market, they’ve got paying customers. They don’t have the product sort of built in a way that’s technology-centric. And so for them, Crowdbotics lets them actually wrangle up the whole team, specify exactly what they need, manage the product, and really participate in a hands on way in building technology, which is pretty powerful.
So it sounds like the full stack, which the full stack demands you guys are facing requires orchestrating a lot of more than one person at Crowdbotics. More than one person that member of the crowd to work on that project.
Anand: That’s right. Our typical deployment for a full stack system involves one front-end engineer, one back-end engineer, and then one designer for a short amount of time. Usually there’s also a product manager who is riding along on the project to make sure things run smoothly. But, that combination of three or four is very typical for somebody who’s building a full stack web application. That said, the fact that we’ve done so many of these already, gives us a real economy of scale. And that’s one of the, when I allude to software being a big part of how we build software at Crowdbotics, that’s a big part of it.
If you’re building a web application for scratch, there’s really only a few core architectures you’re going to pick from, right? Maybe we’ll use Node, maybe we’ll use Rails, maybe we’ll use Django. You’re probably not going to use something exotic.
Anand: Although you could, right? And historically those choices are made for really arbitrary reasons, right? So at Lead Genius you use Django, because you know I sat down with my co-founder and said, “Well I know PHP, and I know Python.” And he says “Well I only know Python.” Okay so we built it in Django. We learned as we went along how to build this web application. But most of the choices we made, eventually you get to the right choices. But if you’ve seen these enough times, you know roughly what the right choices are. And you know how to architect for scale. And you’ve already written a bunch of the code, right? So, you just check some boxes in a system, the code gets generated. And then you’ve got a great scaffolding for building out whatever custom integrations you need.
We do see some of the kind of work that you’re describing, where it’s sort of us hooking up APIs to each other. That’s our typical back-end only work. We also see a lot of stuff that’s front-end only, right? We love seeing static sites. I know you’re an aficionado of this, of this style of development. But they call it the “jam stack” nowadays, right?
Max: Oh okay, not familiar.
Max: Me neither.
Anand: Yeah it’s basically front-end only applications that are driven by APIs to drive a lot of interesting interaction. And those are powerful, those are great for marketing pages. HTML/CSS that kind of stuff, that’s our bread and butter, we can do it all day. But, the really interesting stuff if when people build their whole product on Crowdbotics.
Max: One of the topics that’s come up in previous interviews we’ve done, is about the topic of using third-party code. And something I am curious about in both Crowdbotics’ case and in all of those salaried engineers out there’s case, is what is the decision-making process you guys use to decide whether to use third-party code or not. I know NPM for example, Node Package Manager, has a bajilion modules. Probably most of them untested. Even Django itself, I mean, Django or Rails or Node all have certain licenses. So how do you guys vet whether to incorporate third-party code into a project you do for a client?
Anand: Yeah. This is a great question. I think the reaction a lot of first time engineers have to finding all this code on the internet is, they’ll sit and search through a whole bunch of listed code to try to find something that does what they describe, what they think they need. Then they end up picking something off the shelf that sort of looks like it meets their needs, without looking at the license, without looking at the code coverage. So we have a couple of rules of thumb that we try and enforce. The first one is, we’ve got to use mature code, right? So you can’t pick libraries that are not well-tested and widely used or supported by third-party entities.
Max: Breaking those into metrics, how would you come up with a score for third-party code?
Anand: Yeah so code coverage is actually…if you look at a lot of popular repositories that are widely used, and they do publicly display how much code coverage they have in there. Which is nice, that’s a least some metric of the quality of code. You look at the author as well and you say, “Is this…you know published by Facebook, right? Are you building in React, or are you building in with some random piece of code that somebody put out there?” I think one other thing that is…the last piece of course, before we get into that. The last piece is license, right? So we have rules on what licenses you can use, and what license you can’t use. And that keeps it pretty simple.
Max: Is this a topic that seems like the folks who come to Crowdbotics ask about? Like I have a sense that maybe less technical folks who come to Crowdbotics for their projects, might not even think about this problem. The code you guys might use, might not actually be the property of Crowdbotics. And sometime down the road, if you guys are making the full stack product for somebody, there may be issues of licenses that may come up. Is that ever, does that ever come up in discovery sales calls you guys have?
Anand: Yeah. So it almost never does, and I think I’ll tell you why. It’s because, first of all the issue of code, so much of the code that’s being used today in production systems, and code that’s being produced by companies is open source. And it’s been a transformation since the ’90s, it happened so fast that people almost don’t think about it not that yeah, most of the web actually runs on software that’s open source. And nobody worries about their business model anymore, because nobody expects to make money from the actual literal lines of code, right? You’re making your money from a SAS model or from a business built on top of the code, but the code itself is not sort of the relevant part.
That said, people who are savvy enough to recognize that junior engineers may just grab random code and not actually own it. Those folks occasionally ask, and for them we always provide the guarantee that, “Yeah, we’ll vouch the code. Either it’s under a license that’s open source, and recognized enough that we trust. Or the code’s written by Crowdbotic’s engineers, in which case it’s owned by our customers.”
Max: This is a little bit of a non-sequitur, but I’m personally curious, and I think a lot of our audience is curious about how you decided to put your PhD on hold? And in what context did it make sense to you, or did it not make sense when you made the irrational leap to leave the path of academia, and go into starting a company.
Anand: So I guess I’ll say, I don’t recommend it. In the sense that I was very deep in my doctoral work, when I decided to take a leap to work on the company. And…it is useful to have a PhD, irrespective of anything else. And something I still intend to go back to hopefully soon.
With that said, for me it was a question of the performance of the company. So we had started to make good progress. We had money from investors who were asking us to focus on the business, and the effort seemed to be paying off. So once the company started to get a little bit of success, it started to make more sense to spend more time there.
You know if I look at the folks who have graduated at the same time I was due to graduate, and who didn’t go to startups and instead went to academia, they’ve all had pretty good careers. But they’re different careers. So you know they spend their time working on a variety of interesting problems that are very broad. So a professor might have an organization that’s got six graduate students plus 10 undergraduates. They’re working on a dozen different projects, and they get to think about a lot of interesting questions. The same time, as startup founder you think about a very narrow set of questions very, very, very deeply. I think there are a few upsides, big upsides to startup lifestyle over academia, and vice versa as well.
As a startup founder, I think you get some recognition that is a little bit harder to get in academia. You obviously have the ability to raise capital more easily, I think, than as an academic. Although you’re still chasing money in the same way that academics are always writing grants. But it is easier to get that money in founder land. And of course, if you’re successful as a founder, it can be quite financially rewarding. Though it’s difficult to get to that point. It can also be very satisfying.
Max: I know that one of your kind of passion projects is helping out other entrepreneurs, other startups that are going through the growing pains that you yourself have experienced over the last I guess six, seven years of entrepreneurialism that you’ve been doing. And one of the things that you’ve said to me that has piqued my interest is about how first time founders spend a lot of time not building their businesses. Coming up with things that they do that have nothing to do with growing your business. Is there any question that you recommend these people ask themselves on a regular basis to get them back on track? Like…
Anand: Yeah. Oh man, that’s such a great question. So this is maybe the most common mistake that first time founders make is they focus so much on their interest in whatever the domain is they’re focusing on, they forget that the business is supposed to be a business. It’s trying to make money, and get to scale and grow. And it happens so often that there’s been this whole inversion of how people teach people to start companies now. Where it’s lean startup, right? Where they say, “Sell first, don’t build your product. Go sell it first. And then…”
Max: Sounds like some of Crowdbotics clients are of that format. Is that fair to say, or no?
Anand: We’re very careful. Whenever I take on customers, we don’t just take people’s money who want to build anything, right? And we certainly could. We will really, I really push people to found out and understand do they have a clear customer in mind? Where the technology is the last barrier before they can actually get money from that customer, or start to pilot from that customer or sell to that customer. It’s really dangerous to just spend a lot of money building software, and…not have that turn back into more cash flow.
Those kinda customers are something that a lot of first time founders don’t think about. Or they wait too long to think about, or they spend a year building a product and then they say, “Who’s going to buy it?” And I understand that urge, because…and I came from the same background. Our first company we spent almost a year, maybe two years almost, trying to figure out this question, “Who was gonna be the buyer?” And we just built code the whole time, because we liked building code. And I actually really enjoy writing software, right? It’s fun. But it’s irrelevant if you can’t find a person who’s gonna actually buy that, or pay you money, or some way that you can make money from it.
I think the main question that I would ask myself, and I asked myself this for the current company too. “When I’m building software, is this going to…how long will it take before we get to the point where the software is making money?” And if the answer is more than a day or two, okay then think about a smaller version of this that we can build. Or go get a customer who’s willing to pay for it, and then go build it. Which is a much safer way to go. That’s actually how we did things at Crowdbotics too.
We have of course a long term vision that we’re gonna automate. Literally automate parts of the development cycle, instead of having such a heavily reliance on human beings. But in the short term, we’re able to get to some decent scale, and by wrangling big teams of human beings. And then gradually we introduce automation to help make their lives easier, and their process easier. But that choice meant that we had customers day day one for the company, which was a very liberating position to be in.
Max: Certainly. For members of our audience that are curious about what being a member of the Crowdbotics crowd is like, do you mind sharing kinda like what the model is, what the perspective is as a member of the Crowdbotics crowd?
Mentoring is a big part of the community experience for us inside Crowdbotics. We like to think a lot about this question of, “How do you improve somebody’s skills as an engineer?” Something I care about a lot too.
Max: Oh, absolutely.
Anand: And so we spent some time figuring out ways we could introduce this. So part of that is putting people into interesting projects where they can develop their own skills. Or cross-pollinate skills with other people. Part of that is us doing hackathons and other internal community projects where we can trade skills. And part of that is also us doing direct teaching, right? So supporting people when they’re doing courses, or teaching people, having people who are senior in our community teach an expertise area to somebody else in the community.
Max: So is, sorry for cutting to the chase on this, but is the compensation hourly? And for people whose alternative might be to seek or stay at their salaried job which pays out bi-weekly, how do you find or reassure candidates or members of the crowd that there’s never gonna be slackness in how much work they have, as if they’re hourly employees, hourly working?
Anand: Yeah. So everyone does hourly inside Crowdbotics. And we don’t make that assurance at all, in fact the opposite. We tell people that, “We have work right now, this is a great fit. And next week our project may widen down, we’ll take a break and you can work on something else that’s not affiliated with Crowdbotics.” What we do tell people though is that you get two things. The first is the freedom of being able to work sort of on a basis that you choose, from your own house. Setting your own schedule, you can travel when you work. There’s a lot of upside to things like that.
Of course, the trade-off is that it’s contract-based work. So you may end up with a lot of work, and then take an extended break.
So, we also tell people that we are investing in them. So if somebody enjoys working with us, and they’re doing well, then we give them more and more work until the point where we feel like they’re advancing on their skills. And we try and make assurances that for folks who are working with us over an extended period of time, that we are building their skills up while they’re working with us.
But most people who come to us are not quitting their day jobs to come and work with us. A lot of people who work with Crowdbotics are folks who are either coming out of…they’re working in development centers, they’re finishing up working on their own startups. Sometimes they’ve been working for big companies, and decided they wanted to work in a more remote, flexible environment. They just had kids, or something else that was a life change that pushed them into doing the work that we do. But it’s not typical that, and our staff is not in San Francisco. So it’s not typical that somebody jumps ship from Google, and then decides to work for us 40 hours a week.
Max: This is early days, so feel free to pass on this, but I’m curious if the demographics of the Crowdbotics crowd mirror kinda the salaried workforce in software engineering. By that I mean gender diversity, age diversity, tenure in the job.
Anand: Great question. We have an explicit focus on women, actually. So we are specifically trying to hire and promote female software engineers, and we recruit aggressively from parts of the world where either there’s not an historic bias towards software engineering being male-focused. Or there is a just not a strong software engineering culture. And both of those things allow us to tap into markets where people don’t think to look for software engineers, which is pretty cool. Some of that’s even in the US too.
So our first Crowdbotics hire was a guy in Missouri. And yeah, he couldn’t find local work in software engineering, because…and not because he wasn’t brilliant, because he was. But it’s because Google doesn’t do a lot of recruiting inside Missouri as it happens. I mean, you came outside the Bay Area too, so you know that the further away you go from these sort of big concentration areas of software engineers, the more diverse your choices get or have to get. Not a desire to [inaudible 00:35:49]. So we have a workforce in Kenya, in Indonesia, in Peru, in Mexico City. Basically any place that we think there’s gonna be…the Middle East, population of folks who are not maybe getting the kind of opportunity that their talents would allow them.
We certainly skew young, although not exclusively. We have members of the workforce who are coming out of sort of extremely long careers working inside Fortune 500s, and say, “Okay, I’m done with this. I’m gonna do something that’s more focused around my own life. Let me be an online Crowdbotics software developer.” Great. That’s somebody who’s sort of gray haired and willing to pass on that advice in mentorship and work on a schedule they like, so it works for them.
But a lot more of our folks are people who are let’s say under 30s. And that’s always fun, because they’re really gung-ho about the model. They’re gung-ho about the idea, they’ve really got the passion for the work. Which is, as a software engineer, really important.
Max: So for prospective customers of Crowdbotics, or prospective crowd members of Crowdbotics, what’s the best way to find you guys?
Anand: So you can go to our website. We’re actually officially in stealth, so you’re not gonna find too much. But I mean, of course viewers of Accidental Engineer are welcome to get in touch with us, either/or. Or if they wanna try the product out, they can fill out the form on the site, and we’ll onboard them in the system. Even though we’re in stealth, we’re in production with 40 customers, paying customers. So we were very happy to talk to folks who are coming in through a trusted referral like this, or people who wanna work with us. We always love to talk to new engineers, because that’s something that is the basis of our business.
Max: Awesome. So, Crowdbotics.com, this was Anand Kulkarni. Thanks for joining us Anand.
Anand: Hey, my pleasure.
Max: Hope to do it again very soon.
Anand: Yeah, looking forward to it.