This has to be one of my favorite customer development tips: using Mechanical Turk to do customer interviews.

Nick Soman, Founder of LikeBright, and I discuss how he used Mechanical Turk to interview 100 customers in 4 hours, and how that got him into TechStars Seattle.

If you haven’t heard of Mechanical Turk (mTurk), here’s what you need to know:

• it’s a website were lots of people, do small amounts of work, for small amounts of money
• Who does work on mTurk? Although a bit stale, these stats indicate a fairly representative cross-section of the population.
• Why would they do work for such little $? Something to do during free time. It’s a replacement for solitaire, not a replacement for a job.

Since Nick and I talked, I’ve done a couple dozen interviews this way and the results have been fantastic. Nothing like “getting out of the building” at home, at midnight, with an ice cream sandwich in hand.

[Update – 5/29/2015]

There’s an easier and faster way to do everything listed here. Our friends at Customer Discovery Ninja have automated this entire workflow below and made it easy for folks outside the US to access Turkers.

Feel free to familiarize yourself with the steps below if you like, but the fastest way to talk to customers is going to be Customer Discovery Ninja.

Sample interview and instructions:

[soundcloud url=”http://api.soundcloud.com/tracks/56705993″ iframe=”true” /]

1. Write the interview script

To make sure I accurately test my hypothesis, and so I don’t forget any questions, I always write up my interview script ahead of time.  Coming up with good interview questions is another post on its own, but here’s the intro I use:

Hi there, my name is Justin.  Can I get your first name please? *

Great, thanks ______.  Like I said, my name is Justin and I’m here in Seattle doing a little research.  I’m happy to tell you about the project I’m working on, but so that I don’t accidently bias any of your answers, I’ll wait until we finish the survey.  Is that alright?

Great.  So that I don’t have to slow us down to take notes while we’re chatting, is it okay if I record this call?  Great, here we go.

*Note: about half the time, people hang up after my first line. My assumption is that the hangups were expecting an automated phone survey, as opposed to a personal interview.

2. Get a (new?) Google Voice number

You’re going to post this number on mTurk, and strangers are going to call it so if you’re not down with that, create a new Google Voice account and grab a new number.  I’ve been using my actual Google Voice number and haven’t had any negative affects so far, but do what makes sense for you.

Outside the US/Canada: Google Voice is restricted to US/Canada so if you’re not there, here’s a great blog post on how to borrow a Google Voice number. (Note: may be against Google’s TOS. Use your judgement.)

3. (Optional) Forward Incoming Google Voice calls to your Gmail

I like answering my interview calls via GTalk within Gmail for a couple reasons:

1. The gmail interface makes it easy to record the call
2. It’s easier for me to talk hands-free with my computer

Instructions on forwarding Google Voice calls to your gmail are here: http://support.google.com/chat/bin/answer.py?hl=en&answer=187936.

4. Create a HIT on mTurk

Create a new HIT on http://requester.mturk.com

Describe the HIT

Super Important:

You must turn off “Master Turkers.” Master Turkers are a pre-screened, and very small, subset of the MTurk population. We want any folks on MTurk to be able to contact us, as long as they meet our qualifications. Here’s how to do that:

Pick a Price

Write up the HIT

Feel free to copy and paste this HTML:

<p><span style=”font-family: Arial;”>If you are a parent who picks your kids at day care at least once/week, please call us for a 5-10 minute phone survey.</span></p>
<p><span style=”font-family: Arial;”>Please dial the following number:</span></p>
<ul>
<li><span style=”font-family: Arial;”>*67&nbsp; [your google voice number]</span></li>
</ul>
<p><span style=”font-family: Arial;”>Note: dialing *67 before the actual phone number will protect the privacy of your phone number. &nbsp;</span><span style=”font-family: Arial;”>If you reach voicemail again, please wait 10 minutes.</span></p>
<p><span style=”font-family: Arial;”>&nbsp;<b>Required after Calling</b>&nbsp;- after we finish the survey, we will give you a password to confirm you successfully completed it. Please enter it below:</span></p>
<p><span style=”font-family: Arial;”><b>Password:</b>&nbsp;<textarea rows=”1″ cols=”80″ name=”answer”></textarea></span></p>
<p><span style=”font-family: Arial;”>Thank you very much! &nbsp;We really appreciate your help! &nbsp;</span></p>

*Note: the “password” is a word you tell your interviewee to type in once the interview is complete.  You’ll see what they type in before you approve the HIT (i.e. pay them) so you can ensure only the people who successfully completed the interview get paid.

Outside the US – mTurk needs a US address to sign up.  I’ve heard of people using any US address to create their accounts if they’re outside the US. Again, may be against TOS. Use your judgement.

5. Publish the HITs

6. Get an ice cream sandwich

Now it’s time to wait for your first call. When it comes, answer it with your intro and once you have permission, record the call so you can share it with your team or review it later.

When you’re done, you can review the calls on Google Voice

Video Walkthrough

Using mTurk for the first time can be a bit confusing, so here’s a video walkthrough of the mTurk steps above:

Conclusion

If your target customers are on mTurk, which is likely if you’re building a B2C company, this is an awesome way to get lots of interviews in a short amount of time.

Thoughts, questions, or other suggestions for quick customer interviews? We’d love to hear ’em.

Big thanks again to Nick Soman from LikeBright for sharing this tip, Andy Sack for making him do it, and Dan Shapiro for teaching all of us the power of mTurk for customer development.

Outside the US? Or Don’t want to Mess with Mechanical Turk?

There’s an easier and faster way to do everything listed above. The guys at Customer Discovery Ninja have automated this entire workflow, which also makes it easy for folks outside the US to access US Turkers.

What’s Next?

Join the experiment – follow along via Email or RSS for updates on our next post: Prioritizing Customer Segments with Excel.

Customer Development Made Easy…

The results from our email interview experiment are in and, thank goodness, our customers saved the puppies.

* I didn’t get a chance to send follow-up emails for either of these emails.

Note these results include two new emails that weren’t detailed in the original blog post.  The “French Vanilla” email was the result of a friend telling me I had done everything wrong and that he could do a much better job:

French Vanilla

The Why? Email was inspired by my friend Adam Loving at Linksy who sent out an email to his customers asking for 2-3 words about their online marketing challenges:

Why?

Interesting stuff:

1. Follow-up emails produced the most interviews.  Each person who didn’t reply to my first email got another email a couple days later repeating my request for their time.  Doing that more than doubled the number of interviews I got to do. Highly recommend FollowUpThen.
2. Overall, the interview conversion rate was low.  7% on average, with the “winning” email converting at 12%.
3. Sample size was probably too small to claim anything scientific, but “Rocky Road” generated “3x” the interviews “Vanilla” did.
4. The “Why?” email performed well, but didn’t really let me dive as deep as I would have liked into my customer’s pain.
5. Our readers are smarter than I am.  I thought “Chocolate” would win, but ya’ll were right:

Looks like this experiment provides two takeaways:

1. Be sure to follow-up on requests for interviews.
2. Being humorous/unique/edgy helps. Not a lot, but it helps.

Join the experiment – follow along via Email or RSS for updates on our next post: Customer Interviews using Mechanical Turk.

Customer Development Made Easy…

I went to Lean Startup Machine Seattle and it was, without a doubt, worth it. When LSM comes to your city, I highly recommend going.

There are a couple reasons I typically don’t participate in workshops/conferences:

1. I don’t trust that the people talking know what they’re talking about
2. I can learn what they’re teaching somewhere else (e.g. book, blog, etc.)
3. I’m cheap and I don’t think it’ll be worth the money

Here’s how LSM mitigates those risks.

For Founders, by Founders

The guys running LSM are founders. They know what it’s like to start a company, and they know what it’s like to see it fail.

They’ve used those failures to figure out why they failed. They’ve consumed Blank, Reis, and Maurya’s ideas, distilled them, and produced a process that teaches the most fundamental principle of lean – how to identify our riskiest assumption and test it with an experiment.

This isn’t junk, it’s not bs, it’s not some guy talking to hear his own voice.  It’s stuff I think every founder should learn, provided by other founders.

LSM is an Experience – you can’t read it

The goal of the weekend is to change the way we think.  Instead of being execution focused, lean startups need to be testing & learning focused – but after years of hearing “first to market advantage”, and “those that ship win”, it takes more than reading books and blogs to rewire our brains.  We have to learn by doing.

LSM is learn by doing.

We probably spent less than an hour being “taught” how to walk through the LSM Canvas (different than the business model canvas).  The rest of the time, we were identifying our assumptions, prioritizing them, and testing them…over, and over, and over again, until our brains started to rewire themselves.

You can’t “read” an experience, you have to do it.

Not Going is a Waste of Time

If we agree that doing is a more effective way to learn than simply reading, we’ve got two choices re “doing:”

1. Applying the lean startup techniques we read about to our own startups and over the course of months/years, we’ll have this stuff down cold
2. Learn it in a weekend

It’s really that simple.  LSM is specifically designed to teach the essence of Lean in 54 hours.  Our startups on the other hand are designed to be businesses, not teaching tools.  Using them as such is possible, it’s just not an efficient use of time.

In terms of the amount of time and energy the weekend saved me, I’d say I’d pay $500 for it.  It doesn’t cost that much, but they could charge it and I’d still recommend it to people.

Conclusion

If you’re reading this, I highly recommend you go to an LSM.

Justin

PS – Suggestions for the Future

While the weekend was absolutely worth it, there were a couple things that could be improved:

1. The “mentors” didn’t know lean.  While a couple did, there were certainly a few people walking around and giving advice that really didn’t know process we were trying to learn.  It would be great to require the mentors take a crash course in the techniques beforehand.

2. Cut the speakers.  There were a handful of speeches throughout the weekend, but I wasn’t really sure why.  They killed our momentum and they didn’t have anything to do with Lean – they were just startup oriented talks.  Imo, we can get that stuff elsewhere, make the speeches lean-oriented or cut them.

3. The “pitch” competition at the end wasn’t a good use of time.  Not only did it emphasize the wrong thing (results, as opposed to learning), the judges didn’t have a clue what it really meant to be Lean.

   I’d suggest doing something like a “Lean Tournament” as a finale instead.  Two teams at a time are pitted against one another and given a customer hypothesis, a problem hypothesis, and a solution hypothesis.  The two teams have 1 minute to identify the riskiest assumption and design a test for it.  Lean-educated judges pick the winner and it continues bracket-style until there’s a winner.  A high energy way to end the weekend that emphasizes learning, not execution.

Btw, don’t let these suggestions dissuade you, they comprise maybe 5% of the weekend.  I’m including them for completeness sake.

Join the Experiment – Want to save more time learning lean? Follow along with us via Email or RSS.

Lean Startup Machine is like a Startup Weekend, except exclusively focused on business model validations as opposed to product building.  It’s coming to Seattle next weekend and I’ve decided to go.

Here’s why I’m going (and why you might think about doing the same):

My name is Justin, and I’m Addicted to Writing Code

I love building things. It’s why I became a developer. It’s a rush, it’s a high, it’s nerd cocaine, and I’m addicted.

Combine that with years of mantras like, “Those that ship, win!” and “code, code, code” and as developers, we’re taught to code first, and worry about who will buy this thing, never.  “Developers code, let the business guys figure out how to sell it.”

Of course, now that I do my own startups, I think I’m that business guy, but subconsciously my inner addict is likely running more of the show than I’d care to admit. To be truly lean though, and to increase the likelihood of my startup’s success, means I need to focus not on coding, but on learning:

Deadly Startup Sin #4: Emphasizing execution instead of testing, learning and iteration – Steve Blank, 9 Deadliest Startup Sins

The fastest way to lessen the impact of my addiction, is to immerse myself in learning Lean processes.  Lean Startup Machine feels like rehab for those of us who are addicted to coding.

The First Time, is the Worst Time

Whether it’s the first pancake off the griddle, the first version of a product, or the first time following a process, my first time is always my worst time.

With that in mind, why should I subject Bounce, a business I’d really like to succeed, to my first time practicing Lean Startup processes end-to-end?  At Lean Startup Machine, I can practice Lean techniques on other people’s business models, and take what I learn and apply all the good stuff to my own.

I got a Discount

I’m cheap. Truth is, I feel genetically predisposed to not spending money, and while it’s handy to be a frugal founder, I worry sometimes that I miss out on opportunities like Lean Startup Machine because they cost $.

Luckily, I didn’t have to pay the full $299 price…and if you’d like to go, you won’t have to either.  You’ve got two choices re discounts:

1. 30% off courtesy of John Sechrest – http://bit.ly/LH62we
2. Lean Startup Machine + Startup Weekend Bundle – http://bit.ly/LH6b2s

While financially, they’re essentially the same deal, that second one is particularly interesting if you’re wondering about how Lean Startup Machine compares to Startup Weekend – the Lean Startup Machine team will pay for your next Seattle Startup Weekend.

That means you can validate your business model at Lean Startup Machine, and then build it at a Startup Weekend.

Conclusion

I’ll be at the Lean Startup Machine next weekend, breaking my coding addiction by learning Lean processes on someone else’s business model.  You should come too.

Join the Experiment – Can’t make to Lean Startup Machine but want to hear how it went? Get updates via Email or RSS.

[8/14 Update: results from this experiment are in]

Last year I was hanging out on OkCupid and realized online dating is another startup, replete with landing pages, copy optimization and “filling the funnel.” The only difference is that we’re selling ourselves instead of software.

Hoping to up my conversion rate, I wanted to figure out what style of messages OkCupid members respond to the most, so I did a little a/b testing with four types of messages:

1. “Hi” – Fairly lame, but simple and says, “look at my profile”
2. Generic, short message – Same as above, but not quite as lame as the word “hi.” Something like (literally), “look at my profile.”
3. Personalized positive message – Prove that I’ve read her profile, “Canceling Arrested Development was a travesty, but I take solace in watching It’s Always Sunny while huffing paint.”
4. Personalized Negative Message  – Nice guys finish last…“You’re wearing the same shirt in 3 of your photos.  Don’t get out much?”

Care to guess which performed best?

From one of the respondents:

It’s interesting that I replied to this right away even though you basically insulted me twice.

So how can we apply this to  customer development?  What if we tried the same experimentation when contacting our customers?

Requesting Customer Interviews

First off, I won’t be (intentionally) insulting any of my customers.  That said, I have the email addresses of a couple thousand people who have requested the iPhone version of Bounce, so we’ve got an ample size to run some a/b tests.

For this experiment, I’m looking to do some customer interviews to answer questions like:

• Why did they give me their email address?
• What do they expect Bounce to do?
• How much would they pay for it, if it did what they wanted?

Now let’s see which of the following emails results in the most interviews:

Vanilla

Simple (and boring) but to the point

Chocolate

Trying to add a bit more flavor

Rocky Road

Like the Chocolate version, but a little extra puppy killing

What’s your Hypothesis?

I sent 75 emails out this morning, time to place your bets.  Which one do you think will perform best?

Will the puppies live?

Join the experiment – get notified when the results are in via Email or RSS.

Customer Development Made Easy…

When we started Nimbus Health, we knew we were making some assumptions, but there were a few things we knew too:

1. A launch customer is ready to use our product
2. We know how much they’ll pay
3. There are 50+ companies just like our first customer
4. Our first customer is influential and will refer us to those customers

Of course, each of these things that we “knew”, turned out to be assumptions as well. Bonus! They were all wrong.

1. Our customer was ready to use the product, but it wouldn’t matter.  There was another decision maker involved (their customers) who would make the final call.  Almost none of them did.
2. “Our pricing isn’t just cheaper than your existing solution, it’s easier to understand!” They didn’t want cheaper or “easier to understand” – they wanted “easier to pass onto their clients.”
3. That the 50+ other companies “understood” the pain we were solving was true.  That they felt is so badly they’d be willing to pay for a solution was not.
4. Referrals, HA! Half the other companies are competitors and those that aren’t don’t really get along that well anyway.

So, wtf? How are we going to be able to do anything if even the things we know to be true, turn out to be false? Enter the Lean Canvas.

Lean Canvas = Assumption Destruction

Lean Canvas is technically a morphing of Alexander Osterwalder’s Business Model Canvas by Ash Mauyra, but what really matters is that it is a single sheet of paper that makes it obvious which of our assumptions are wrong.

I haven’t filled one of these out before, so we’ll be learning together. Here’s how I’m thinking it’s going to work (btw, you should follow along fill one out for your business too):

1. We fill out each of the section of the canvas
2. Recognize that every single thing we’ve written down is a hypothesis
3. Call out each of the experiments we’re doing to test these hypotheses
4. Iterate on our canvas until we’ve got a business model we’d bet our mother’s retirement on

Like you, I don’t have a lot of time to waste on documentation, so I didn’t.  Here’s what I come up with in half an hour for Bounce:

If you want to see how I went about filling it out:

Fwiw, the most helpful thing so far was to really call out my problem statement. That has really helped to drive a lot of clarity into my customer value proposition. Now I’m looking forward to coming up with experiments to test these hypotheses.

What’s Next

Join the experiment – subscribe: Email | RSS

Teases

A comment thread with the extremely knowledgeable Tristan Kromer, reminded me of several misleading responses I’ve heard from customers.  At first glance it sounds like they’re saying, “Yeah, I’d pay for that”, but are really just being teases.

Consider the following replies to, “Would you pay for it?”:

• “Yeah, I’d definitely recommend it to my boss”
• “I’m probably not your target audience, but I know other people would pay for it”
• “I don’t write the checks, but I’ll definitely bring it up at the budget meeting”

Good news! We can tell our advisors that we’re going to get discussed at a budget meeting! Wrong. These are responses I’ve heard from customers who like the idea (yeah!), but none of the above have the authority to cut me a check.

Teases are the worst kind of customer. In fact, they’re not our customers at all because they’re not decision makers. We need to talk to the folks with the money before we know the real answer to our question.

Gold Diggers

Now, consider these replies:

• “I like the pitch. Come back when you have a product.”
• “Hmm, maybe if it also did _______.”

That’s right…these customers are gold diggers. These customers didn’t find our initial value prop compelling enough, so they want more!

To be clear, this is very valuable feedback, it’s just not positive, valuable feedback. If they don’t find our pitch compelling enough to start talking $ and cents, we’re trying to build something that isn’t solving a pressing concern. Time to incorporate their feedback or hit the drawing board.

Finally, we have the liars:

These customers want so desperately to give us good news, they delude themselves; if we’re not careful – they’ll do the same to us.

Actions speak louder than words. If they haven’t established a pattern of buying products like ours before, we’ve got a long road to hoe, before we nail this deal.

Beware of customers who sound positive, but stop short of talking about money. These sirens will smash your business model against the rocks of reality.

Let $ in our pockets be our compass.

So we want to ask customers if they’ll pay for Bounce before we build it, but first we need to figure out who they are.

Step 1 – What problem do we solve?

Since people don’t buy products, they buy solutions to problems, we started by identifying the problem we solve:

Being late is stressful. Bounce eliminates that stress by making it easy to be on time.

Step 2 – Who has the problem we’re solving?

Anyone who doesn’t like being late.

Step 3 – How will we tell people with this problem about our solution?

Hmm…now we’ve got a problem.

No one identifies with the customer segment we’ve called out here.  No one joins the, “People who don’t like being late” group on LinkedIn.  No one reads ThePunctualityBlog.com.  As we learned with OnCompare, if there’s no way for us to interact with these people, they’re not a customer segment.

Step 2 (again) – Who has the problem we’re solving?

Who has lots of meetings outside their office as part of their job?
Real Estate agents, sales people, event planners, founders, general contractors, etc.

What kind of companies “sell” being on time? 
Florists, caterers, utility installers/repairers, plumbers, electricians, etc.

Who is routinely affected by bad traffic?
Long distance commuters, van pools, people who commute over a bridge, etc.

Who really wants to be on time for their personal life? 
Single people who are dating, busy parents, the “chronically late”.

Who is unfamiliar with the local traffic patterns? 
Business travelers, people who have just moved to an area.

Step 3 – How will we tell people with this problem about our solution?

Almost all of the customers above self-organize into groups.  Either actual LinkedIn, Meetup or trade groups, or they’ve formed ad hoc communities on blogs or forums.  Or perhaps, they have a common set of suppliers, or they advertise in the same set of magazines. For example:

• Real Estate Agents – LinkedIn, meetup, trade group, blogs and forums
• Florists – trade groups, all interact with FTD, all interact with wedding planners
• Bridge commuters – listen to the radio, subscribe to traffic twitter feeds, download traffic apps
• People who have just moved to an area – read hyperlocal blogs, new college students read college newspapers

Etc.

Step 4 – Identify the Influencers

Now that we know the groups our customers hang out in, we need to identify the influencers of those groups.  For blogs that means finding the authors, for meetup and linked in groups it’s finding the moderators, if we need press it’s finding the person who is going to write our story, etc.

Of course, we need more than their name, we need at least their email address. To find that info for every one of our influencers requires hours and hours of tedious net-scouring, and to be perfectly honest, I hate don’t enjoy it.

Luckily, my friend Marcos Polanco introduced me to oDesk – a fantastic resource for internet research.  The idea is simple, pay someone else $5/hr to do the tedious work.  I write up detailed instructions on how to find someone’s email address including things like:

• Do google searches for:
  • [email protected]
  • [email protected]
  • [email protected]
• Use www.Jigsaw.com (which is good advice for everyone, btw)

And then I tell them to “Find the contact info of the top 5 hyperlocal bloggers in each of these cities: Los Angeles, New York, Chicago, San Francisco and Seattle.”

5 hours, and $25 later, I’ve got 25 email address.  So much better than me doing it myself.

Step 5 – Provide Value

Now that we’ve got our influencers’ contact information, it’s time to reach out.  Of course sending a, “You should tell your group about my awesome product!” email isn’t going to work.  If we’re going to get anywhere with influencers, we need to provide them and their group value.

Hey [local blogger name],

I saw your article on how awful traffic is getting in Chicago [link to article].  A friend and I are launching a Kickstarter project [link] to build an app that makes getting around Chicago easier – despite worsening traffic.

Think [blog name] readers would be interested in something like that?”

When I’ve done this before I’ve pitched the actual product, not a Kickstarter project to fund the product, so I’m not sure what kind of reaction we’ll get. Either way, we’ll use FollowUpThen to bug them about it at least one more time. ;)

To Recap

Those are the 5 steps we’re using to identify, segment and contact our customers:

1. What problem are we solving?
2. Who has that problem?
3. How will we tell people with this problem about our solution?
4. Identify influencers
5. Provide value

I’ll keep you posted on how it works out. In the meantime, alterations or alternatives to this approach are welcome.

Join the experiment – subscribe: Email | RSS

The scenario:

1. I want to get some feedback from a potential customer
2. I find their contact information
3. Send a succinct, but personalized, email asking to chat or meet up
4. Crickets

What now? Well, if I even remember that I emailed them (unlikely) I could chalk it up as, “I’m not providing enough value for them to talk to me.” But what if this person, like me, is busy, forgetful, or both? Worth following up one more time, right?

Enter FollowUpThen, as proselytized by my good friend Rishi Shah at Flying Cart.

1. It’s free
2. It’s easy
3. It sends you a reminder to follow up with your customer if she didn’t get back to you

Now I send customer emails and bcc [email protected].  If my customer doesn’t get back to me before then, FollowUpThen will remind me to get back to them.

PS – It’s like a free version of Boomerang, which is what I searched for to find it.

Join the experiment – subscribe: Email | RSS