Using mTurk to interview 100 customers (in 4 hours)

This has to be one of my favorite customer development tips: using Mechanical Turk to do customer interviews.

Nick Soman, Founder of LikeBright, and I discuss how he used Mechanical Turk to interview 100 customers in 4 hours, and how that got him into TechStars Seattle.

If you haven’t heard of Mechanical Turk (mTurk), here’s what you need to know:

  • it’s a website were lots of people, do small amounts of work, for small amounts of money
  • Who does work on mTurk? Although a bit stale, these stats indicate a fairly representative cross-section of the population.
  • Why would they do work for such little $? Something to do during free time. It’s a replacement for solitaire, not a replacement for a job.

Since Nick and I talked, I’ve done a couple dozen interviews this way and the results have been fantastic. Nothing like “getting out of the building” at home, at midnight, with an ice cream sandwich in hand.

Sample interview and instructions:

1. Write the interview script

To make sure I accurately test my hypothesis, and so I don’t forget any questions, I always write up my interview script ahead of time.  Coming up with good interview questions is another post on its own, but here’s the intro I use:

Hi there, my name is Justin.  Can I get your first name please? *

Great, thanks ______.  Like I said, my name is Justin and I’m here in Seattle doing a little research.  I’m happy to tell you about the project I’m working on, but so that I don’t accidently bias any of your answers, I’ll wait until we finish the survey.  Is that alright?

Great.  So that I don’t have to slow us down to take notes while we’re chatting, is it okay if I record this call?  Great, here we go.

*Note: about half the time, people hang up after my first line. My assumption is that the hangups were expecting an automated phone survey, as opposed to a personal interview.

2. Get a (new?) Google Voice number

You’re going to post this number on mTurk, and strangers are going to call it so if you’re not down with that, create a new Google Voice account and grab a new number.  I’ve been using my actual Google Voice number and haven’t had any negative affects so far, but do what makes sense for you.

Outside the US/Canada: Google Voice is restricted to US/Canada so if you’re not there, here’s a great blog post on how to borrow a Google Voice number. (Note: may be against Google’s TOS. Use your judgement.)

3. (Optional) Forward Incoming Google Voice calls to your Gmail

I like answering my interview calls via GTalk within Gmail for a couple reasons:

  1. The gmail interface makes it easy to record the call
  2. It’s easier for me to talk hands-free with my computer

Instructions on forwarding Google Voice calls to your gmail are here: http://support.google.com/chat/bin/answer.py?hl=en&answer=187936.

4. Create a HIT on mTurk

Create a new HIT on http://requester.mturk.com

Select "other" template
Select “other” template

Describe the HIT

Describe the HIT
Describe the HIT

Super Important:

You must turn off “Master Turkers.” Master Turkers are a pre-screened, and very small, subset of the MTurk population. We want any folks on MTurk to be able to contact us, as long as they meet our qualifications. Here’s how to do that:

Pick a Price

Pick a price
Pick a price

Write up the HIT

Write up the HIT
Write up the HIT

Feel free to copy and paste this HTML:

<p><span style=”font-family: Arial;”>If you are a parent who picks your kids at day care at least once/week, please call us for a 5-10 minute phone survey.</span></p>
<p><span style=”font-family: Arial;”>Please dial the following number:</span></p>
<ul>
<li><span style=”font-family: Arial;”>*67&nbsp; [your google voice number]</span></li>
</ul>
<p><span style=”font-family: Arial;”>Note: dialing *67 before the actual phone number will protect the privacy of your phone number. &nbsp;</span><span style=”font-family: Arial;”>If you reach voicemail again, please wait 10 minutes.</span></p>
<p><span style=”font-family: Arial;”>&nbsp;<b>Required after Calling</b>&nbsp;- after we finish the survey, we will give you a password to confirm you successfully completed it. Please enter it below:</span></p>
<p><span style=”font-family: Arial;”><b>Password:</b>&nbsp;<textarea rows=”1″ cols=”80″ name=”answer”></textarea></span></p>
<p><span style=”font-family: Arial;”>Thank you very much! &nbsp;We really appreciate your help! &nbsp;</span></p>

*Note: the “password” is a word you tell your interviewee to type in once the interview is complete.  You’ll see what they type in before you approve the HIT (i.e. pay them) so you can ensure only the people who successfully completed the interview get paid.

Outside the US – mTurk needs a US address to sign up.  I’ve heard of people using any US address to create their accounts if they’re outside the US. Again, may be against TOS. Use your judgement.

5. Publish the HITs

Create a new batch
HIT it!

6. Get an ice cream sandwich

Now it’s time to wait for your first call. When it comes, answer it with your intro and once you have permission, record the call so you can share it with your team or review it later.

Incoming call
Incoming call

Record call
Record call

When you’re done, you can review the calls on Google Voice

Review recording
Review recording

Video Walkthrough

Using mTurk for the first time can be a bit confusing, so here’s a video walkthrough of the mTurk steps above:

Conclusion

If your target customers are on mTurk, which is likely if you’re building a B2C company, this is an awesome way to get lots of interviews in a short amount of time.

Thoughts, questions, or other suggestions for quick customer interviews? We’d love to hear ‘em.

Big thanks again to Nick Soman from LikeBright for sharing this tip, Andy Sack for making him do it, and Dan Shapiro for teaching all of us the power of mTurk for customer development.

What’s Next?

Join the experiment – follow along via Email or RSS for updates on our next post: Prioritizing Customer Segments with Excel.

We want to Help

If you want help interviewing – you’re not alone :)

There’s a group of us who are asking, and answering, questions about interviewing every week.

  • What do I ask customers?
  • How do I get them to give up their time for an interview if I don’t have a product?
  • When can I use surveys and when should I interview?
  • When do I talk about my solution?

Join a Customer Acquisition Team to trade tips with other founders who are actively interviewing their customers.

41 comments

  1. markhoroszowski

    Great post, and great interview. I’ve been using mTurk for surveys, finding emails, and testing features – this takes it to a new level that I’m excited to try.

    Thanks a TON for posting the tips.

  2. pmmucsd

    Google Consumer Surveys is a really fast and affordable way to interview 1000s of users/customers that are representative of the US population .

    • markhoroszowski

      @pmmucsd – GCS is definitely a useful resource, but you are much more limited by the types of questions you can ask, and the price points since it dictates not only price per question, but also batch size. For scrappier startups, no surprise if GSC will be too expensive, and too limited. And you can’t collect respondent data to do actual interviews, which in the case here with Bounce and LikeBright, were essential to understanding customer needs.

  3. Hakon

    Wow, hell of a post! Thanks for the great tips and applicable content! This is a fantastic addition to the utility belt. =)

    I really liked the idea of doing this as a group and getting some friends involved. Nice way to inject some new perspective into your conversations about what you’re working on.

  4. Shell At Cloudmebaby

    Let me first say that I am very impressed with this post great work. I would also like to throw out a little bit of information from the workers (turkers) end of the spectrum. There are a few forums out there dedicated to mtruk and working in the cloud that can answer any further questions people may have about placing work on mechanical turk. If you would like some worker insight either as a requester or a worker I invite you to join us at CloudMeBaby.com. We are always willing to help out and I am sure you will find the wealth of information both refreshing and informative. Again, thank you for the very informative post.

  5. Wei Lieh Ng

    Hey Justin, great post! It’s because of this that I’ve switched directions from considering professional user recruitment (read: expensive) to MTurk instead.

    I’ve just created a HIT to survey online and fitness habits of Asian people. Started with Location = India as I read that it’s most represented on MTurk. Will share my HIT response for India and let everyone know how that went..

    @Anyone know of similar services targetting SEA and China ?

  6. Andy

    Thank you for posting this guide for new turkers. Many times, the platform seems to be overwhelming for new comers. I liked how you posted screenshots of every step!

  7. Justin Wilcox

    Unfortunately I can’t see the HIT you posted (just get a blank screen). Want to try pasting the text of it here or a screenshot?

    Thanks,
    Justin

  8. Eric Harper

    Thanks for posting this detailed process breakdown. One of our customers at DashScribe used our service to automate the interviews after piloting the first few with a manual process. They are getting some really outstanding feedback from the process. DashScribe gives you the transcript + original audio for each response.

    We’d be happy to share a few free trials to DashScribe with readers of this blog if you’d also like to automate more of the process. The first three readers to contact us at DashScribe will get the trial.

  9. StartupPanda

    That’s a pretty awesome idea!
    “Lean Analytics” brought me here, and I’m really glad it did. I just set it up for the first time (Video really helped btw) and excited to see what happens.
    For what other purposes have you uses mturk so far? I’m trying to get in touch with college-students who hate giving presentations – But I’m unsure if I can find my focus group there?
    Clemens

    • Hakon

      Hey Clemens. Nice thing about college students is that they’re usually a pretty easy demographic to get a hold of. Instead of mech turk, why not stop by a few college campuses during busy hours and just start interviewing as many as you can? You can probably get a bunch of interviews for free, but I’ll bet you get a TON of interviews by bringing a bag full of snickers (the big ones – not the bite-sized ones). In-person interviews are usually the ideal source of data (though Skype/G+ is pretty good).

      • StartupPanda

        Hey Hakon! Thanks for the advice – here comes the tricky part tho: How do you get hold of them, when you’re not in the country? ;)
        Am currently in Germany but trying to reach out to US students in particular. Would absolutely love to get people on the Phone, but virtual Snickers just aint that delicious. University won’t hand out any contacts. Any thoughts?

        • Hakon

          Haha, that’s certainly a challenging situation! Very worthy of a discussion with fellow readers.

          If your customer base is out here is the US, you should consider moving out here. =D But since that’s a big leap, here are a few other ideas:

          (1) Schools often have community message boards and plenty of students read craigslist. You might try posting on those. I think with those channels, you might need a bigger incentive though, like $5. Or if you already have something implemented, you can offer some free usage. If no one is interested in free usage, there’s some (in)validation in that.
          (2) You can probably hire a student to collect surveys. There are obvious challenges related to remote management with this one.
          (3) I’m pretty sure there are services that will handle data collection from college students. Not sure what they are, but I remember seeing lots of people trying to get students to sign up for credit cards when I was in school. These services are probably more expensive, but remove some of the risk of remote management.

          These don’t feel like exceptional ideas, but maybe offer some decent food for thought. Hope to see some other readers help generate more ideas.

          • Justin Wilcox

            Big thanks Hakon for the suggestions, and Clemens for the question.

            Clemens, this is a terrific opportunity to recognize it will be as hard to sell your product to customers, as it will be to interview them. By that I mean, take the difficulty inherent in reaching your target audience as a data point when considering the viability of this venture. “If I can’t even talk to them, how the hell am I going to sell to them?”

            What websites do US college students use?
            What things do they search for?
            Are there Stumble Upon or Facebook ads that you could drive to a landing page?
            What happens if you replace virtual snickers with a $5 Starbucks or Amazon gift card?

            Btw, have you tried posting something on mTurk? I’d be surprised if you let a HIT sit for a day if you didn’t get a couple folks calling you…

            All the best,
            Justin

      • StartupPanda

        Hey guys!
        First things first, we’re working on moving to the US – check ;) However we don’t want to wait with the first customer interviews and gathering some qualitative feedback from our target group.
        MTurk worked wonders today, the $1,50 incentive was more than enough (fyi: the *67 appears not to work with Skype-numbers, thats why people couldn’t reach me initially).

        Justin, thanks again for the Blog Post and sharing that brilliant Mturk idea. Stumbleupon & FB will be next in line howevery right now I just want talk to people. I got 6 calls in the first 10 minutes and was on complete overload… The last 2 days I tried reaching out to people and got 2 [Target group] customers on the line – painful. Obviously channels were still sh*t. I didn’t expect that that today… Now it’s back to revising my questions.

        Any other great Posts or ideas for opening up new channels?

  10. Sedarius

    Hey Guys,
    Great post! Have a quick question. I’m in the Philippines and google voice doesn’t seem to work here. I tried setting everything up with Skype but the 67 option doesn’t work on skype. Seems like i’m between a rock and a hard place on this. Does anyone have a recommendation on an alternative i could use?

    Any feedback would be appreciated!

    • Justin Wilcox

      Are you sure *67 doesn’t work when someone calls your Skype number? I’ve never tried it but I would assume if I dial your *67 + [your Skype number] you won’t see my phone number. (Update: looks like one of our other comments had a problem with this as well, so wouldn’t be surprised if it didn’t work for you)

      If it doesn’t work, you can still give it a shot. Only potential downside is that someone on mTurk flags you as obtaining personally identifiable information via a HIT and takes it down.

      Let us know how it goes!

      Thanks,
      Justin

      • Sedarius

        Yeah, sure about the skype thing unfortunately- http://community.skype.com/t5/Windows-desktop-client/blocking-caller-ID/td-p/242424

        The caller will receive the following message when they try to make the call, “this caller will not accept calls from caller id blocked”

        Anyway, I did it without the 67 via skype and got some results. Will keep tweaking the messaging- maybe i can still get the volume even without blocking the number. Will keep going at it and let you & the community know how it goes.

        Thanks again

  11. Kevin Dewalt (@kevindewalt)

    Hey Justin!

    I’ve been recommending this approach to lots of startups but one recently followed-up with me to let me know that Amazon is now forbidding this practice. What is most bizarre is I cannot find this link (appears dead) or any mention of the policy on Amazon’s site.

    So maybe it is a new policy?

    Here is the email they received:

    ————————————————————————————————-
    ——=_NextPart_000_0082_01CE6DC3.45DAA390
    Content-Type: text/plain;
    boundary=”=======AVGMAIL-0459173C=======”;
    charset=”UTF-8″
    Content-Transfer-Encoding: quoted-printable

    Hello XXXX,

    I am sorry for any concerns over your HITs that violated our Mechanical =
    Turk policies.

    I have reviewed your HITs and your HITs requested Workers to call a =
    phone number. Asking Workers to call a number is not allowed, even if =
    you provide information how to block their number to keep their phone =
    numbers private.

    Here is a link to the Amazon Mechanical Turk policies:

    https://www.mturk.com/mturk/help?helpPage=3Dpolicies

    I hope this helps. Please let us know if you have any further questions.

    Thank you for your inquiry. Did I solve your problem?

    If yes, please click here:
    http://www.amazon.com/gp/help/survey?p=3DAYDM34L2GAHO7&k=3Dhy

    If no, please click here:
    http://www.amazon.com/gp/help/survey?p=3DAYDM34L2GAHO7&k=3Dhn

    Best regards,=20

    John W.
    Amazon Mechanical Turk
    ————————————————————————————-

    • Justin Wilcox

      Huge thanks for the heads up. I’ve never seen that before but it would be completely lame if that were the case.

      I ran a workshop last weekend where we did some 40 interviews of turkers so I don’t think mTurk has put in any automation to prevent this, but please let me know if you get anymore reports of this.

      I’ve got some ideas for a workaround if it’s necessary, but it’ll take a week or so to implement.

      BTW, did the person who emailed you get any clarification on what rule they broke?

      Thanks again. Hope you’re doing well my friend.

      Justin

      • Jack

        Hi Kevin and Justin,

        Do you know if Amazon is still forbidding this practice? I’d like to set up a few interview in the next week, but don’t want to waste the time trying if Amazon is filtering for this.

        Thanks,
        Jack

        • Justin Wilcox

          I was able to receive calls when I placed HITs in June, from Puerto Rico. I’ve also had a couple people comment lately that it’s working for them.

          I say, give it a try and let us know how it goes!

          Thanks,
          Justin

  12. Salman Merchant

    I’m currently using Amazon’s MTurk for customer discovery interviews and it’s been going relatively well. My product is fairly niche, but even then the response rate has been fantastic. I honed in on my requirements for users, and even with slightly narrow requirements I received roughly 40 calls in one hour. Needless to say I wasn’t able to take every single one, but I’ve finished approximately 30 calls over the course of a few days and some callers are even happy extending their calls to 25/30 minutes depending on how important the problem is to them.

    -Salman

  13. J Connolly

    Has anyone set up HIT scheduling? There are certain time slots when different members of our team are available, and would like to automate posting the correct HIT for that slot.

    If not, would you pay for this service? just kidding… sort of :)

  14. Adam

    Guys, this article is terrific and completely aligned with the challenges we face everyday at Elevatr in helping entrepreneurs discover their customers. Thanks for the invaluable insight and the prod to begin experimenting with mTurk.
    Best,
    Adam

  15. allyreevesAlly

    I’m posting this because I thought this was a great idea, am trying it, and I’m telling a recent “How it worked for me” story. (Nov.2013) I recently created and posted an MTurk “Hit” using these guidelines. I didn’t have any issues with getting a number posted or receiving calls through Google Voice. I’ve had my post up for about 3hrs. In the first hour received 6 calls and fielded 2. I’ve set up a Skype interview so perhaps my Hit is too complex to pull in the bucket loads of users that others have gotten. One out of the two calls I fielded had tech issues with joining so it was a no go.

    To summarize- in about 3 hours I’ve been able to gather only one interview. I revised some of my criteria for participation (casting the net wider) but I actually have a negligible stream of people calling in to begin the session- which priced at $10 for 30min.: looks pretty attractive (or so I assumed.) It costs me $4.00 each time I pull the Hit down and edit or adjust. Just something to keep in mind. For now, the stream of participation is slow. I may try this experiment again with something that has a lower bar to entry like a survey of 10 questions or casual review of some website, but it hasn’t worked out for me (so far) as a way to set up live interviews. I’m hopeful though- and going to try some adjustments to get this working. I will post an update later!

  16. Rafael

    This is an incredible tip, wow. Just started using this yesterday and have completed 15 great interviews, 14 of which were asking me to include them in our private beta launch when it’s ready, and even offering their email address (some without me even asking for it). This is amazing, possibly the best use of dollars I can imagine for an early stage startup. Talk about getting out of the building!

    One quick note: I’ve only gotten a couple of hangups, and some have called back. I always answer “Rafael speaking” and they say “Hi I’m calling about the mTurk phone survey”.

  17. dnnielsen

    FYI for people outside the US.

    MTurk has been made a lot harder (impossible) to use if you are outside the US. The reason is the requirement for an Amazon Payments account that cannot be created with an SSN or EID number.

    I’m going through forums for any work-around (based in Denmark), but if anyone here has a way to access Mturk as a requester from outside the US please let me know!

Leave a Reply