This has to be one of my favorite customer development tips: using Mechanical Turk to do customer interviews.
Nick Soman, Founder of LikeBright, and I discuss how he used Mechanical Turk to interview 100 customers in 4 hours, and how that got him into TechStars Seattle.
If you haven’t heard of Mechanical Turk (mTurk), here’s what you need to know:
- it’s a website were lots of people, do small amounts of work, for small amounts of money
- Who does work on mTurk? Although a bit stale, these stats indicate a fairly representative cross-section of the population.
- Why would they do work for such little $? Something to do during free time. It’s a replacement for solitaire, not a replacement for a job.
Since Nick and I talked, I’ve done a couple dozen interviews this way and the results have been fantastic. Nothing like “getting out of the building” at home, at midnight, with an ice cream sandwich in hand.
[Update – 5/29/2015]
There’s an easier and faster way to do everything listed here. Our friends at Customer Discovery Ninja have automated this entire workflow below and made it easy for folks outside the US to access Turkers.
Feel free to familiarize yourself with the steps below if you like, but the fastest way to talk to customers is going to be Customer Discovery Ninja.
Sample interview and instructions:
1. Write the interview script
To make sure I accurately test my hypothesis, and so I don’t forget any questions, I always write up my interview script ahead of time. Coming up with good interview questions is another post on its own, but here’s the intro I use:
Hi there, my name is Justin. Can I get your first name please? *
Great, thanks ______. Like I said, my name is Justin and I’m here in Seattle doing a little research. I’m happy to tell you about the project I’m working on, but so that I don’t accidently bias any of your answers, I’ll wait until we finish the survey. Is that alright?
Great. So that I don’t have to slow us down to take notes while we’re chatting, is it okay if I record this call? Great, here we go.
*Note: about half the time, people hang up after my first line. My assumption is that the hangups were expecting an automated phone survey, as opposed to a personal interview.
2. Get a (new?) Google Voice number
You’re going to post this number on mTurk, and strangers are going to call it so if you’re not down with that, create a new Google Voice account and grab a new number. I’ve been using my actual Google Voice number and haven’t had any negative affects so far, but do what makes sense for you.
Outside the US/Canada: Google Voice is restricted to US/Canada so if you’re not there, here’s a great blog post on how to borrow a Google Voice number. (Note: may be against Google’s TOS. Use your judgement.)
3. (Optional) Forward Incoming Google Voice calls to your Gmail
I like answering my interview calls via GTalk within Gmail for a couple reasons:
- The gmail interface makes it easy to record the call
- It’s easier for me to talk hands-free with my computer
Instructions on forwarding Google Voice calls to your gmail are here: http://support.google.com/chat/bin/answer.py?hl=en&answer=187936.
4. Create a HIT on mTurk
Create a new HIT on http://requester.mturk.com
Describe the HIT
Super Important:
You must turn off “Master Turkers.” Master Turkers are a pre-screened, and very small, subset of the MTurk population. We want any folks on MTurk to be able to contact us, as long as they meet our qualifications. Here’s how to do that:
Pick a Price
Write up the HIT
Feel free to copy and paste this HTML:
<p><span style=”font-family: Arial;”>If you are a parent who picks your kids at day care at least once/week, please call us for a 5-10 minute phone survey.</span></p>
<p><span style=”font-family: Arial;”>Please dial the following number:</span></p>
<ul>
<li><span style=”font-family: Arial;”>*67 [your google voice number]</span></li>
</ul>
<p><span style=”font-family: Arial;”>Note: dialing *67 before the actual phone number will protect the privacy of your phone number. </span><span style=”font-family: Arial;”>If you reach voicemail again, please wait 10 minutes.</span></p>
<p><span style=”font-family: Arial;”> <b>Required after Calling</b> - after we finish the survey, we will give you a password to confirm you successfully completed it. Please enter it below:</span></p>
<p><span style=”font-family: Arial;”><b>Password:</b> <textarea rows=”1″ cols=”80″ name=”answer”></textarea></span></p>
<p><span style=”font-family: Arial;”>Thank you very much! We really appreciate your help! </span></p>
*Note: the “password” is a word you tell your interviewee to type in once the interview is complete. You’ll see what they type in before you approve the HIT (i.e. pay them) so you can ensure only the people who successfully completed the interview get paid.
Outside the US – mTurk needs a US address to sign up. I’ve heard of people using any US address to create their accounts if they’re outside the US. Again, may be against TOS. Use your judgement.
5. Publish the HITs
6. Get an ice cream sandwich
Now it’s time to wait for your first call. When it comes, answer it with your intro and once you have permission, record the call so you can share it with your team or review it later.
 |
 |
When you’re done, you can review the calls on Google Voice
Video Walkthrough
Using mTurk for the first time can be a bit confusing, so here’s a video walkthrough of the mTurk steps above:
Conclusion
If your target customers are on mTurk, which is likely if you’re building a B2C company, this is an awesome way to get lots of interviews in a short amount of time.
Thoughts, questions, or other suggestions for quick customer interviews? We’d love to hear ’em.
Big thanks again to Nick Soman from LikeBright for sharing this tip, Andy Sack for making him do it, and Dan Shapiro for teaching all of us the power of mTurk for customer development.
Outside the US? Or Don’t want to Mess with Mechanical Turk?
There’s an easier and faster way to do everything listed above. The guys at Customer Discovery Ninja have automated this entire workflow, which also makes it easy for folks outside the US to access US Turkers.
What’s Next?
Join the experiment – follow along via Email or RSS for updates on our next post: Prioritizing Customer Segments with Excel.
Justin- thank you so much for this. I just did it. It’s like summoning customers from the internet! Magic!
Aww, Customer Discovery Ninja appears to be offline/gone.
My experience as well trying to connect with customer Discovery Ninja – any alternative options and/or recommendations?
Justin, A great post + I am finding the focus framework book v valuable
I live out of the USA and thought you might like to know that the ” customer discovery ninja” site has closed – you do not happen to know where they relocated to?
Thanks
Hi Justin,
Awesome post, we’ve used this method several times over the past few months to conduct tons of interviews. One thing we’ve struggled with is accessing our beachhead market (or at least, what we think our beachhead market is) to do interviews. We’ve been trying to reach out to the ‘latte + iphone + big city’ crowd, and they tend not to be on mturk, craigslist, etc. and generally aren’t motivated by a few bucks for their time. Any thoughts on how we might better reach this crowd? Is this a case of if you can’t readily reach a customer group, then they shouldn’t be your beachhead market?
Thanks!
Great question Brian, thanks for asking.
A couple things pop-out at me:
I’m concerned about the broadness of your segment. No one self-identifies as being part of the “latte + iphone + big city crowd”, so it’s going to be difficult for you to not only find them (as you’ve seen) and it might be difficult to have a candid, open conversation about the problems they’re actively taking action to solve.
You will probably be better served by narrowing your customer segment significantly to start with, identifying where that group of customers searches for answers to their problems, and trying to find interviews there.
I love this part of your question, “Is this a case of if you can’t readily reach a customer group, then they shouldn’t be your beachhead market?” because my gut is it a question a lot of people have.
I wonder, in this analogy, is the “beachhead” you’ve defined with your customer segment Normandy…or all of France? It sounds like the latter at this point. Ultimately, your product will free the entire world, but you have to start with a single, narrowly defined landing-point to get any traction. As I mention above, seriously consider narrowing even further and see if that points you to better places to find interviews.
Fwiw, there are several exercises in the FOCUS Framework that walk you through exactly how to solve this problem (in particular “Customer Storming & Role-izing”, “50 Ways to talk to 5 People”, “Adoption Behavior Curve” and “Interview Channel Costing”). Definitely take a look at them, I think they’ll help quite a bit.
All the best,
Justin
Same thing here. I can’t use Mturk as I’m not in the US. I’ll try to find a way with Facebook ads… It will cost me more but I want to do fewer but longer interviews… like 20 minutes and I can target audience this way… My target doesn’t have a lot of free time…
Hey Laura, definitely check out Customer Discovery Ninja. Their workflow is based on what we outline in the blog and should be accessible to people outside the US: http://www.customerdiscovery.ninja/
FYI for people outside the US.
MTurk has been made a lot harder (impossible) to use if you are outside the US. The reason is the requirement for an Amazon Payments account that cannot be created with an SSN or EID number.
I’m going through forums for any work-around (based in Denmark), but if anyone here has a way to access Mturk as a requester from outside the US please let me know!
Hey dnnielsen,
I’m working on a tool that allows you to use MTurk from outside of the US. Could you grab a time slot here – bit.ly/CDNinfo – so we can talk? Same goes to anyone else reading this :) I’ll be happy to tell you all about it on the call but a little customer discovery beforehand will go a long way!
– Steven
@dnnielsen, I highly recommend chatting with Steven. We’re working on a way to give you a hand.
Justin
Hi Justin, first off – have just come across your videos on You Tube and Customer Dev Labs through there and I think they offer amazing value for the self starter – for FREE :-) Thank You so much. I wish you all the very best with your FOCUS program. I’m just starting out as an Entrepreneur but being based out of the UK am unable to use the MTurk method for potential customer discovery interviews.. plus as my product is B2C meant for an international customer base, am thinking about how to get interviews from folks from other parts of the world (the USA will do for a start though). I clicked on that link in Steven’s comment but it doesn’t look like there are any schedules there – plus an international phone call. Any suggestions or alternatives for customer interviews? I have written to “middlemen” websites like WorkFusion, Alegion etc asking if they would help post my HITS on MTurk – replies awaited.
Hey Karthik, like I mentioned to Laura, definitely check out Customer Discovery Ninja. Their workflow is based on what we outline in the blog and should be accessible to people outside the US: http://www.customerdiscovery.ninja/
This is an incredible tip, wow. Just started using this yesterday and have completed 15 great interviews, 14 of which were asking me to include them in our private beta launch when it’s ready, and even offering their email address (some without me even asking for it). This is amazing, possibly the best use of dollars I can imagine for an early stage startup. Talk about getting out of the building!
One quick note: I’ve only gotten a couple of hangups, and some have called back. I always answer “Rafael speaking” and they say “Hi I’m calling about the mTurk phone survey”.
I followed this advice to be able to record Google Hangout since the call recording features of Google Voice didn’t seem to be working anymore. http://www.badbamboo.com/index.php/2011/08/29/recording-a-google-hangout-for-a-podcast-with-soundflower-linein-and-garageband/
Posted a HIIT directly after reading this article and an hour later have already talked to 6 potential customers. Really quality, engaging conversations. Mad props for this article and tip.
That is a great post! Thank you for all the hacks :)
I’m posting this because I thought this was a great idea, am trying it, and I’m telling a recent “How it worked for me” story. (Nov.2013) I recently created and posted an MTurk “Hit” using these guidelines. I didn’t have any issues with getting a number posted or receiving calls through Google Voice. I’ve had my post up for about 3hrs. In the first hour received 6 calls and fielded 2. I’ve set up a Skype interview so perhaps my Hit is too complex to pull in the bucket loads of users that others have gotten. One out of the two calls I fielded had tech issues with joining so it was a no go.
To summarize- in about 3 hours I’ve been able to gather only one interview. I revised some of my criteria for participation (casting the net wider) but I actually have a negligible stream of people calling in to begin the session- which priced at $10 for 30min.: looks pretty attractive (or so I assumed.) It costs me $4.00 each time I pull the Hit down and edit or adjust. Just something to keep in mind. For now, the stream of participation is slow. I may try this experiment again with something that has a lower bar to entry like a survey of 10 questions or casual review of some website, but it hasn’t worked out for me (so far) as a way to set up live interviews. I’m hopeful though- and going to try some adjustments to get this working. I will post an update later!
Wonderful post Justin! The step-by-step guide helped me tremendously when I first started using MTurk.
Guys, this article is terrific and completely aligned with the challenges we face everyday at Elevatr in helping entrepreneurs discover their customers. Thanks for the invaluable insight and the prod to begin experimenting with mTurk.
Best,
Adam
Has anyone set up HIT scheduling? There are certain time slots when different members of our team are available, and would like to automate posting the correct HIT for that slot.
If not, would you pay for this service? just kidding… sort of :)
I’m currently using Amazon’s MTurk for customer discovery interviews and it’s been going relatively well. My product is fairly niche, but even then the response rate has been fantastic. I honed in on my requirements for users, and even with slightly narrow requirements I received roughly 40 calls in one hour. Needless to say I wasn’t able to take every single one, but I’ve finished approximately 30 calls over the course of a few days and some callers are even happy extending their calls to 25/30 minutes depending on how important the problem is to them.
-Salman
Hey Justin!
I’ve been recommending this approach to lots of startups but one recently followed-up with me to let me know that Amazon is now forbidding this practice. What is most bizarre is I cannot find this link (appears dead) or any mention of the policy on Amazon’s site.
So maybe it is a new policy?
Here is the email they received:
————————————————————————————————-
——=_NextPart_000_0082_01CE6DC3.45DAA390
Content-Type: text/plain;
boundary=”=======AVGMAIL-0459173C=======”;
charset=”UTF-8″
Content-Transfer-Encoding: quoted-printable
Hello XXXX,
I am sorry for any concerns over your HITs that violated our Mechanical =
Turk policies.
I have reviewed your HITs and your HITs requested Workers to call a =
phone number. Asking Workers to call a number is not allowed, even if =
you provide information how to block their number to keep their phone =
numbers private.
Here is a link to the Amazon Mechanical Turk policies:
https://www.mturk.com/mturk/help?helpPage=3Dpolicies
I hope this helps. Please let us know if you have any further questions.
Thank you for your inquiry. Did I solve your problem?
If yes, please click here:
http://www.amazon.com/gp/help/survey?p=3DAYDM34L2GAHO7&k=3Dhy
If no, please click here:
http://www.amazon.com/gp/help/survey?p=3DAYDM34L2GAHO7&k=3Dhn
Best regards,=20
John W.
Amazon Mechanical Turk
————————————————————————————-
Huge thanks for the heads up. I’ve never seen that before but it would be completely lame if that were the case.
I ran a workshop last weekend where we did some 40 interviews of turkers so I don’t think mTurk has put in any automation to prevent this, but please let me know if you get anymore reports of this.
I’ve got some ideas for a workaround if it’s necessary, but it’ll take a week or so to implement.
BTW, did the person who emailed you get any clarification on what rule they broke?
Thanks again. Hope you’re doing well my friend.
Justin
Hi Kevin and Justin,
Do you know if Amazon is still forbidding this practice? I’d like to set up a few interview in the next week, but don’t want to waste the time trying if Amazon is filtering for this.
Thanks,
Jack
I was able to receive calls when I placed HITs in June, from Puerto Rico. I’ve also had a couple people comment lately that it’s working for them.
I say, give it a try and let us know how it goes!
Thanks,
Justin
Hey Guys,
Great post! Have a quick question. I’m in the Philippines and google voice doesn’t seem to work here. I tried setting everything up with Skype but the 67 option doesn’t work on skype. Seems like i’m between a rock and a hard place on this. Does anyone have a recommendation on an alternative i could use?
Any feedback would be appreciated!
Are you sure *67 doesn’t work when someone calls your Skype number? I’ve never tried it but I would assume if I dial your *67 + [your Skype number] you won’t see my phone number. (Update: looks like one of our other comments had a problem with this as well, so wouldn’t be surprised if it didn’t work for you)
If it doesn’t work, you can still give it a shot. Only potential downside is that someone on mTurk flags you as obtaining personally identifiable information via a HIT and takes it down.
Let us know how it goes!
Thanks,
Justin
Yeah, sure about the skype thing unfortunately- http://community.skype.com/t5/Windows-desktop-client/blocking-caller-ID/td-p/242424
The caller will receive the following message when they try to make the call, “this caller will not accept calls from caller id blocked”
Anyway, I did it without the 67 via skype and got some results. Will keep tweaking the messaging- maybe i can still get the volume even without blocking the number. Will keep going at it and let you & the community know how it goes.
Thanks again
That’s a pretty awesome idea!
“Lean Analytics” brought me here, and I’m really glad it did. I just set it up for the first time (Video really helped btw) and excited to see what happens.
For what other purposes have you uses mturk so far? I’m trying to get in touch with college-students who hate giving presentations – But I’m unsure if I can find my focus group there?
Clemens
Hey Clemens. Nice thing about college students is that they’re usually a pretty easy demographic to get a hold of. Instead of mech turk, why not stop by a few college campuses during busy hours and just start interviewing as many as you can? You can probably get a bunch of interviews for free, but I’ll bet you get a TON of interviews by bringing a bag full of snickers (the big ones – not the bite-sized ones). In-person interviews are usually the ideal source of data (though Skype/G+ is pretty good).
Hey Hakon! Thanks for the advice – here comes the tricky part tho: How do you get hold of them, when you’re not in the country? ;)
Am currently in Germany but trying to reach out to US students in particular. Would absolutely love to get people on the Phone, but virtual Snickers just aint that delicious. University won’t hand out any contacts. Any thoughts?
Haha, that’s certainly a challenging situation! Very worthy of a discussion with fellow readers.
If your customer base is out here is the US, you should consider moving out here. =D But since that’s a big leap, here are a few other ideas:
(1) Schools often have community message boards and plenty of students read craigslist. You might try posting on those. I think with those channels, you might need a bigger incentive though, like $5. Or if you already have something implemented, you can offer some free usage. If no one is interested in free usage, there’s some (in)validation in that.
(2) You can probably hire a student to collect surveys. There are obvious challenges related to remote management with this one.
(3) I’m pretty sure there are services that will handle data collection from college students. Not sure what they are, but I remember seeing lots of people trying to get students to sign up for credit cards when I was in school. These services are probably more expensive, but remove some of the risk of remote management.
These don’t feel like exceptional ideas, but maybe offer some decent food for thought. Hope to see some other readers help generate more ideas.
Big thanks Hakon for the suggestions, and Clemens for the question.
Clemens, this is a terrific opportunity to recognize it will be as hard to sell your product to customers, as it will be to interview them. By that I mean, take the difficulty inherent in reaching your target audience as a data point when considering the viability of this venture. “If I can’t even talk to them, how the hell am I going to sell to them?”
What websites do US college students use?
What things do they search for?
Are there Stumble Upon or Facebook ads that you could drive to a landing page?
What happens if you replace virtual snickers with a $5 Starbucks or Amazon gift card?
Btw, have you tried posting something on mTurk? I’d be surprised if you let a HIT sit for a day if you didn’t get a couple folks calling you…
All the best,
Justin
Hey guys!
First things first, we’re working on moving to the US – check ;) However we don’t want to wait with the first customer interviews and gathering some qualitative feedback from our target group.
MTurk worked wonders today, the $1,50 incentive was more than enough (fyi: the *67 appears not to work with Skype-numbers, thats why people couldn’t reach me initially).
Justin, thanks again for the Blog Post and sharing that brilliant Mturk idea. Stumbleupon & FB will be next in line howevery right now I just want talk to people. I got 6 calls in the first 10 minutes and was on complete overload… The last 2 days I tried reaching out to people and got 2 [Target group] customers on the line – painful. Obviously channels were still sh*t. I didn’t expect that that today… Now it’s back to revising my questions.
Any other great Posts or ideas for opening up new channels?
This is awesome! #customerdevelopment
I’m going to start doing this immediately, can’t believe I didn’t think of this!
Also, quick tip. To be able to record, you have to update your settings in Google Voice first, otherwise the record button doesn’t actually do anything! http://support.google.com/voice/bin/answer.py?hl=en&answer=115082
Also, just to confirm, my first HITS didn’t get any requests, but I disabled “Master Workers” and increased the price to $1.50 (per 10 minutes) and works great!
Hi,
I was wondering if I can use google voice here in India.Please help with any alternatives.
Thanks for posting this detailed process breakdown. One of our customers at DashScribe used our service to automate the interviews after piloting the first few with a manual process. They are getting some really outstanding feedback from the process. DashScribe gives you the transcript + original audio for each response.
We’d be happy to share a few free trials to DashScribe with readers of this blog if you’d also like to automate more of the process. The first three readers to contact us at DashScribe will get the trial.
Unfortunately I can’t see the HIT you posted (just get a blank screen). Want to try pasting the text of it here or a screenshot?
Thanks,
Justin
Justin! Great post.
No one is calling me though?
I suspect 1 of 3 problems:
1. Pricing too low (I’ve tried increasing)
2. HIT too complex, maybe I gave too much detail
3. Time of day. Is it better to do this at night when Turkers aren’t at work/school?
See my HIT here: https://requester.mturk.com/batches/1051199
Thanks,
Sam
Thanks, Justin. What do you recommend I do if I want to survey small business operators?
Hey Magdy! Heck out previous pots on finding people using Quora and LinkedIn – that’s how I found people from small businesses.
Even though Quora post references “investors”, that tactics used to find customers is relevant regardless of your purpose: https://customerdevlabs.com/2012/08/28/help-investors-believe-you-show-them-customer-quotes/
Here the post on LinkedIn: https://customerdevlabs.com/2012/06/24/anybody-that-knocks-linkedin-does-not-know-how-to-use-it/
Good luck – and let us know how it goes!
Thank you very much, Mark. Very useful. I appreciate it.
Super useful guide. Thanks so much! We just started using this approach for customer interviews and it’s really helpful!
Thank you for posting this guide for new turkers. Many times, the platform seems to be overwhelming for new comers. I liked how you posted screenshots of every step!
Hey Justin, great post! It’s because of this that I’ve switched directions from considering professional user recruitment (read: expensive) to MTurk instead.
I’ve just created a HIT to survey online and fitness habits of Asian people. Started with Location = India as I read that it’s most represented on MTurk. Will share my HIT response for India and let everyone know how that went..
@Anyone know of similar services targetting SEA and China ?
Let me first say that I am very impressed with this post great work. I would also like to throw out a little bit of information from the workers (turkers) end of the spectrum. There are a few forums out there dedicated to mtruk and working in the cloud that can answer any further questions people may have about placing work on mechanical turk. If you would like some worker insight either as a requester or a worker I invite you to join us at CloudMeBaby.com. We are always willing to help out and I am sure you will find the wealth of information both refreshing and informative. Again, thank you for the very informative post.
Wow, hell of a post! Thanks for the great tips and applicable content! This is a fantastic addition to the utility belt. =)
I really liked the idea of doing this as a group and getting some friends involved. Nice way to inject some new perspective into your conversations about what you’re working on.
Google Consumer Surveys is a really fast and affordable way to interview 1000s of users/customers that are representative of the US population .
@pmmucsd – GCS is definitely a useful resource, but you are much more limited by the types of questions you can ask, and the price points since it dictates not only price per question, but also batch size. For scrappier startups, no surprise if GSC will be too expensive, and too limited. And you can’t collect respondent data to do actual interviews, which in the case here with Bounce and LikeBright, were essential to understanding customer needs.
Huge fan of this post. Going to push this out to LSM Singapore attendees next week :)
Great post, and great interview. I’ve been using mTurk for surveys, finding emails, and testing features – this takes it to a new level that I’m excited to try.
Thanks a TON for posting the tips.