Price Testing: How to Test your Price

Imagine you come up with an app that eliminates the need for haircuts. It’s amazing, it’s innovative, it’s game-changing.

You want to milk this opportunity for everything it’s worth, but you’ve got some questions on how to do that:

Pricing. Since just about everybody can use it, will you make more money charging $0.99 for it, or since it’ll save a few people a lot of money, should you charge $99?

Target Segment. Is it best to target Supercuts customers who clearly hate getting haircuts so much they’re willing to walk around looking like shit?  Or should you target vain power brokers who would love to look perfect all the time?

Marketing. What’s the best marketing message:

Look your hairy best!


No more hair salon small talk.  You’re welcome.

Before Crowdtesting

If you wanted to know which combination of the above generated the most revenue, you had one choice:

  1. Build the product, and then a/b test it.

Sure, you could a/b test landing pages before you built the product to see which combination got you more email addresses, but you’re not going to be able to pay employees in email addresses.  And what if the landing page skews your results (e.g. what if Supercuts customers give out their email addresses more easily than CEOs)?

At the end of the day, we want to make our business decisions not counting visitors, Facebook likes, or email signups.  We want to base them on $.  Crowdtesting helps us do that.

What is Crowdtesting?

Crowdtesting is the combination of crowdfunding and a/b testing to validate business model hypotheses. But that’s a boring definition.  This is better:

Crowdtesting will tell you how much $ your idea is worth, before you build it.

Here’s the video version from the 2012 Lean Startup Conference:

Crowdtesting in action

Some screenshots from the “crowdtesting” experiment currently running for Bounce.

Crowdtesting Step-by-Step

Step #1 Come up with an idea for a product

While the principles apply to any opportunity, this particular technique is probably best suited for B2C opportunities that want to make money.  If you’re doing an ad supported, or “why does everyone keep asking us how we’ll make money?” play, I’m sorry, I don’t have answers for you.

Step #2 Figure out what you want to test

In my case, I wanted to know which of these would make more money:

  1. Selling Bounce directly to people who run late
  2. Offering Bounce as a gift people could give to others who run late

and then I wanted to know the optimal price point for the winner of the above.

Step #3 Fork Selfstarter

The Lockitron folks were kind enough to open source the crowdfunding platform they built – Selfstarter.  That makes it easy for us to launch our own crowdfunding project, and incorporate a/b testing.

Step #4 – Setup Amazon Payments

It can take a couple days to get your Amazon Payments account approved, which you’ll use to accept people’s credit card info, so be sure to start this process well in advance. Instructions are in the Selfstarter FAQ.

Note: I got a “sorry you don’t meet the requirements of a business account”-type email when I first signed up but when I told them what I was doing, I eventually got approval.

Step #5 – Setup A/B Testing

The Experiments functionality within Google Analytics is terrific for this kind of project.  I actually tried to use them but I ran into trouble running two Selfstarters at once.  I’m sure it’s a solvable problem, I just didn’t have time to resolve it.

Instead I simply ran one variation at the time (e.g. $10 instead of $5), and added a “variation” column to my orders table so I could count the number of pre-orders for each variation. Since the vast majority of my traffic was driven by an email list I segmented (more below), I was able to control exactly how many people saw each variation, and could compare apples-to-apples.

Step #6 – Launch

Yeah, baby!

Step #7 – Send traffic

Now you need to send traffic to your campaign. The great part about this step is that it’s fantastic practice to see how easy it’ll be to generate interest if you actually build the product.

As for how much you’ll need, I’m no statistician, but my thinking is, “just enough to run your experiment.”  So far my experiments have been fairly conclusive, so I’ve gotten by with a couple thousand visitors for each variation.

As far as how I generated my traffic.  I had posted a landing page collecting email addresses for people interested in Bounce for iPhone, before.  The majority of my traffic came from Hacker News, press, and emails to folks who signed up on Bounce’s landing page.

Step #8 – Analyze the results

Once the orders come in, the fun starts!


The metric I found most useful was $/visitor.  For example:

  • With a $5 price tag, 1.4% of visitors pre-order Bounce. That’s $.07/visitor.
  • With a $10 price tag, 1.7% of visitors pre-order Bounce. That’s $.17/visitor.

Sample size: 11,108

The combination of messaging and price points that generates the biggest $/visitor is the combination you want.

Important Notes:

There are a few non-obvious things you want to keep in mind when launching a crowdtesting project:

Crowdtesting isn’t perfect.  I’m clearly a fan, but crowdtesting is by no means perfect. It’s probably best suited for B2C plays that can charge money. We also don’t have data yet on how accurately it predicts sales once the product launches. You need to be very aware of where your traffic comes from, because that can skew your results as well. Some mitigations for these issues are below. Overall, crowdtesting is no silver bullet, but it’s clearly better than email sign-ups on a landing page or building a product no one wants.

Your goal ISN’T to raise lots of money.  Avoid trying to bring in tons of money during this campaign, that’s not the goal.  The goal is to test a hypothesis in a scenario that resembles market conditions when your product actually launches.  If you make getting lots of money a priority, you’re likely to try tactics that won’t be applicable when your product launches.  In fact, I eliminated the total $ raised from my crowdtest altogether – Bounce’s published “goal” was in terms of backers (5,000).

Don’t sell your story.  A lot of consumers support crowdfunding projects not because they want the rewards, but because they want to support the person raising the money.  You don’t want that.  You want people pre-ordering your product because they want it so badly, they’re willing to put down money early, to ensure it gets made.  Avoid talking much about yourself during the campaign and instead focus on the value prop – that’s what you’re trying to test.

Do no harm.  Crowdtesting only works because consumers trust the crowdfunding process.  If crowdtesters start charging credit cards before they build their products, don’t deliver on the promises they make, or otherwise mislead consumers, our customers will lose faith in the process and the tool will lose its value.

Set your goal high.  Remember, your primary objective is to test a hypothesis.  If you set a goal that’s low and you hit it, you’re obligated to build the product (see “Do no harm”), even if it turns out your $’s/visitor is much less than you need for this venture to be profitable.  Instead, set your goal to a high, but not outrageous level so that if you happen to hit it, you’d be happy to build the product.

Prime the pump.  You know how baristas never put an empty tip jar out?  It’s because people feel weird being the first to do something (e.g. first person to tip, first person to back a crowdfunding project, etc.).  Since we want to mimic real-world market conditions as quickly as possible, we don’t want the first visitors to our site to feel like “oh, well no one else has supported this, so it must be dumb.”  To avoid that, I started my backer count at 114 (with a goal of trying to reach 5,000).  It wasn’t so high that I felt consumers would feel mislead about the popularity of the product, but it also wasn’t an empty tip jar.

Don’t charge credit cards until the product launches.  Kickstarter lets you have customer money as soon as you reach your goal, but unless you have to have the money to pay for upfront costs, I’m a much bigger fan of not charging customers until you’ve actually launched the product.  This will build more trust with your customers, it gives you an incentive to deliver, and if you happen to miss a deadline, at least you earn interest on your customer’s cash.

Only the beginning

This was a first crack at crowdtesting. As we learn more we’ll continue to post information. If you’re interested, you can subscribe for updates via Email or RSS.

Please, if you have any questions or comments send them our way. Also, if you run your own crowdtest, we’d love to hear about it (and have you post about it)!

Customer Development Made Easy…

Crowdtesting: what do you think?

I’m giving a talk next week on crowdtesting and I’d really love your thoughts before I do. I’ll be giving it at the Lean Startup Conference so if you’re reading this, you know the target audience. It’s you.

Here’s a run through (note: it’s < 5 minutes):


  1. What questions do you have?
  2. What can I cut?  Aka, when did you tune out and flip to Facebook?
  3. Anything else?

You’re my customers so let me have it in the comments, or you can email me: [email protected].

See it Live

If you’d like to hear the final version live (oh, and some other no-name speakers Eric Ries, Steve Blank, Ash Maurya, etc.) I’ve got two suggestions:

  1. If you’ve got $900 and can/will be in SF, come in person! Scratch that, I’ve got your back.  Take 15% off (coupon code: speaker15)
  2. Watch for almost free at a local simulcast.  John Sechrest is running one in Seattle, and there are also tons of others running throughout the world.

Thanks again for your feedback!

Join the experiment – subscribe via Email or RSS for tips on how to run your own crowdtesting experiment which I’ll be writing up soon-ish.

Help Investors Believe you – show them Customer Quotes

When my mentor asked me for specific quotes from “potential customers” to demonstrate demand in an investor presentation the next day, I was up S*&% creek.

We don’t want investors “guessing” whether or not there’s demand for our products, but we don’t have time to walk through every single experiment and interview we’ve done during a pitch.  What we can do though, is share the excitement, demand, and desperation for our product via real customer quotes:

“This is Super needed in the marketplace. We will pay you for this” – Impact Investor*

“Our members will position themselves for better jobs if they have skills-based volunteering experience” – Career site for impact professionals*

“Existing programs are way too expensive, and my frustration in trying to find a legitimate opportunity could cause me to do nothing.” – Professional interested in skills-based travel volunteering*

“We can’t find the volunteers we need in order to grow our social enterprise.” – Potential hosting enterprise in Turkey*

*Names excluded for this blog post. Permission was specific to being used in presentation decks.
Make sure to respect your customers (and potential ones).

Unfortunately, through all my early customer interviews I had failed to collect quality quotes (until now).  I know I heard some great quotes, and probably scratched some down on paper, but I did a terrible job at documenting them, and I never asked for permission to use them.

I needed new customers to interview, FAST, and I needed enough quotes from them to ensure that I had a pool of HIGH QUALITY quotes. Sure, I could have gone back to people I interviewed, but I decided to use this as an opportunity to reach more people and do more validation.

And as my new venture, MovingWorlds, is working to effectively scale international, skills-based volunteering, I needed quotes about professionals who, in their own words, expressed needs that our solution addresses.

Using Quora to Find Potential Customers and Validate Our Problem and Solution Hypotheses

I was blown away how easy it was to find people with the EXACT challenges we were trying to solve. I used Quora in 2 ways

  1. Find people with relevant questions, and react to their posts
  2. Asked for specific quotes about my product

Finding people was easy. I started typing in “Skills-based volunteering” and before I could finish Quora was suggesting a myriad of topics that were addressing both parts of my marketplace – people looking for opportunities, and enterprises looking for volunteers (as well as companies looking to promote skills-based volunteering:

I was hoping for 2 things from finding people on Quora: first, to get feedback directly in Quora, and also to direct some traffic to a survey.

After I looked through Quora for a few minutes to better understand that types of questions and answers people were giving about this skills-based volunteering topic, I went to SurveyMonkey and setup a survey to capture the data I needed. I was then able to leave comments on Quora, and in exchange, ask for people to take few moments to give me feedback on Quora, and in my survey.

NOTE: Be respectful about searching for feedback here. People come to Quora for answers, not for spammy trawlers.

Here is an example of how I responded to a question so that I was still offering valuable feedback, but also asking for the opportunity to get a survey response, and maybe even a phone interview:

In addition to survey responses and comments on Quora, I also received personal messages as follow-up.

In a  relatively short period of time I was able get useful quotes that I needed to use as anecdotes and proof points in my pitch deck, find new customers to interview, better understand customer needs, and find matches for the next phase: concierge MVP (I’ll write more on concierge testing, later).

Using Mechanical Turk to Get Customer Validation Quotes

I didn’t have the time or resources to setup extensive customer interviews through Mechanical Turk like Justin wrote about on a previous post, so instead, I setup a quick survey to do the following:

  • Only capture professionals willing to volunteer their time
  • Make sure respondents were from our target demographic (from the US, had a professional skill, and attained a college degree)
  • Get quotes about challenges and opportunities for finding skills-based volunteering opportunities abroad

Step 1: Setup survey

I used SurveyMonkey and added qualifier questions to ensure respondents weren’t faking it. Surveyors got bounced with a “reject” code  if they didn’t have the right education level or live in the right place. “Trick” questions like ‘age’ and ‘location’ and ‘date of birth’ were included to cross-check data in case people were trying to spam the system (e.g. ask for age on one page, and then ask for date of birth on another, if they don’t match, its most likely worthless data)

Step 2: Setup HITs in Mechanical Turk.

I used the following project (screen shot below) to get people for the survey. I found that adding qualifiers in the ask produced better results. In this case, I had two main qualifiers:

  1. Must be a resident of the U.S.A. and have at least a Bachelor’s degree. I then confirmed these facts in the survey with specific questions (e.g. where do you live and what is your highest level of education achieved)
  2. Adding a line “Important: Be sure to read each question carefully. Randomly answering questions will be detected and will result in rejection of the HIT” also helped improve results

You can view the survey by following this link if you want to see the exact questions. Here is a screen shot of the project request on Mechanical Turk:

Here is the code for Mechanical Turk  project request that created the field above:

<h3>Answer a &lt;5 min survey about international travel and volunteering</h3>
<div class=”highlight-box”>

We are conducting a survey to better understand international travel and volunteering habits of people living in the US. We need to understand your opinion about professionals who want to travel and/or donate their expertise to make the world a better place. Select the link below to complete the survey.&nbsp; At the end of the survey, you will receive a code to paste into the box below to receive credit for taking our survey.
<p>In order to qualify for this survey, you <i><b>must </b></i>be from and/or living in the United States, <i><b>and </b></i>have achieved at least a Bachelor’s degree.</p>
<p>Survey link: <a href=”” target=”_blank”></a></p>

IMPORTANT: Be sure to read each question carefully. Randomly answering quesitons will be detected and will result in rejection of the HIT.
<p>Provide the survey code here:</p>
<p><input type=”text” size=”10″ id=”Q2age” name=”Q2age” /></p>
<p><style type=”text/css”>
.highlight-box { border:solid 0px #98BE10; background:#FCF9CE; color:#222222; padding:4px; text-align:left; font-size: smaller;}
<p>&nbsp;Watch the video on our home page to learn more:&nbsp;<a href=”” target=”_blank”></a></p>

In the survey, in addition to asking for people to provide a quote if they were interested in skills-based volunteering, I also asked for email addresses if I could follow-up to schedule an interview. I got nearly a 50% response rate – within 1 day – from qualified leads to schedule an interview:

I have not yet had the time to follow-up with all my survey responders, but I have gotten some useful feedback and a great list of qualified people to interview and get quotes from.

Extra Credit Tip:

Don’t make the same mistake I did and not have powerful quotes and anecdotes ready to go when potential partners are asking for them :)

For all of you early on in the interview process, make sure to collect quality quotes (and get permission to share them) from the people you interview. To help organize this, my team and I setup a Google Form that made it easy to capture quotes. We could either submit ourselves, or invite others to leave us a quote through a Google Form. Key fields: Quote, Date, First Name, Last Name, Permission to Publish (Yes, No, maybe – get confirmation for each use), Email, OK to follow-up for more quotes (Yes, No). Looks like this:

Join the experiment – want more ideas on validating your business model and sharing that validation with others? Subscribe via Email or RSS for our next update: How to Interview Complete Strangers.

Customer Development Made Easy…

Anybody that Knocks LinkedIn Doesn’t Know How to Use It

LinkedIn is a powerful, easy-to-use customer discovery tool that is effective at free, and awesome at premium. At MovingWorlds, we have found over 50% of our potential customers and partners all through LinkedIn by using these tricks.

4 Easy Ways to Find Potential Customers on LinkedIn

(To discover potential customers that we wrote about in our previous post, How We Found Customers to Start Developing, we were interested in finding talent management professionals at Fortune 500 organizations).

The Best Way to Find Groups of People on LinkedIn is to Start with Google

Seriously. Take 60 seconds and do a couple Google searches to see if lists already exist on LinkedIn. When I was looking for potential customers, I wanted to find only find people working at Fortune 500 companies. Since LinkedIn now uses Skills and Experience as a search feature, and this is listed as a category, it made this super easy for Google to take me directly to the most useful page. This is true for people as well.

Using Google to find relevant lists and people in LinkedIn
LinkedIn Fortune 500 Professionals

Use the Advanced Search in LinkedIn

I’m shocked how many people don’t know how to use advanced search in their email clients (read tips), Twitter (read tips), Google (read tips), and on LinkedIn. Take 10 minutes to learn it and save yourself a ton of time in the future.

There are a LOT of ways to used the advanced search filters… take time to play with different options until you find what works for you. Look at all the different categories you can segment by:

Types of Advanced Filters on LinkedIn

Since I was specifically looking for Talent Management professionals at Fortune 500, I wanted to narrow the big list down to people with decision making power. So I used filters to find people who meet these exact criteria:

  • anybody with ‘talent management’ in their profile
  • and was a Director, VP, or above
  • and was at a company of at least 250 employees
  • and was at a Fortune 500 company

There are lots of other filters, but this created a nice list for me. NOTE: some filters are for premium members only. I signed up for the ‘Business‘ account and prepaid for one year – it’s more than worth every penny. I’ve found potential investors, partners, reporters, and potential customers.

Talent Management professionals at Fortune 500 companies on LinkedIn

Participate in LinkedIn Groups to Find People you Should Know About, but Don’t

A lot of people that you should be talking to don’t show up in your results. Usually this is because they have a weird title, or for some reason, you’ve chosen a filter(s) that leave them out. So after finding people using the above methods, I always do another search in LinkedIn to find relevant groups that potential customers are likely to be a part of. Continuing the same example above, I look for people in groups related to talent management, HR, and society of human resources professionals. Once I found these groups, I looked for the most active people in them and tried to message them directly (sometimes you can InMail people for free if you are in the same group).

Using LinkedIn Groups to find potential customers

In addition to messaging top contributors, I’ll join a lot of these groups to get more insight. I look at conversations and respond to polls, provide connections, and comment on entries so that I am a valued group member. I’ll also ask the group if they have suggestions on who to talk to. You will be surprised at how eager people are to help, as long as you add value. As an example, we were travelling down to the Bay Area, so I asked the Bay Area Sustainability for tips on who to contact – we got 4 comments and GREAT leads. In addition to comments, a few members messaged me directly to help us setup additional meetings.

We got 4 great leads by asking for them on LinkedIn Groups

Use InMail, but Don’t Abuse It

Yes, InMails cost money. But they also get results. You get a certain amount with your premium membership, so try them out. Just be respectful of the community and of people. Don’t post garbage in groups, and don’t reach out to people until you have a compelling value proposition. For tips on how we wrote messages, see how we Engaged Potential Customers.

NOTE: Do not just try and ‘connect’ with people as a free way to message them.  Use InMail.

If you are blindly reaching out to targets you have identified, do not just click on the “connect” button. It barely ever works, and honestly, it’s abusing the system and ruins it for everybody else. If you don’t know somebody and want to message them, use the InMail option.

How to connect to people on LinkedIn - Use InMail
How to connect to people on LinkedIn – Use InMail

If you need help finding potential customers for any stage of your lean start-up process, LinkedIn can be a powerful tool, and is worth a 1-month trial and some experimenting. Like anything else at your startup, if it doesn’t work, then pivot.

In future posts, I’ll share tips on how we found potential customers using Twitter, Blogs, Press, and Google Groups. Subscribe via Email or RSS to get more tips!

How else do you use LinkedIn for Lean Startups Efforts?

Our 3 Steps for Engaging Customers

This is the 2nd post in a 3 part series about how MovingWorlds is developing its customer base before product launch. Read post 1: How we Found Customers to Start Developing.

There is an old saying “I hear and I forget, I see and I understand, I do and I remember.”

People don’t like hearing you talk about your idea, and usually they don’t even like seeing it. They like doing… touching, feeling, brainstorming.

Just liked vested employees are better employees, vested partners are better partners (yes, we call our customers partners because they are the most important thing to us). Spending time with potential customers is also a vital part of the customer validation and creation steps outlined in Steven Blank’s customer development model. So how do we engage potential customers before we have product?


We ask. We say things like “We’re new at this and we want to add value or get out of the way. If you can afford time to provide us advice to add more value to ecosystem, we would be grateful for your time”. People by far prefer to give advice then be sold to. Here is an example of a cold-call email and response using LinkedIn InMails:


We don’t have much, but we give what we have freely. This includes (but is not limited to), introductions to new connections (my linkedin network is your linkedin network), resource sharing (especially of new and relevant trends and articles), and our time (I was shocked by how many times I’ve been taken up on the offer and produced value by saying things like “I’m happy to jump on the phone or meet for coffee to brainstorm any of your challenges”).


We actually listen to the advice that we get, incorporate it, and then share it back with the people that gave it to us. People love to see their advice taken seriously, and a nice side – though unintended – benefit is that it makes them vested partners.

Examples here are just over email, but I do even more over the phone and in person.

Engaging customers is hard work that inevitably results in a lot of negative responses. But from every negative comes a valuable filtering affect: learning which potential customers are good, long-term partners, and which are sexy leads that do nothing more than distract you.

How do you engage potential customers?

In my next post on customer development, I’ll talk about how we’re making the ask: turning potential customers into paying customers and evangelists. Subscribe by email or RSS to get it directly to your inbox.