by Justin Wilcox -

Price Testing: How to Test your Price

[Updated] July 14, 2017

Imagine you come up with an app that eliminates the need for haircuts. It’s amazing, it’s innovative, it’s game-changing.

You want to milk this opportunity for everything it’s worth, but you’ve got some questions on how to do that:

Pricing. Since just about everybody can use it, will you make more money charging $0.99 for it, or since it’ll save a few people a lot of money, should you charge $99?

Target Segment. Is it best to target Supercuts customers who clearly hate getting haircuts so much they’re willing to walk around looking like shit?  Or should you target vain power brokers who would love to look perfect all the time?

Marketing. What’s the best marketing message:

Look your hairy best!

or

No more hair salon small talk.  You’re welcome.

Before Crowdtesting

If you wanted to know which combination of the above generated the most revenue, you had one choice:

  1. Build the product, and then a/b test it.

Sure, you could a/b test landing pages before you built the product to see which combination got you more email addresses, but you’re not going to be able to pay employees in email addresses.  And what if the landing page skews your results (e.g. what if Supercuts customers give out their email addresses more easily than CEOs)?

At the end of the day, we want to make our business decisions not counting visitors, Facebook likes, or email signups.  We want to base them on $.  Crowdtesting helps us do that.

What is Crowdtesting?

Crowdtesting is the combination of crowdfunding and a/b testing to validate business model hypotheses. But that’s a boring definition.  This is better:

Crowdtesting will tell you how much $ your idea is worth, before you build it.

Here’s the video version from the 2012 Lean Startup Conference:

Crowdtesting in action

Some screenshots from the “crowdtesting” experiment currently running for Bounce.

Crowdtesting Step-by-Step

Step #1 – Come up with an idea for a product

While the principles apply to any opportunity, this particular technique is probably best suited for B2C opportunities that want to make money.  If you’re doing an ad supported, or “why does everyone keep asking us how we’ll make money?” play, I’m sorry, I don’t have answers for you.

Step #2 – Figure out what you want to test

In my case, I wanted to know which of these would make more money:

  1. Selling Bounce directly to people who run late
  2. Offering Bounce as a gift people could give to others who run late

and then I wanted to know the optimal price point for the winner of the above.

Step #3 – Fork Selfstarter

The Lockitron folks were kind enough to open source the crowdfunding platform they built – Selfstarter.  That makes it easy for us to launch our own crowdfunding project, and incorporate a/b testing.

Step #4 – Setup Amazon Payments

It can take a couple days to get your Amazon Payments account approved, which you’ll use to accept people’s credit card info, so be sure to start this process well in advance. Instructions are in the Selfstarter FAQ.

Note: I got a “sorry you don’t meet the requirements of a business account”-type email when I first signed up but when I told them what I was doing, I eventually got approval.

Step #5 – Setup A/B Testing

The Experiments functionality within Google Analytics is terrific for this kind of project.  I actually tried to use them but I ran into trouble running two Selfstarters at once.  I’m sure it’s a solvable problem, I just didn’t have time to resolve it.

Instead I simply ran one variation at the time (e.g. $10 instead of $5), and added a “variation” column to my orders table so I could count the number of pre-orders for each variation. Since the vast majority of my traffic was driven by an email list I segmented (more below), I was able to control exactly how many people saw each variation, and could compare apples-to-apples.

Step #6 – Launch

Yeah, baby!

Step #7 – Send traffic

Now you need to send traffic to your campaign. The great part about this step is that it’s fantastic practice to see how easy it’ll be to generate interest if you actually build the product.

As for how much you’ll need, I’m no statistician, but my thinking is, “just enough to run your experiment.”  So far my experiments have been fairly conclusive, so I’ve gotten by with a couple thousand visitors for each variation.

As far as how I generated my traffic.  I had posted a landing page collecting email addresses for people interested in Bounce for iPhone, before.  The majority of my traffic came from Hacker News, press, and emails to folks who signed up on Bounce’s landing page.

Step #8 – Analyze the results

Once the orders come in, the fun starts!

conversion_rate_spreadsheet

The metric I found most useful was $/visitor.  For example:

  • With a $5 price tag, 1.4% of visitors pre-order Bounce. That’s $.07/visitor.
  • With a $10 price tag, 1.7% of visitors pre-order Bounce. That’s $.17/visitor.

Sample size: 11,108

The combination of messaging and price points that generates the biggest $/visitor is the combination you want.

Important Notes:

There are a few non-obvious things you want to keep in mind when launching a crowdtesting project:

Crowdtesting isn’t perfect.  I’m clearly a fan, but crowdtesting is by no means perfect. It’s probably best suited for B2C plays that can charge money. We also don’t have data yet on how accurately it predicts sales once the product launches. You need to be very aware of where your traffic comes from, because that can skew your results as well. Some mitigations for these issues are below. Overall, crowdtesting is no silver bullet, but it’s clearly better than email sign-ups on a landing page or building a product no one wants.

Your goal ISN’T to raise lots of money.  Avoid trying to bring in tons of money during this campaign, that’s not the goal.  The goal is to test a hypothesis in a scenario that resembles market conditions when your product actually launches.  If you make getting lots of money a priority, you’re likely to try tactics that won’t be applicable when your product launches.  In fact, I eliminated the total $ raised from my crowdtest altogether – Bounce’s published “goal” was in terms of backers (5,000).

Don’t sell your story.  A lot of consumers support crowdfunding projects not because they want the rewards, but because they want to support the person raising the money.  You don’t want that.  You want people pre-ordering your product because they want it so badly, they’re willing to put down money early, to ensure it gets made.  Avoid talking much about yourself during the campaign and instead focus on the value prop – that’s what you’re trying to test.

Do no harm.  Crowdtesting only works because consumers trust the crowdfunding process.  If crowdtesters start charging credit cards before they build their products, don’t deliver on the promises they make, or otherwise mislead consumers, our customers will lose faith in the process and the tool will lose its value.

Set your goal high.  Remember, your primary objective is to test a hypothesis.  If you set a goal that’s low and you hit it, you’re obligated to build the product (see “Do no harm”), even if it turns out your $’s/visitor is much less than you need for this venture to be profitable.  Instead, set your goal to a high, but not outrageous level so that if you happen to hit it, you’d be happy to build the product.

Prime the pump.  You know how baristas never put an empty tip jar out?  It’s because people feel weird being the first to do something (e.g. first person to tip, first person to back a crowdfunding project, etc.).  Since we want to mimic real-world market conditions as quickly as possible, we don’t want the first visitors to our site to feel like “oh, well no one else has supported this, so it must be dumb.”  To avoid that, I started my backer count at 114 (with a goal of trying to reach 5,000).  It wasn’t so high that I felt consumers would feel mislead about the popularity of the product, but it also wasn’t an empty tip jar.

Don’t charge credit cards until the product launches.  Kickstarter lets you have customer money as soon as you reach your goal, but unless you have to have the money to pay for upfront costs, I’m a much bigger fan of not charging customers until you’ve actually launched the product.  This will build more trust with your customers, it gives you an incentive to deliver, and if you happen to miss a deadline, at least you earn interest on your customer’s cash.

Only the beginning

This was a first crack at crowdtesting. As we learn more we’ll continue to post information. If you’re interested, you can subscribe for updates via Email or RSS.

Please, if you have any questions or comments send them our way. Also, if you run your own crowdtest, we’d love to hear about it (and have you post about it)!

Customer Development Made Easy…

by Justin Wilcox -

Crowdtesting: what do you think?

[Updated] May 20, 2014

I’m giving a talk next week on crowdtesting and I’d really love your thoughts before I do. I’ll be giving it at the Lean Startup Conference so if you’re reading this, you know the target audience. It’s you.

Here’s a run through (note: it’s < 5 minutes):

So…

  1. What questions do you have?
  2. What can I cut?  Aka, when did you tune out and flip to Facebook?
  3. Anything else?

You’re my customers so let me have it in the comments, or you can email me: [email protected].

See it Live

If you’d like to hear the final version live (oh, and some other no-name speakers Eric Ries, Steve Blank, Ash Maurya, etc.) I’ve got two suggestions:

  1. If you’ve got $900 and can/will be in SF, come in person! Scratch that, I’ve got your back.  Take 15% off (coupon code: speaker15)
  2. Watch for almost free at a local simulcast.  John Sechrest is running one in Seattle, and there are also tons of others running throughout the world.

Thanks again for your feedback!

Join the experiment – subscribe via Email or RSS for tips on how to run your own crowdtesting experiment which I’ll be writing up soon-ish.

by Justin Wilcox -

We Should Hang out Sometime

[Updated] May 20, 2014

Sorry for the lack of articles lately.  Pick any of the following as an excuse:

  1. Been busy building stuff
  2. Been busy testing stuff
  3. Been fighting with my parole officer over the definition of “recreational” drug use

That said, I’ll be at the following events and if you will be too, coffee is on me.  It would be my pleasure to treat for caffeine and chat about lean, customer dev, and anything else that suits you, since I haven’t been of any help here in the last couple months.

Lean Startup Machine Seattle: Nov 2nd – 4th

Lean startup machine logoAs I’ve mentioned before, LSM is an incredible experience.  I highly recommend it to anyone serious about starting a company.  The lessons you’ll learn over a weekend will literally save you months of time.

If that endorsement isn’t enough, here are two more reasons you should go:

  1. LSM will pay for you to attend a Startup Weekend.  That means you’ll learn how to test your business model assumptions during LSM, and use that knowledge to build a product customers actually want at a Startup Weekend – all for the price of one LSM ticket.  Double the learning and double the experience for the price of one LSM ticket.
  2. There are a handful of 50% coupon codes left.  If this link still works, it’ll get you admission to Lean Startup Machine and Startup Weekend, for $149.  I previously said LSM alone was worth $500, so to get LSM + SW (plus food & drink) for $149…it’s really…amazing.

I’ll be there helping teams test their hypotheses.  Really, there’s no reason you shouldn’t be there.

NWEN Entrepreneur University Seattle: Nov 9th

NWEN Entrepreneur UniversityNWEN’s got some big names speaking at their Entrepreneur University.  If you’ve ever heard of Cranium, Molly Moon’s Ice Cream, or Jawfish Games, you’ll know the keynote speakers.

You may have also heard of some of the lesser known speakers.  Mark and I will be doing a breakout session on “Interviewing Customers the Easy Way.”  It will be a live version of our How to Interview 100 Customers in 4 Hours with Mechanical Turk post.

We’ll pick a couple startups out of the audience, learn a little bit about their target customers, and then see if we can interview a couple of them during the session so everyone can see easy and powerful interviewing customers is.

Founder’s Institute Seattle: Nov 27th

Founders InstituteIf you’re a member of the Founder’s Institute Seattle session, I’ll see you for a talk on Product Development.  More specifically, we’ll be narrowing down the definition of MVP (which runs the gamut depending who you talk to).

If you’re not a member of FI, I think they’ll record the talk so if possible, I’ll post a link here afterwards.

And certainly not least…

Lean Startup Conference San Francisco: Dec 3rd – 4th

Lean Startup ConferenceI am nothing less than humbled and honored to be speaking at this year’s Lean Startup Conference with the likes of Steve Blank, Eric Reis, Ash Maurya, Patrick Vlaskovits, and on, and on… (oh, how clumsy of me. I just dropped all those names…)

My talk, “Crowdtesting: Using Crowdfunding to Test an MVP” will be only 5 minutes of what’s bound to be a day of innovative ways to build successful companies faster.  It feels like a combination of TED + Ignite dedicated to Lean Startups.  Come if you can.

Huge Thanks

To Ramphis Castro & Marcos Polanco for getting the speaking ball rolling, and to Mark Horoszowski for not only his encouragement, but his enthusiasm, coaching (and videography) during the Lean Startup Conference application process.

I’m Serious

With this much yapping, I’m may not be posting much over the next couple months, so like I say, please hit me up if you’re attending any of the above.  I’d love help however I can.

Join the experiment – I’ve got killer data for our next post (testing your startup’s name). Subscribe via Email or RSS to make sure you get it.

by markhoroszowski -

Help Investors Believe you – show them Customer Quotes

[Updated] October 12, 2014

When my mentor asked me for specific quotes from “potential customers” to demonstrate demand in an investor presentation the next day, I was up S*&% creek.

We don’t want investors “guessing” whether or not there’s demand for our products, but we don’t have time to walk through every single experiment and interview we’ve done during a pitch.  What we can do though, is share the excitement, demand, and desperation for our product via real customer quotes:

“This is Super needed in the marketplace. We will pay you for this” – Impact Investor*

“Our members will position themselves for better jobs if they have skills-based volunteering experience” – Career site for impact professionals*

“Existing programs are way too expensive, and my frustration in trying to find a legitimate opportunity could cause me to do nothing.” – Professional interested in skills-based travel volunteering*

“We can’t find the volunteers we need in order to grow our social enterprise.” – Potential hosting enterprise in Turkey*

*Names excluded for this blog post. Permission was specific to being used in presentation decks.
Make sure to respect your customers (and potential ones).

Unfortunately, through all my early customer interviews I had failed to collect quality quotes (until now).  I know I heard some great quotes, and probably scratched some down on paper, but I did a terrible job at documenting them, and I never asked for permission to use them.

I needed new customers to interview, FAST, and I needed enough quotes from them to ensure that I had a pool of HIGH QUALITY quotes. Sure, I could have gone back to people I interviewed, but I decided to use this as an opportunity to reach more people and do more validation.

And as my new venture, MovingWorlds, is working to effectively scale international, skills-based volunteering, I needed quotes about professionals who, in their own words, expressed needs that our solution addresses.

Using Quora to Find Potential Customers and Validate Our Problem and Solution Hypotheses

I was blown away how easy it was to find people with the EXACT challenges we were trying to solve. I used Quora in 2 ways

  1. Find people with relevant questions, and react to their posts
  2. Asked for specific quotes about my product

Finding people was easy. I started typing in “Skills-based volunteering” and before I could finish Quora was suggesting a myriad of topics that were addressing both parts of my marketplace – people looking for opportunities, and enterprises looking for volunteers (as well as companies looking to promote skills-based volunteering:

I was hoping for 2 things from finding people on Quora: first, to get feedback directly in Quora, and also to direct some traffic to a survey.

After I looked through Quora for a few minutes to better understand that types of questions and answers people were giving about this skills-based volunteering topic, I went to SurveyMonkey and setup a survey to capture the data I needed. I was then able to leave comments on Quora, and in exchange, ask for people to take few moments to give me feedback on Quora, and in my survey.

NOTE: Be respectful about searching for feedback here. People come to Quora for answers, not for spammy trawlers.

Here is an example of how I responded to a question so that I was still offering valuable feedback, but also asking for the opportunity to get a survey response, and maybe even a phone interview:

In addition to survey responses and comments on Quora, I also received personal messages as follow-up.

In a  relatively short period of time I was able get useful quotes that I needed to use as anecdotes and proof points in my pitch deck, find new customers to interview, better understand customer needs, and find matches for the next phase: concierge MVP (I’ll write more on concierge testing, later).

Using Mechanical Turk to Get Customer Validation Quotes

I didn’t have the time or resources to setup extensive customer interviews through Mechanical Turk like Justin wrote about on a previous post, so instead, I setup a quick survey to do the following:

  • Only capture professionals willing to volunteer their time
  • Make sure respondents were from our target demographic (from the US, had a professional skill, and attained a college degree)
  • Get quotes about challenges and opportunities for finding skills-based volunteering opportunities abroad

Step 1: Setup survey

I used SurveyMonkey and added qualifier questions to ensure respondents weren’t faking it. Surveyors got bounced with a “reject” code  if they didn’t have the right education level or live in the right place. “Trick” questions like ‘age’ and ‘location’ and ‘date of birth’ were included to cross-check data in case people were trying to spam the system (e.g. ask for age on one page, and then ask for date of birth on another, if they don’t match, its most likely worthless data)

Step 2: Setup HITs in Mechanical Turk.

I used the following project (screen shot below) to get people for the survey. I found that adding qualifiers in the ask produced better results. In this case, I had two main qualifiers:

  1. Must be a resident of the U.S.A. and have at least a Bachelor’s degree. I then confirmed these facts in the survey with specific questions (e.g. where do you live and what is your highest level of education achieved)
  2. Adding a line “Important: Be sure to read each question carefully. Randomly answering questions will be detected and will result in rejection of the HIT” also helped improve results

You can view the survey by following this link if you want to see the exact questions. Here is a screen shot of the project request on Mechanical Turk:

Here is the code for Mechanical Turk  project request that created the field above:

<h3>Answer a &lt;5 min survey about international travel and volunteering</h3>
<div class=”highlight-box”>

We are conducting a survey to better understand international travel and volunteering habits of people living in the US. We need to understand your opinion about professionals who want to travel and/or donate their expertise to make the world a better place. Select the link below to complete the survey.&nbsp; At the end of the survey, you will receive a code to paste into the box below to receive credit for taking our survey.
<p>In order to qualify for this survey, you <i><b>must </b></i>be from and/or living in the United States, <i><b>and </b></i>have achieved at least a Bachelor’s degree.</p>
</div>
<p>Survey link: <a href=”https://www.surveymonkey.com/s/VolunteeringAbroad” target=”_blank”>https://www.surveymonkey.com/s/VolunteeringAbroad</a></p>

IMPORTANT: Be sure to read each question carefully. Randomly answering quesitons will be detected and will result in rejection of the HIT.
<p>Provide the survey code here:</p>
<p><input type=”text” size=”10″ id=”Q2age” name=”Q2age” /></p>
<p><style type=”text/css”>
<!–
.highlight-box { border:solid 0px #98BE10; background:#FCF9CE; color:#222222; padding:4px; text-align:left; font-size: smaller;}
–>
</style></p>
<p>&nbsp;Watch the video on our home page to learn more:&nbsp;<a href=”http://www.movingworlds.org” target=”_blank”>http://www.movingworlds.org</a></p>

In the survey, in addition to asking for people to provide a quote if they were interested in skills-based volunteering, I also asked for email addresses if I could follow-up to schedule an interview. I got nearly a 50% response rate – within 1 day – from qualified leads to schedule an interview:

I have not yet had the time to follow-up with all my survey responders, but I have gotten some useful feedback and a great list of qualified people to interview and get quotes from.

Extra Credit Tip:

Don’t make the same mistake I did and not have powerful quotes and anecdotes ready to go when potential partners are asking for them :)

For all of you early on in the interview process, make sure to collect quality quotes (and get permission to share them) from the people you interview. To help organize this, my team and I setup a Google Form that made it easy to capture quotes. We could either submit ourselves, or invite others to leave us a quote through a Google Form. Key fields: Quote, Date, First Name, Last Name, Permission to Publish (Yes, No, maybe – get confirmation for each use), Email, OK to follow-up for more quotes (Yes, No). Looks like this:

Join the experiment – want more ideas on validating your business model and sharing that validation with others? Subscribe via Email or RSS for our next update: How to Interview Complete Strangers.

Customer Development Made Easy…