Imagine you come up with an app that eliminates the need for haircuts. It’s amazing, it’s innovative, it’s game-changing.
You want to milk this opportunity for everything it’s worth, but you’ve got some questions on how to do that:
Pricing. Since just about everybody can use it, will you make more money charging $0.99 for it, or since it’ll save a few people a lot of money, should you charge $99?
Target Segment. Is it best to target Supercuts customers who clearly hate getting haircuts so much they’re willing to walk around looking like shit? Or should you target vain power brokers who would love to look perfect all the time?
Marketing. What’s the best marketing message:
Look your hairy best!
or
No more hair salon small talk. You’re welcome.
Before Crowdtesting
If you wanted to know which combination of the above generated the most revenue, you had one choice:
- Build the product, and then a/b test it.
Sure, you could a/b test landing pages before you built the product to see which combination got you more email addresses, but you’re not going to be able to pay employees in email addresses. And what if the landing page skews your results (e.g. what if Supercuts customers give out their email addresses more easily than CEOs)?
At the end of the day, we want to make our business decisions not counting visitors, Facebook likes, or email signups. We want to base them on $. Crowdtesting helps us do that.
What is Crowdtesting?
Crowdtesting is the combination of crowdfunding and a/b testing to validate business model hypotheses. But that’s a boring definition. This is better:
Crowdtesting will tell you how much $ your idea is worth, before you build it.
Here’s the video version from the 2012 Lean Startup Conference:
Crowdtesting in action
Some screenshots from the “crowdtesting” experiment currently running for Bounce.
Crowdtesting Step-by-Step
Step #1 – Come up with an idea for a product
While the principles apply to any opportunity, this particular technique is probably best suited for B2C opportunities that want to make money. If you’re doing an ad supported, or “why does everyone keep asking us how we’ll make money?” play, I’m sorry, I don’t have answers for you.
Step #2 – Figure out what you want to test
In my case, I wanted to know which of these would make more money:
- Selling Bounce directly to people who run late
- Offering Bounce as a gift people could give to others who run late
and then I wanted to know the optimal price point for the winner of the above.
Step #3 – Fork Selfstarter
The Lockitron folks were kind enough to open source the crowdfunding platform they built – Selfstarter. That makes it easy for us to launch our own crowdfunding project, and incorporate a/b testing.
Step #4 – Setup Amazon Payments
It can take a couple days to get your Amazon Payments account approved, which you’ll use to accept people’s credit card info, so be sure to start this process well in advance. Instructions are in the Selfstarter FAQ.
Note: I got a “sorry you don’t meet the requirements of a business account”-type email when I first signed up but when I told them what I was doing, I eventually got approval.
Step #5 – Setup A/B Testing
The Experiments functionality within Google Analytics is terrific for this kind of project. I actually tried to use them but I ran into trouble running two Selfstarters at once. I’m sure it’s a solvable problem, I just didn’t have time to resolve it.
Instead I simply ran one variation at the time (e.g. $10 instead of $5), and added a “variation” column to my orders table so I could count the number of pre-orders for each variation. Since the vast majority of my traffic was driven by an email list I segmented (more below), I was able to control exactly how many people saw each variation, and could compare apples-to-apples.
Step #6 – Launch
Yeah, baby!
Step #7 – Send traffic
Now you need to send traffic to your campaign. The great part about this step is that it’s fantastic practice to see how easy it’ll be to generate interest if you actually build the product.
As for how much you’ll need, I’m no statistician, but my thinking is, “just enough to run your experiment.” So far my experiments have been fairly conclusive, so I’ve gotten by with a couple thousand visitors for each variation.
As far as how I generated my traffic. I had posted a landing page collecting email addresses for people interested in Bounce for iPhone, before. The majority of my traffic came from Hacker News, press, and emails to folks who signed up on Bounce’s landing page.
Step #8 – Analyze the results
Once the orders come in, the fun starts!
The metric I found most useful was $/visitor. For example:
- With a $5 price tag, 1.4% of visitors pre-order Bounce. That’s $.07/visitor.
- With a $10 price tag, 1.7% of visitors pre-order Bounce. That’s $.17/visitor.
Sample size: 11,108
The combination of messaging and price points that generates the biggest $/visitor is the combination you want.
Important Notes:
There are a few non-obvious things you want to keep in mind when launching a crowdtesting project:
Crowdtesting isn’t perfect. I’m clearly a fan, but crowdtesting is by no means perfect. It’s probably best suited for B2C plays that can charge money. We also don’t have data yet on how accurately it predicts sales once the product launches. You need to be very aware of where your traffic comes from, because that can skew your results as well. Some mitigations for these issues are below. Overall, crowdtesting is no silver bullet, but it’s clearly better than email sign-ups on a landing page or building a product no one wants.
Your goal ISN’T to raise lots of money. Avoid trying to bring in tons of money during this campaign, that’s not the goal. The goal is to test a hypothesis in a scenario that resembles market conditions when your product actually launches. If you make getting lots of money a priority, you’re likely to try tactics that won’t be applicable when your product launches. In fact, I eliminated the total $ raised from my crowdtest altogether – Bounce’s published “goal” was in terms of backers (5,000).
Don’t sell your story. A lot of consumers support crowdfunding projects not because they want the rewards, but because they want to support the person raising the money. You don’t want that. You want people pre-ordering your product because they want it so badly, they’re willing to put down money early, to ensure it gets made. Avoid talking much about yourself during the campaign and instead focus on the value prop – that’s what you’re trying to test.
Do no harm. Crowdtesting only works because consumers trust the crowdfunding process. If crowdtesters start charging credit cards before they build their products, don’t deliver on the promises they make, or otherwise mislead consumers, our customers will lose faith in the process and the tool will lose its value.
Set your goal high. Remember, your primary objective is to test a hypothesis. If you set a goal that’s low and you hit it, you’re obligated to build the product (see “Do no harm”), even if it turns out your $’s/visitor is much less than you need for this venture to be profitable. Instead, set your goal to a high, but not outrageous level so that if you happen to hit it, you’d be happy to build the product.
Prime the pump. You know how baristas never put an empty tip jar out? It’s because people feel weird being the first to do something (e.g. first person to tip, first person to back a crowdfunding project, etc.). Since we want to mimic real-world market conditions as quickly as possible, we don’t want the first visitors to our site to feel like “oh, well no one else has supported this, so it must be dumb.” To avoid that, I started my backer count at 114 (with a goal of trying to reach 5,000). It wasn’t so high that I felt consumers would feel mislead about the popularity of the product, but it also wasn’t an empty tip jar.
Don’t charge credit cards until the product launches. Kickstarter lets you have customer money as soon as you reach your goal, but unless you have to have the money to pay for upfront costs, I’m a much bigger fan of not charging customers until you’ve actually launched the product. This will build more trust with your customers, it gives you an incentive to deliver, and if you happen to miss a deadline, at least you earn interest on your customer’s cash.
Only the beginning
This was a first crack at crowdtesting. As we learn more we’ll continue to post information. If you’re interested, you can subscribe for updates via Email or RSS.
Please, if you have any questions or comments send them our way. Also, if you run your own crowdtest, we’d love to hear about it (and have you post about it)!
