Crowdtesting: what do you think?

I’m giving a talk next week on crowdtesting and I’d really love your thoughts before I do. I’ll be giving it at the Lean Startup Conference so if you’re reading this, you know the target audience. It’s you.

Here’s a run through (note: it’s < 5 minutes):


  1. What questions do you have?
  2. What can I cut?  Aka, when did you tune out and flip to Facebook?
  3. Anything else?

You’re my customers so let me have it in the comments, or you can email me: [email protected].

See it Live

If you’d like to hear the final version live (oh, and some other no-name speakers Eric Ries, Steve Blank, Ash Maurya, etc.) I’ve got two suggestions:

  1. If you’ve got $900 and can/will be in SF, come in person! Scratch that, I’ve got your back.  Take 15% off (coupon code: speaker15)
  2. Watch for almost free at a local simulcast.  John Sechrest is running one in Seattle, and there are also tons of others running throughout the world.

Thanks again for your feedback!

Join the experiment – subscribe via Email or RSS for tips on how to run your own crowdtesting experiment which I’ll be writing up soon-ish.

  1. Justin, do you need to talk about getting enough data to ensure that the differences are statistically significant? For example, for your test, you’d have to have something like 75-80K visitors to ensure a 95% confidence rate.

    1. Thanks so much for the suggestion Jim. Based on it and a couple other questions along the same line, I added the sample size of the experiments to the deck. Doesn’t correspond to 95% confidence level, but my current opinion is that’s not the threshold we need to meet to be lean. Would be interested to here yours, or other folks’ thoughts on where statistical significance belongs when we’re trying to be lean.

      Thanks again!

  2. This is already an engaging and interesting talk. I had two ideas to move it to ultimate rock star level.

    First, I love the “Guess less”. I’d lead with it or with something else that gives a stronger sense of why I should care about this. You have a specific example of a question that you would like to know “how much will they pay?” but your crowdtesting method could be used for a broad variety of questions that entrepreneurs have. Don’t sell yourself short.

    Second, I’d really like a story about one of your A/B tests. The pricing test, seems like the natural one to choose.You don’t have to change the visuals, just change a few sentences. You can say what you were expecting and how the results of the test were different. Or you can ask the audience to predict and then tell them what actually happened. Just something to make it more tangible and less abstract.

    Good luck with the talk! I’ll be excited to hear how it goes.

    1. Fantastic suggestions Lori, thanks so much for taking the time!! I did end up rewriting the beginning of the talk to have a more consistent theme, although it changed from “guess less” to “is your startup a business or a hobby?”

      Also a great call on asking the audience what $ they thought would be optimal, in fact I incorporated that into the talk at one point. Unfortunately, I didn’t have enough data on some of the popular choices I know people would have had (e.g. $.99, ad supported, etc.) so the reveal would have been a little underwhelming.

      Thanks again for the suggestions. I’m pretty pleased with the final version and will post it for ya’ll once it’s live. Hope you’re well, Lori – wishing you all the best.

  3. It was great and I really like the info. My one comment is that you should probably speed up a bit. You were talking pretty slow and maybe that will be needed when you present but it left me wanting a fast forward button.

    Hey Justin, thank you for sharing this great info!


    1. Thanks Frank. Really good call. The final version has more energy, and more speed.

  4. Hi Justin,

    I thought the video was a great example of ‘say dog, show dog’ presentation style, very clear (if a little slow, as commented above). Just from the video though, this was not made clear to me:
    1. What does ‘Fork Now’ mean?
    2. The link was broken at the time I visited it – 28 Nov.

    All the best for your presentation!


    1. Thanks Wei. Good call too, I appreciate you echo’ng Frank’s slowness comment. The final version will pick up the pace.

      I’ve also cut the “fork” reference, which in this case means to create a new copy of an open source project and start working on it, and the link should work.

      Thanks again for taking the time!

Leave a Reply