Step-by-Step Instructions

Problems it Solves

1. What exactly will you measure?

In this exercise you'll learn what to measure to determine if your Utility Test is a success or failure.

2. Is your product solving the problem?

You'll also learn how to determine if you're doing a good enough job to solve your customers' problems.

3. What is the right level of quality?

How do you know when you are solving the problem "well enough?" We all know that too little quality is a problem, but could there be such a thing as too much quality?

The Crisis of Quality

Let's start with this concept of "quality."

Quality is a variable, just like any aspect of your business. Your price is a variable, how much design energy you put into your product is a variable, and it turns out that the quality of your product is also a variable.

I'm sure you've had experiences with products that had too little quality, but there is also such a thing of a product having too much quality.

So what is the right level of quality for your product?

Think for a moment about one product: a defibrillator.

The quality of this product is paramount. This product:

• Needs to work every time.
• It needs to not break in physically demanding situations.
• Needs to work on all different types of bodies.
• Needs to be designed so well that anyone can operate it.

This product demands a high level of quality to solve the problem it's designed to solve.

Now think about a different product: a paperclip.

A paperclip needs a certain level of quality. It needs to:

• To be flexible
• Not ruin papers
• Last long enough to get the job done

Paperclips need to be good, but we don't need paperclips made of precious metals, they don't need to be organically sourced, they don't need to be elegantly designed, or polished to an incredible shine. Paperclips demand an appropriate level of quality, but no one is asking for paperclip providers to go over the top creating the finest paper holders in the world.

At the same time, you could imagine a founder who was so passionate about their product that they wanted to build the best paperclip in the world, and become obsessed with getting a high level of quality into a product that does not necessarily need it.

The question is, how do you know the right level of quality for your customer segment?

Vanity Metrics

In a previous exercise, we talked about the concept of Emotional Delta as a way to prioritize your features.

Just to recap, Emotional Delta is capturing the difference between the emotional state of your customer before they have engaged with your product and after. Your goal is to create the biggest positive emotional change possible.

Emotional Delta, then, is one thing you could measure as part of your Utility Test.

The benefit to measuring Emotional Delta is that it is a clear measure of your customer’s experience with your product and how well you are addressing their problem. By measuring Emotional Delta, you get a sense of how much happier you're making your customer.

The problem with measuring Emotional Delta is that it is a vanity metric.

A vanity metric, is a metric that makes you feel good; one that gives you the sense that because of your product, something good is happening in the world. However, a vanity metric does not necessarily lead to Victory. While it may be indirectly related to your Victory, or a step along the way, it is not ultimately tied to your Victory.

For example, if your product is a paperclip, a vanity metric might be “Total number of pages bound world-wide.” While you'd be able to brag about this number, it is not particularly interesting unless your personal Victory is to bind 1 billion papers in the world.

What you actually want to measure, perhaps, is paperclip revenue generated. That metric is much more likely to get you to your Victory.

Now revisit the Defibrillator example. While the purpose of a defibrillator is to save lives, from a business perspective, the number of lives saved is not your goal - you can't take saved lives to the bank, or use them to hire more employees.

While “saving lives” is important - the more lives you save the more people will recommend your product, the higher you can charge, etc. - it is still a vanity metric because it makes you feel good, but isn't directly associated with your Victory.

Understanding the difference between Victory metrics, and vanity metrics is key at this stage.

At the end of the day, you want to measure things that are directly tied to your Victory. So if you know Emotional Delta is a vanity metric...

What should you measure?

For most things, you will measure revenue. In other words, measuring Emotional Delta is key for prioritizing features because it is an easier way to compare the value of various potential aspects of your product. However, it is not as useful when you are measuring utility, because it is not directly tied to your Ultimate Victory.

To avoid measuring a vanity metric, measure these metrics instead:

• Customer Lifetime Value
• Viral Coefficient

Both of these metrics are ones you're familiar with by now. We reviewed them, and you estimated yours during the Currency Calculator exercise - which you'll want to reference for this next section.

In your Currency Calculator, you have the Total Lifetime Value indicated in Step 2. Now you can use your Utility Test to measure how realistic this number is. This number is one metric you can use to measure the utility of your product.

You may also use the Viral Coefficient, which you calculated in Step 4.

Both Lifetime Value and Viral Coefficient are tied directly to the utility of your product.

If your product is low quality, it will lower your Lifetime Value metrics. At the same time, poor quality will also affect the number of referrals you get from existing customers. Therefore, your Viral Coefficient will not be as high as it could be.

Both of these metrics connect the quality and utility of your product, to your Ultimate Victory.

In your Utility Test, you will focus your attention on one of these metrics - by now you've learned that it's best to always focus your attention on one metric at a time. Once you optimize one of the metrics, you can repeat the process to maximize the other.

Which do you start with? Here are some general guidelines:

• If you have recurring currency, or currency that comes in on a regular basis – i.e. ad based revenue from frequent usage of your product or a monthly subscription – or if you are going to ask for your Victory Currency more than once from your customer, start by optimizing your Customer Lifetime Value.

For example, if you want to build a SaaS product with a monthly subscription, you want to optimize a Customer's Lifetime Value before you the Viral Coefficient: it doesn’t make any sense to get additional customers referrals if you’re not making the most of the customers you already have.

• If you have a one-time purchase, or if the currency you are asking for comes on an annual basis (i.e. annually billing vs monthly), then I recommend you start by measuring the Viral Coefficient.

In the “one-time” purchase scenarios, it is difficult to measure the quality, or Utility, of your product via Lifetime Value because you may have already maxed that number out, no matter how poor the utility of your product is. Instead, you'll want to measure how many additional customers new customers bring to your solution.

In the "infrequent" purchase scenario, customers may pay you multiple times, however, the length of time between purchases (e.g. annually) is too long to iterate your product if it has quality issues. In this case, better to measure your utility by Viral Coefficient, so you can improve your product more quickly.

Measuring Option #1: Total Lifetime Value

 Take out the your Utility Testing Metric worksheet.

I'll start by explaining how to calculate your Total Lifetime Value. Viral Coefficient is described below.

Step 1

For the Lifetime Value section, I am using the example of a monthly subscription for Mentoring as my product.

In the first space, write in the Period, or length of time between each purchase. In my mentoring example, this Period is one month.

Step 2

Now write in the amount of money you will request during each Period. This number should already be in your Currency Calculator. For my example, I am charging my customers $99/month for this Mentoring subscription service.

Step 3

Write in the number of Periods you expect from each customer. This number is also in your Currency Calculator under Total Lifetime Value: how long do you expect each customer to stay on?

Step 4

In this step, you will be measuring your customer churn rate.

What is your Churn Rate?

Churn is the percentage of customers who opt-out of your product during a given payment period; it's a way to estimate how many payment periods your average customers will stay with you.

In other words, if you can estimate how many customers opted-out of your product between between month A and month B, you can extrapolate what your average length of subscription will be.

You can use this concept of Churn to estimate a customers LTV so you don’t have to wait an entire (in my example) 6 month customer-lifetime-cycle to determine how many customers have stayed and how many have left. 

To calculate your Churn rate, simply divide 1 by the number of estimated payment periods you expect a customer to stay with you; that will generate your Churn rate. In my example, I get a 16.7 % Churn rate.

In order to test this Churn rate, I will measure the number of customers who stay and those who leave over 2-3 Periods. If my monthly Churn rate is higher than 16.7% on average for each of those periods, then I know that my Customer Lifetime Value is less than I expected. For example, if I find that I’m losing 20% of my customers each month, I can estimate that customers will actually stay with me for 5 months, not six.

On the other hand, if my Churn rate is lower than 16.7%, then I’m good to go! For example, I may find that my Churn rate is 10%. This means that my customers are actually staying with me for 10 months, rather than 6.

The Churn rate will help me get a more solid estimate of how long I can actually expect a customer to stay.

Step 5

Now you will come up with an Action Plan. Your Action Plan will be entirely dependent on your company and product.

In this step, you are coming up with a way to measure your Churn rate, with a manual version of your product, over 2-3 payment periods.

In my example, I will enroll 10 people in my Mentoring program at $99 per month. I will measure churn across 2 months to estimate my Lifetime Value. If, over these two months, 2 people drop out, I'd have a Churn rate of 20%. This means that the average number of months someone stays in my program is more likely to be 5 months instead of 6.

If I think that this is the result of an initial drop-off and things may steady over time, I might think about extending my experiment to three months and measure the Churn rate that next month.

It is important to remember: when doing something manually, it is likely that your Churn rate will be lower (more favorable) than when you replace your solution with an automated version. Presumably, working with you individually, you'll provide a higher quality, more hands-on solution than your first automated solutions.

Therefore, if you aren’t able to meet your Churn rate will a personalized, manual service, it is very unlikely you will be able to reach that same number when you automate, so keep your numbers high!

Alternative

If the currency you are looking for is usage, for example with an ad-based platform, you can measure stickiness instead of Churn.

Stickiness is how often are people coming back to use your product. Similar to above, you can measure how often people are coming back over a weekly/monthly period. Using that calculation, you can measure how much revenue you generate per visit, and estimate your Customer's Lifetime Value.

Measuring Option #2: Viral Coefficient (K)

Just a reminder, the Viral Coefficient is a measurement depicting: for every customer you get, how many additional customers will they bring to you? In other words, for each of your customers, how many additional people will they tell, and of those, how many additional people will convert?

For this part of the worksheet, I’m going to use the FOCUS Workbooks (what you are reading right now) as my example.

 Step 6

 On your Currency Calculator, you have already calculated your Viral Coefficient (typically abbreviated "K" or "K Factor"). In Step 6, write in your projected K value from the Currency Calculator, Step 5.

In my example, my K Factor is .13.

Step 7

In this step you will estimate the Referral Timeline.

The Referral Timeline represents how long it will take for someone using your product to tell someone else about it, then how long will it take that person to become your customer. This is important to estimate because it will tell you how long your experiment needs to run: you need to let your customer use your product long enough to let this cycle happen.

In some cases, this may be very short. If the product is a social network where you have to tell your friends in order to use it, the timeline may be a few days. For other products, the timeline may be longer. For example, if your product is an accounting service, a customer may have to use it for a year before they have enough experience to start spreading the word.

For my example, the FOCUS Workbooks, I imagine it will take my customers about a month to get around to using the books, another month to fall in love with them and tell someone else about them, then another month for that person to sign up. I will estimate that my referral timeline is about 3 months: once one customer starts to use the product, I should start seeing the word of mouth referrals appear after 3 months.

Step 8

Here you will write in your Prompt. A Prompt is what you will tell your customers to remind them to share their experience with someone else.

For example, you might have a reminder pop up that invites them to share what they’ve just done on Facebook. You might see examples in Kickstarter or Groupons, where a certain number of people need to convert before the first customer gets their benefit. In that case, customers are prompted at check-out to share the campaign with their friends.

In Step 8, write in your own Prompt.

For this step, it's OK to write in "nothing"; perhaps you want to measure the natural Viral Coefficient. You’d like to know, if you do nothing, how many people will naturally spread the word. In my case, that's what I'm going to do.

Step 9

Here you will develop your Offer. This is what you will give your customer in exchange for spreading the word. Again, sometimes this is nothing, but other times it is something of value to either the original customer and/or the new customer.

This might include a sign-up bonus, or an affiliate marketer bonus – or both. Lyft and Uber do a great job at this: for every person you refer to their platform, you and the person you referred get free usage. Dropbox does this as well: if you successfully refer someone, you will get free extra storage space.

In my case, I’m actually going to choose the “nothing” again because I would like to see what naturally happens. Down the line I can come back and add Prompts or Offers and see how the improve my Viral Coefficient.

Step 10

Here you will develop an Action Plan for measuring your K Factor.

In my case, I will run a 4-month test: I have to let enough time pass for the referral to take place. I will ask each person who signs up within that 4-month period to tell me how they heard about FOCUS. Using that data, I'll calculate the total number of people who heard about FOCUS from a referral to calculate my Viral Coefficient.

Are my month-4 referrals greater than my estimated K? Are my referrals greater than 13% of my month-1 customers?

For example, let's say I have 10 customers after my first month. By month 4, if I have 50 customers, 2 of which were referrals from my first-month customers, then I'd calculate my K as:

2 referred customers / 10 first-month customers = .2

In other words, for every 10 customers I get, I'm getting 2 referrals. In this case, with a K Factor of .2, I'd say I have a 20% Viral Coefficient, which would be higher than my estimated 13%.

If I only have one new referral by month 4, that means that K = .1: I only have a 10% referral rate, which is lower than I expected. In this case, I would not be meeting my success criteria.

Another Example

Here is another example where I am using a Prompt and an Offer to be a bit more aggressive about my referrals.

In this case, my estimated K is the same: .13.

I might shorten my timeline because I want to get my data back more quickly. I’ll write in 2 months for Step 7.

I can try offering a Prompt to try to make the referrals happen more rapidly, more frequently, or increase the success rate.

In my case, I could say:

Do you know other founders who are looking for Product-Market Fit?

I can add this as a splash screen or have it show up after each exercise.

For the Offer, I could propose:

For every person you bring to FOCUS, you and your friend will get a free mentoring session.

In my new action plan, I will be running a 3-month test. I will present them with the Prompt and Offer after using the platform for one month. Finally, I will add the question, "How did you find out about FOCUS?", to the payment page.

Again, at month 3, I will measure, how many of my customers are coming as direct referrals from my month-1 customers? If my K > .13, I'll have met my success criteria.

What’s Next

Your goal of Utility Testing is to identify the optimal solution for solving your customers' problems, while achieving your Victory.

In this chapter, you have learned what metrics to measure, and how to measure them, to iterate your way to your optimal solution.

You also now know how much quality is enough:

If your Customer Lifetime Value and Viral Coefficient metrics are high enough for you to achieve your Victory, your solution has sufficient quality.

Next you will define your actual Utility Experiment. This is similar to your Offer and Currency experiments, including what to set up, what to look for, and what to do if your experiment fails.