Step-by-Step Instructions

Click to open Significance Calculator

In this chapter you will learn how to use A/B testing to optimize your Currency Conversation Rates. Let's get started!

Problems it Solves

• How do you increase your Currency Conversion?

If it turns out that one of your experiments did not meet your success criteria, how can you optimize your Conversion Rate? What tools and strategies do you have to move from the "optimize" range of your Success Metric Stoplight into the "success" range?

• Learn to run a valid A/B test.

You will make sure you know how to run a valid A/B test that will allow you to trust your data.

• Prepare for price testing.

This chapter will lay the foundation you need for testing your price to make sure you are charging the optimal amount.

A/B Testing

A/B testing is the best tool for optimizing just about anything. It is important to know how to do it right, how to test effectively, so that you can trust your data is a real indicator of your customers' reality.

All of the steps on your Currency Ladder, and in your Currency Calculator are available for optimization via A/B testing. If fact, you can test each step of the Ladder and ultimately optimize your Conversion Rate at every possible juncture you user can take.

A Familiar Example

I would like to provide you with an example you may be familiar with, especially if you found out about FOCUS through my blog.

On my blog you will see a header that says something like "How to Find Your Product Market Fit" and then a link to the FOCUS Framework. You may see that exact phrase, or you may see something slightly different.

In fact, there are a number of different possible things that you might see on my title bar. Over time, I've tested 35 possible combinations of tag lines and links that may appear each time you go to my blog. I used a tool called Hello Bar to present different combinations of headers to visitors of my site, which also tracks each variation's Click-Through Rate (CTR).

Here are some example of the results:

One of my title bar tests said: "Find Your Focus." 919 visitors saw this title bar when they came to my blog. A total of 9 people clicked on it. By dividing the total number of visitors by the number of clicks, you can see my CTR (one of my forms of currency on my Currency Ladder) for this title bar was 1.0%.

Similarly, "HOW to run experiments" generated a CTR of 1.3% and "HOW to find your Product-Market Fit" reached the high of 2.3% - which was by far the most effective way to describe the book series you're ready right now!

While the difference between a 1% and a 2.3% CTR sounds trivial, the latter will drive literally 2.3 times as many people to my landing page over the former. That's a 130% increase in my conversion rate.

This is what it looks like to optimize a particular type of currency. In this case, the currency I was trying to optimize was clicks, as measure by my "click-through rate."

Not Just For Currency

A/B testing does not just have to be for optimizing your currency. You can also use this same technique to optimize your Offer.

Before choosing to write this book, I tested a number of different "solutions" to helping founders solve the problem of feeling overwhelmed. These included:

• 1-1 Mentoring and
• a Video Course.

As you can see in the image above, 1-1 Mentoring did not get a very high click-through rate (0.9%), but the Video Course came in a close second to the Customer Development Workbook. Due to the relative popularity of the Video Course offer, it's likely I'll consider developing that down the road - especially since I already know that's an offer my customers respond to.

A/B testing can be used anywhere you are trying to optimize a particular action you want your customer to take.

Without A/B Testing...

It is safe to say that without A/B testing, the resource you're reading right now, this workbook, would not exist. The A/B tests told me what was most important to build (a workbook, as opposed to a video course or mentoring program), and what would be most helpful to you, my reader, in helping to make sense of the overwhelming nature of starting a company.

The shape and content of FOCUS Framework has been determined by Early Adopter's response to these early tests. Your customers can do the same for you if you use A/B testing properly.

The Principles of A/B Testing

1. Statistical Significance Matters

Statistical significance is a measure of how likely the data you are seeing in an experiment is the result of random chance.

Imagine your A variation is converting at 2% and your B variation is converting at 3%. Statistical significance will tell you how likely it is that the B variation is actually better than the A variation, or whether it just appears that way due to random chance.

Statistical significance becomes important because at this point you are looking for quantitative data: cold-hard numbers.

This is different than the type of data you were collecting about your Early Adopters during your interviews. Interviews are qualitative data: thoughts and feelings and observations of a small number of people, that you use to develop your initial assumptions. During this quantitative research, you're using statistical significance to validate the assumptions you created as a result of your qualitative research.

In quantitative data, you are generally looking for 90-95% confidence level that the difference between two variations is not due to chance.

2. This Is Not Just For B2C

A lot of people think that A/B testing is only for businesses that involve online ads, landing pages, and a ton of traffic, ignoring how powerful this testing can be in many other circumstances.

In enterprise sales, even though you don't have a landing page, you have many ways to test your ladder steps of currency:

• A/B test your email content or subject line

As you send out cold emails, you can try different subject lines to see if that changes whether someone opens the email. You can change the content inside to see if it changes the likelihood they will respond.

• A/B test your cold calls

You can try out different messages or different scripts to measure the number of people who take the next step on your Currency Ladder, or how far into the conversation you are able to get.

• A/B test your slide decks

As you give presentations to various enterprises, you can change the content of your presentation or present different slides to measure a response, such as how many people follow up with you after you present.

• You can A/B test at conferences

At conferences, you may be able to A/B test one sales pitch one day and another on the next day.

These are not what I would call hard core, highly quantitative A/B tests, in that some of these things may be effected by other variables, such as time or situation (i.e. at a conference, one slide deck may perform better on the first day of the event than on the last day).

However, it is still worth using these same tools because you can combine your quantitative and qualitative data to get the information you need.

3. Avoid "Multivariate" Testing

Multivariate testing is where you test multiple variables at the time (e.g multiple variations of subject line and email content in the same test).

Caution! This kind of testing is reserved for companies that are testing on huge sample sizes (i.e. lots of potential customers). For every variable you introduce into a test, you are requiring more and more people to participate in the experiment to generate statistical significance. In fact, for every variable you add, your need for customers to participate goes up exponentially.

Test one variable at a time.

Before you've found your Product-Market Fit, you will have the most success by testing only one variable at a time. Your data will be clear and you will be able to separate the significant differences from the noise.

4. No Touching

You may be tempted very early on during a test to say that it is "so obvious" that one variation if performing better than another. However, until you reach your statistical significance benchmark, you need to let your test run.

Don't touch your test, and don't make any official determinations, until you've reached your statistical significance goals!

5. Run Variations in Parallel, Not Serial

You are better off running two variations at the same time than you are running one variation for a period of time, and then replacing it with another variation for the same amount of time and comparing the difference.

Running variations in parallel will eliminate any variation that may stem from differences in time: day to day, week to week, month to month, etc. If you run experiments back to back, one now, one next week, you will not know if your variance is due to the difference variations, or due to differences in the time of month or year.

There are differences in time that we often do not consider. For example, people get paid at different times. Many people are paid the first week of the month and are better able to make new purchases then.

The example above is a graph of the weekly sales revenue of FOCUS. Notice how much variation there is week-to-week, despite the fact I made no changes during this time to my Offer or Currency Ladder. No price changes, no product changes, no copy changes...nothing.

You can see that if I was trying to run A/B tests back to back with one another, with this level of weekly variation, I wouldn't be able to tell if my experiments were actually having an impact, or whether it was just natural statistical variation.

Tools to Use

Now that you know the principles for your A/B testing, let's look at some of the tools that will help you conduct your experiment.

• Instapage

This tool allows you to make beautiful landing pages. This tool will also incorporate the A/B test for you within those pages.

• Unbounce

This tool is very similar to above, though the landing pages are more basic and not as sophisticated as above. However, the functionality is extremely sophisticated which means you may be able to do more with this tool if there is something Instapage is not permitting.

• Hello Bar

This is the tool that I used to test some of the currencies that led you to purchase the FOCUS Framework. This service allows you to customize your header bar with different messages for your customers.

• Google Content Experiments

This tool will help you A/B test your content on your webpage and it is free.

• Optimizely

This tool is a bit easier to use and offers free and paid plans with more functionality than Google Content Experiments.

• Mailchimp

This is a fantastic tool for integrating A/B testing into your cold emails.

How to Calculate Statistical Significance

Step 1

Take out your Currency Calculator then go to this website: TestSignificance.com

Take a look at the Total Conversion Rate for your primary customer segment.

Now, on the Test Significance website, type in a number in "Current Conversion Rate" that is half of the number you have in your "total" box. In my case, that is 0.61%.

Now set your Confidence Level at 90% or 95% and click Calculate.

Now, let's pretend you were running a test and you were getting half the conversion rate you needed to achieve your Victory. In my case, it would be 0.61%.

If you find the row that shows you the "half" success metric for your A conversion rate, and the "full" success metric (in my case 1.21%) for your B conversion rate, the tool will tell you the minimum number of participants you'll need to have 90% or 95% confidence in the results you were observing.

In other words, if I am getting a 0.61% conversion rate, I will make some changes to my Currency Ladder to see if I can reach 1.21%. If my alternate variation starts to reach a 1.21% conversion rate, how will I know if I'm seeing "real" results, and not just statistical noise?

The answer is, in my case, when my B variation has produced a 1.21% conversion rate after 2,708 customers have participated in my A/B test.

---

Going forward, use this tool to calculate how many participants you need to participate in your experiments in order to determine the statistical significance of your A/B testing results.

Recap

Now you know how to increase your conversion rate by using A/B testing, as well as tools to use to make these tests happen.

You also now have the principles required to run a valid A/B test, so you know that you can trust your data.

You also have the foundation necessary to understand How to Test Your Price.

What's Next

Up next though, is how to improve your conversion rate via Solution Interviews.