tl;dr
Using clever survey questions to quickly test the memorability, spellability and emotional response of potential domain names. Glad we did – almost picked a lousy one.
Problem: Domain Squatters
I’ve found what should be the joyous process of naming a company, quickly devolves into in argument over which of the abysmal choices of available domain names suck the least.
Some friends and I who were building a crowdfunding aggregator wanted to avoid all that, so we came up with a way to test our company/domain names with customer data. Here’s how…
Step 1: Crowdsource Ideas
SquadHelp is 99Designs for domain names. 
That means you pay a couple bucks and SquadHelp users will find you some 500+ domain names, all of which are currently available. And if you don’t like any of them, you don’t pay anything.
Looking over the list of possible names for our crowdfunding aggregator, we chose our top 3:
- altFunder.com
- ChangeFunding.com
- ThingsWeStart.com
altFunder.com was our favorite name going in to the experiment, the next step of which was deciding…
Step 2: What to Measure?
After reading up on what makes for a good company name, we decided to measure the following characteristics of each name:
- Memorability – If users can’t remember your name, they can’t tell their friends about it.
- Spellability & Hearability – If users can’t spell your domain, they could become someone else’s.
- Associations
- Emotional associations – what feelings do these names evoke? For more on why this is important see Selling the Why
- Image associations – Names people automatically associate images with are more memorable.
- Competitor associations – The internet told us you want to avoid a name that gives your competition more clout. Seemed like reasonable advice.
Next up…
Step 3: Design the Experiment
Here’s a video tour of the survey we designed (i.e. why you’re reading this post):
To test hear/spellability (try it):
1. Play this audio clip only once and type in the name of the website you hear:
To test memorability:
After speaking & showing the domain names, we’d ask them a bunch of “busy work” questions (e.g. demographic questions, trivial math questions, etc.) to help them forgot the less memorable names:
12. What is the 9th word of the 1st paragraph on this page?
- eat
- faster
- rubber
- printing
With their minds distracted for a bit, we’d then ask:
To test Associations:
For each of the associations (e.g. emotional, image, and competitor) we asked them simple questions like:
- What images come to mind when you hear this time?
- How do you feel when you hear this name?
- What organizations come to mind when you hear this name?
Btw, we used nsurvey for this – an open source survey tool that gave us the power & flexibility we needed. With the survey designed, it was time to…
Step 4: Run the Experiment
The site was B2C, so we decided to throw our survey up on Mechanical Turk. We paid $0.10 for each respondent and we stopped after we got 50 (all told $5).
Fraud Detection: we had quality assurance questions built into the survey (e.g. what’s the 9th word in the second paragraph, what’s 16 + 32, etc.) to detect cheaters. We gave cheaters a different password than people who took the time to fill out the survey thoroughly.
Step 5: Evaluating the Results
Once the survey results were in, I threw them in a spreadsheet and scored each of the questions like so:
Hearability/Spellability: +2 points for a name each time it was spelled properly
Here are the Hear/Spellability results for our names:
Memorability: +2 points each time the name was properly recalled
Image associations: scored subjectively by me (actual responses below)
Positive (+1 point): “people protesting”, “trees, plants, soil, nature”, “tattooed girls and rock music”
Neutral (0 points): “people”, “money”, “change”, “coins”
Negative (-1 point): “a steaming pile”, “a payday loan center”, “my cluttered basement”
Feeling associations:
Positive (+1 point): “curious”, “hopeful”, “interested to know what people are starting”
Neutral (0 points): “nothing”, “innovation, kind of a cheesy name”, “mildly inspired”
Negative (-1 point): “annoyed”, “confusion, boredom”, “lame”
Competitor associations:
Positive (+1 point): “none”, “no particular companies come to mind”
Neutral (0 points): “charity”, “Kickstarter”, “red cross”
Negative (-1 point): “republican websites”, “Obama campaign”, “payday loans”
Then we weighted the score of people who we thought were most like our targeted users (e.g. responses from Facebook and Kickstarter users got bumps) and added them all up:
View/download the complete spreadsheet here: http://sdrv.ms/Z9941S
Conclusion
We were wrong! We were leaning towards altFunder.com because it was short & edgy (and thus we assumed more memorable).
Turns out, ThingsWeStart.com was not only more memorable, it was also easier to hear/spell and it had much more positive associations.
For a couple bucks and a couple hours, we saved ourselves thousands of $’s by not picking the crappy name and having to change it later, or trying to buy one from a squatter.
Takeaway:
Tips & Tricks
– Before you post a survey about your domain names, be sure to purchase them all. Don’t want one of your clever respondents to snipe them from you.
– Most domain registrars offer a return policy. We bought all 3 domains we were testing, ran our test, and then got a refund for the 2 we didn’t use!
– Another great test would be to measure CTR on ads, or A/B test on a landing page. We didn’t have time for that, but I’d love to hear if you’ve run an experiment like that.
As always, comments and questions are appreciated!
If you liked this – consider sharing it so your friends don’t buy shitty domain names. You may also want to subscribe for new posts via Email or RSS. Our next article will be detailed explanation of Steve Blank & Startup Weekend’s Customer Development collaboration – SWNext – and how we think it fits into the greater startup ecosystem.
Hey Justin,
How did you add the audio recordings to the questions in nsurvey, or what program did you use record then upload?? I am trying to replicate this process but you made it look so easy :D
Hey Drew, thanks for the question. I think I used SoundCloud’s embed functionality. My guess is that NSurvey offer the ability to add HTML to a question and I put SoundCloud’s embed code in as HTML.
Some info on embedding SoundCloud: https://soundcloud.com/pages/embed
Hope that helps!
Justin
Thanks for sharing this Justin! If we hadn’t done this test, we would of chose the name “Kaizen” which it turns out is very difficult for people to spell and is associated with negative images.
For anyone who is trying to replicate this process and hasn’t used Amazon Mechanical Turk before, we put together a step by step guide and built a business name generator template in Google Sheets that is free to use. http://handground.com/blog/business-name-generator-5-hacks-to-find-and-test-the-perfect-business-name/
Here is a screenshot of the results:
http://i2.wp.com/handground.com/blog/wp-content/uploads/2014/10/Screen-Shot-2014-10-13-at-1.11.41-PM.png?zoom=2&resize=474%2C214
Justin the information is invaluable, thank you for the great post! I am embarking on a similar exercise based on this. I had a quick question, on mTurk did you use Project type “Survey” or “Other” to create the HIT? Also how many respondents did you get to consider it a relevant sample for statistical significance. Thanks in advance.
Hi Justin, where do you purchase your domain names? I can’t seem to find any place that will allow me to return the names.
Any of the GoDaddy run sites should support a return policy. http://support.godaddy.com/help/article/8849/refund-policy
IIRC, it was 7 or 10 days when I last tried. You might want to contact them though to confirm that’s still true.
Good luck!
Justin
“couple” means two
Hi Justin,
I’m not too sure what happened but it looks as though my post from last night is no longer there! Anyways, great article, it really has been a useful tool for me. Just one question, how did you create the last question in nsurvey which gave users who answered the CAPTCHA questions correctly, the correct code for MTurk?
Thanks for your help on this!
Great question Ash! I actually had two different “Thank You” pages for the survey. One page with the correct code, the other an incorrect one. I then added some simple conditional logic to the survey that said, “If they answered correctly to the CAPTCHAs, send them to the good page. If they answered incorrectly, send them to the bad page.”
That make sense?
Good luck!
great article. did you use squadhelp for the measurement? I could find the naming option, but not the testing. Wasn’t sure if you used a different website for the other portions…. thanks!
Squadhelp only helps with the naming. You’ll need to do the measurement yourself.
Best of luck!
Justin
Which host did you use that allows you to return the domain names you didn’t use? My host doesn’t do it..
Any of the GoDaddy affiliates should work – or at least did when I ran this experiment. Best of luck!
Thanks!
I found this to be a very helpful guide for creating a memorability survey. Thanks for sharing!
Awesome :)
Justin, thank you so much for this – used this process to pick our startup name about 2 months and it has worked out perfectly. Keep this stuff coming, I love it!
Hey Justin. Absolutely awesome idea! I’m coming to your blog quite often lately ;)
Can you tell me what tool you used to build the questionnaire? I tried surveymonkey, but couldn’t implement soundcloud-files (with the basic account).
What’s the best way to go here?
What would change if test was for B2B site (and not B2C as in this case)?
I loved this method so much I just tried it out for a B2B company I’m working with. Changes I made:
1) I didn’t put it on Mechanical Turk but rather posted it on LinkedIn and emailed my contacts who fit the proper profile
2) Instead of busy work like adding up numbers, etc, I asked a couple screens of demo-type questions. I thought the random busy work would just confuse and irritate my respondents, and since I wasn’t paying them I needed them to respond out of goodwill.
Thanks for the suggestions Beth! How did the experiment turn out? Any valuable new information?
Not as decisive as I hoped, actually. I kind of wanted one of the names to be either a total loser or a clear winner. They were fairly even.
The test was for B2C site. How would it be different if done for B2B site?
Concept would be similar, you want to test the hearability, memorability and associations with a B2B brand.
One way would be to go to a conference with your customers and try telling people different domain names. Make a note of how many times you need to repeat each domain name you say out loud.
You could also make up a small set of business cards for each domain name and hand out an equal number of them. See if any of the domains generates more traffic than the others.
Similar concepts – get your customers to demonstrate which name you should choose – just may need slightly tweaking for each customer base.
Best of luck!
Justin
Have you ever spoken at the Lean Startup Conference? This would make a great presentation for them. And they are accepting applications until 5/16. =) http://www.startuplessonslearned.com/2013/05/how-to-get-free-speaker-training-with.html
Also, what survey software did you use? Is that just basic Survey Monkey?
Thanks Beth! We used NSurvey.
Re the Lean Startup Conference, I have spoken there!
Oh yeah! I remember that one. (I spoke last year as well.)
Also putting in a vote for “This would be a good product”! Think what people pay naming firms, how much time they spend checking for domains, etc. One service that knit all this together and helped test it… people would pay for that.
Great test… but couldn’t help wondering how Google, Yahoo or eBay would have faired on this test if they ran it prior to launch.
Fair question. That said, if you had to choose between having a more memorable name, and a less memorable one…the choice would be obvious.
I suppose what your question raises though is, does a name actually matter at all? I don’t know. My very un-data-driven answer to that question is, “Probably a little.”
Well done! It’s amazing how many people assume their own instincts on a name are always right. Yet, testing can be a daunting challenge. You made the test strong by measuring meaningful qualities, but you also made the test approachable/accessible. Doubt it would only take a couple hours (!) but great work. Wish more people did this in advance if picking their shitty company names.
Justin, awesome “hack”!
2 other things we’re also using:
– “Pronounceability”: give Turkers a list of the potential names and have them call a Google Voice Mail and read the names. You can then go through all the recordings to see how people were pronouncing the names. In our case, some names were “made up” words and we expected to get different pronunciations, but we used that to see if some of these different pronunciations could results in spellability problems later down the road. So we took the Voice messages from Turkers (one male and one female) and used those for the spellability tests (which we ran in the same you you did).
– Mission Alignment: We describe the company mission and ask them which names better reflect that mission
Those are awesome ideas Pedro! Love the “pronounceability” idea, especially if you’re testing non-native language words and then reusing them as your spellability test. Great stuff!
Btw, what’s your new name?
ah ah! That’s an even better question! We’re running final tests this weekend. Will shoot you an email soon with news. I can also share some data if you find it useful.
+1 for Pedro. Once it becomes a branding exercise, there’s all kinds of of work to do. But those are great. Another question we sometimes ask is “Scanability” of a domain – basically whether your eyes can scan a name and actually comprehend the words that are being squished together.
Ex:
ThingsWeStart scans well
Thingsstevecandoonsundays does not (repetitive letters, etc.)
Case-sensitivity aside, it’s nice when a domain name seems to jump off a page when written in all lower caps.
Awesome post! Another resource to check possible trademark infringement: http://www.trademarkia.com/. I remember when we discussed whether Q… was a good name for the startup I was developing and you were right. It turns out it did horrible in the three categories of hear/spellability, memorability and associations.
Great to hear from you Antonio! Stoked to hear about the problem Skyin solves.
Reblogged this on Nick Hortovanyi's blog.
This is great! Please also consider looping in a trademark lawyer before the final decision, to help you evaluate whether the name you’ve selected is a good one legally as well. They can help you confirm whether it’s unlikely to conflict with someone already out there, and can function as a “strong” trademark. This is truly an ounce of prevention (a lawyer’s look-see) vs. a pound of cure (letters, lawsuit, and total re-brand) consideration!
Fair point, thanks for bringing it up Alina. That said, we can probably save ourselves a couple (hundred) bucks by using the online trademark search tool: http://1.usa.gov/YMHsyJ. If the name that users choose is one that conflicts w/ an existing trademark, perhaps then it’s time to get an official legal opinion.
Thoughts?
Nicely done Justin! If you make this into a product, I’d pay to test company names.
Thanks Matt – you’re the second person to mention that…wonder if there’s something there.
This is awesome! I wouldn’t have thought to check associations.
I’ve been wondering whether there are better names than Logical Increments for my thing. Now I know how to find out.
Thanks for sharing, Justin!
Wow, this is a great approach to naming your website, and laying it out step by step is super helpful. I think the use of Mechanical Turk is a great idea, as it provides you with essentially on-demand survey results at such a low cost.
How long did you have to wait to get your 50 respondents?
If I recall it only took one evening. Whatever it was, it wasn’t very long and if it’s taking longer than you like, you can up the payment amount. (and thanks for the question!)
Nice! When you posting on hacker news?
sweet!