I run a little website called Days, where users input their date of birth and the website tells them how many days old they are. It’s kind of perfect for A/B experiments because it doesn’t have much going on and has a very clear “flow” -- there’s exactly 1 thing that users are meant to do when they reach the site, and that’s “enter their date of birth then press the button.” It was also lucky enough to be linked from a fun site called Random Useful Websites, which (as the name suggests)is a simple page that has users click a button and get sent instantly to a random website. That means lots of Days users come in having no idea what site they're going to see and with no preconceived notion of what the site is/does -- perfect for cleanly A/B testing which features and design elements turn off users.
Here I will document A/B tests I run: as always, your mileage may vary but perhaps the results will be useful. I am currently using Optimizely, but have heard very good things about VWO as well. (If you're new to A/B testing, read the always-excellent Kalaumeus first).
1) Removing a cute quote from the header
In the original version of the site, I put a quote I like (from the Philip Larkin poem Days just after the header. I enjoyed it, and I figured no-one else would care.
I was wrong.
A first surprise was that the website with the quote at the header was only achieving a 53% conversion rate for people hitting the "Let's go!" button. That is, only just over half of users were actually using the website at all -- the other half were giving up before even calculating their age in days. Before I started testing, I was not tracking these figures and so did not even realise that so many people were giving up on my website before they'd even begun; I wonder how many other projects out there face a similar drop-off, unbeknownst to their owners or designers.
A second surprise was that this conversion rate could be improved to 73% -- an extra 1 in 5 of total users were now actually using the website -- just by removing the Larkin quote.
Sorry Larkin, you had to go. Somehow I feel you would understand, though.
2) Shifting pronouns
The original design of my website is written in second-person: “Your date of birth is [date]”, “You will be [X] days old on....” I have a long-running interest in pronouns in writing, and how different pronouns/framings change readers’ experiences. So I wanted to know what would happen if I changed the pronouns to make the site first-person: “My date of birth is”, “I will be [X] days old on...”
There was no statistically significant result after 100+ samples in each condition.
Limited time only: Get a free copy of Write Harder + (very) occasional & thoroughly excellent emails.