Ultimate A/B Testing resources and how I used them

Reading time: About 2 minutes

Definition: A/B testing allows you to test variations of content simultaneously to see which performs best.

A/B testing overviews

Using Javascript to run A/B tests

  • For flexiblity use Genetify, a stand alone Javascript file that allows you to do testing on variations in content and CSS.
  • For ease-of-implementation use Google Website Optimizer, a more turnkey solution than Genetify. Just plug stuff into Google Website Optimizer and it does almost everything for you.

Google Website Optimizer tutorials

Genetify tutorials

New A/B tests on Brian Cray.com

I’ve been wanting to run an A/B test on Brian Cray.com for a while. This morning I setup some tests in my blog post footer using Google Website Optimizer to test which words influenced the most action.

Here are the tests:

Variations of “Add comments”

  • Add comments (original)
  • Respond
  • New comments

Variations of “Subscribe”

  • Subscribe (original)
  • RSS

Variations of “Share”

  • Share (original)
  • Bookmark
  • Save

Variations of “Follow me”

  • Follow me (original)
  • @briancray
  • Twitter

What will be the results?

What do you think the results will be? Have you run similar A/B tests? You should share your experiences with others in the comments!

10 comments skip to comment form

  1. Bendesign said— 19 hours later

    Nice Blog. Thank you.

    #1
  2. Matthew said— 4 days later

    Thanks for including UX Booth :) . Nice post.

    #2
  3. Brian Cray said— 4 days later

    No problem Matthew–I’m actually in the process of finishing a post for David for UX Booth :P

    #3
  4. Jason Cohen said— 2 weeks later

    I found your blog recently through HackerNews, ironically about you losing the ability to get traffic from Delicious alone. :-)

    On A/B testing math, check this out: http://blog.asmartbear.com/blog/easy-statistics-for-adwords-ab-testing-and-hamsters.html

    #4
  5. Usability said— 3 weeks later

    The a/b tests that you are running are for tiny bits and pieces – a few word changes here and there in the footer. I guess you are starting from small changes (as regards to usability and readability), then thinking big. it seems you are using google website optimizer to measure the results. I know of clixpy, which seems to be a third party tool to see user actions as they happen in your site (so you could see real time results of how users are trying to find the info they need). I guess it could also help with this, as a more advanced kind of tool.

    #5
  6. Brian Cray said— 3 weeks later

    @Usability: Testing massive differences with A/B testing makes the cause of the conversion changes more difficult to understand. For this reason small changes are highly recommended to large changes.

    #6
  7. Usability said— 3 weeks later

    Great Brian, let us know of the results. I guess ‘respond’ is out when it comes to adding comments. I guess people would find it easy when a comment is called a comment (rather than ‘respond’).

    yahoo has the equivalent of google website optimizer – they call this test and target. Its not free of course, being more enterprise-class, it seems to be suited for a/b, multivariate and other types of testing also.

    http://www.omniture.com/en/products/conversion/testandtarget

    #7
  8. Mike Darnell said— 1 month later

    Split testing can be done for video as well. By automatically generating slightly different videos promoting the same product and continuously optimizing the video to be displayed based on how well each version preforms in terms of sales generated. We’ve been able to take split testing a step further.

    The results speak for themselves:

    “Since launching online videos earlier this year with Treepodia we’ve experienced a 30% increase in product page conversion rates”
    Roy Hessel – CEO – EyeBuyDirect.com

    Original article here: http://bit.ly/18um4X

    Cheers,
    Mike
    http://Treepodia.com
    @treepodia

    #8
  9. Rex Dixon said— 5 months later

    @MikeDarnell – Was interested in your Treepodia.com biz and how you measure success or failure of a video test? Is there any type of process or “how to” on measuring the results for a/b testing or multivariate testing?

    #9
  10. Rex Dixon said— 5 months later

    Here are two sites that might be useful – http://www.performable.com/ and http://www.abtests.com/ – Performable actually just announced a $3 million round from Charles River Ventures today on Boston.com ( http://bit.ly/7oQ7IU )!

    #10
  11. Respond to this post—

Return to navigation
966