Comments on: Split Testing: Interpreting An Example http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm Email Marketing Tips by AWeber Sat, 11 Oct 2008 01:11:10 +0000 http://wordpress.org/?v=2.6.1 By: Gordon http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-10108 Gordon Sat, 09 Jun 2007 16:11:33 +0000 http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-10108 Interesting article and I like the idea of testing .... testing .... testing .... But I have to say with my cynics hat on, I'm not all that impressed by the different opening percentages. They're all remarkably close to each other and I'm pretty certain they're not different enough to get anywhere near statistical significance. I'm not knocking the idea - just doubting the stats before everyone goes crazy including the word Secrets in all their postings Interesting article and I like the idea of testing …. testing …. testing ….

But I have to say with my cynics hat on, I’m not all that impressed by the different opening percentages. They’re all remarkably close to each other and I’m pretty certain they’re not different enough to get anywhere near statistical significance.

I’m not knocking the idea - just doubting the stats before everyone goes crazy including the word Secrets in all their postings

]]> By: Mo http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-1435 Mo Fri, 19 Jan 2007 00:58:46 +0000 http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-1435 I saw this article titled "16 Tests (and Results) to Improve Email Response Rates" which also mentions use of testing to find best format. Shared results are also interesting to check http://www.marketingsherpa.com/sample.cfm?ident=29840 I saw this article titled "16 Tests (and Results) to Improve Email Response Rates" which also mentions use of testing to find best format. Shared results are also interesting to check http://www.marketingsherpa.com/sample.cfm?ident=29840

]]> By: Sten http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-920 Sten Thu, 16 Nov 2006 16:41:07 +0000 http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-920 I've created a small script that computes a value so you can know whether the difference is significant or not in an A/B split test like you're talking about. It is free and available here: http://programmerer.com/ab.php (Explanation on that page.) Sincerely, Sten I’ve created a small script that computes a value so you can know whether the difference is significant or not in an A/B split test like you’re talking about.

It is free and available here:
http://programmerer.com/ab.php

(Explanation on that page.)

Sincerely,
Sten

]]> By: Justin Premick http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-913 Justin Premick Wed, 15 Nov 2006 15:24:03 +0000 http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-913 Steve, Thanks for that. I can't really comment on how accurate that method is compared to using standard deviations, but it does seem a lot quicker/easier to use than standard deviations, so if you've found it to be accurate, I say run with it! Steve,

Thanks for that.

I can’t really comment on how accurate that method is compared to using standard deviations, but it does seem a lot quicker/easier to use than standard deviations, so if you’ve found it to be accurate, I say run with it!

]]> By: Steve Shepherd http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-912 Steve Shepherd Tue, 14 Nov 2006 22:26:58 +0000 http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-912 Justin After looking at a pretty cool php split testing product that uses taguchi the developer of the product talked about how to measure whether the difference was statistically significant. He worked on a square root of the sample size being the value that determined whether the difference was significant. ie The square root of 100 is 10 which is a 10% difference. however at 2400 samples you only need to get a 2% difference to be signficant. The bigger the sample the easier it is to get a significant result. Hope this helps. Steve Shepherd founder of theexclusive.info website PS. I use aweber all the time and it is FANTASTIC!! Justin

After looking at a pretty cool php split testing product that uses taguchi the developer of the product talked about how to measure whether the difference was statistically significant.

He worked on a square root of the sample size being the value that determined whether the difference was significant.

ie The square root of 100 is 10 which is a 10% difference.
however at 2400 samples you only need to get a 2% difference to be signficant.

The bigger the sample the easier it is to get a significant result.
Hope this helps.

Steve Shepherd
founder of theexclusive.info website
PS. I use aweber all the time and it is FANTASTIC!!

]]> By: Robert http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-896 Robert Mon, 13 Nov 2006 10:28:54 +0000 http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-896 Justin ! you are in the business of selling, babe ! Try to become a direct marketing genius ! How to Get More Customers from Free Downloads {!firstname_fix} How to Convert Free Downloads to Customers Here the people just think (old shit -we use it always, nothing new to us)while mostly they dont do. How to Convert more Prospects to Paid Customers by simply doing one easy thing all people can do, but most dont (yes you got it ! by using downloads) Conversion Secrets to get more Paid Customers you can possibly handle (Does this sounds great ? Read all dankennedystuff you can get. Justin !

you are in the business of selling, babe !

Try to become a direct marketing genius !

How to Get More Customers from Free Downloads
{!firstname_fix} How to Convert Free Downloads to Customers
Here the people just think (old shit -we use it always, nothing new to us)while mostly they dont do.

How to Convert more Prospects to Paid Customers by simply doing one easy thing all people can do, but most dont
(yes you got it ! by using downloads)

Conversion Secrets to get more Paid Customers you can possibly handle
(Does this sounds great ?

Read all dankennedystuff you can get.

]]> By: Justin Premick http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-880 Justin Premick Wed, 08 Nov 2006 19:03:14 +0000 http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-880 Steve, You're absolutely correct in stating that a sample size must be sufficiently large before we start testing and drawing conclusions. After all, if you send a message to 10 people, and 2 of them open it, that's a 20% open rate, but it's still only 2 opens. The sample size in this test is statistically significant. While I can't disclose precise numbers, I can tell you that each message in the split test above was sent to and opened by thousands of subscribers. Steve,

You’re absolutely correct in stating that a sample size must be sufficiently large before we start testing and drawing conclusions. After all, if you send a message to 10 people, and 2 of them open it, that’s a 20% open rate, but it’s still only 2 opens.

The sample size in this test is statistically significant. While I can’t disclose precise numbers, I can tell you that each message in the split test above was sent to and opened by thousands of subscribers.

]]> By: Steve Seltzer http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-879 Steve Seltzer Wed, 08 Nov 2006 18:53:43 +0000 http://www.aweber.com/blog/case-studies/split-testing-interpreting-an-example.htm#comment-879 Be careful when split testing that your sample size is valid. As a real eye-opener, try split testing identical pages. The results will NOT be evenly distributed. This means that you want to see an ever-increasing difference between your test pages and that the sample size be sufficiently large (I use at least 250 hits for at least one of the pages). I'm not sure how much confidence I would give to Justin's example as the percentage differences between the four groups are so small. The critical question is: How large was the sample size? Be careful when split testing that your sample size is valid.

As a real eye-opener, try split testing identical pages. The results will NOT be evenly distributed.

This means that you want to see an ever-increasing difference between your test pages and that the sample size be sufficiently large (I use at least 250 hits for at least one of the pages).

I’m not sure how much confidence I would give to Justin’s example as the percentage differences between the four groups are so small. The critical question is: How large was the sample size?

]]>