What are the most annoying scientific hypotheses

A | B testing - 5 mistakes to avoid

A | B tests are great. With their help, you can significantly improve the performance of your website and ensure that the conversion rate goes up. But they can also backfire. This is when you make a mistake that skews the result.

Mistakes can be costly

In the best case scenario, this will only cost you a few conversions (which is bad enough, because you actually want to optimize and not "make it worse"). In the worst case, such a mistake can also be very expensive - if, for example, you completely threw your functioning landing pages over the heap because you got the wrong result. Then not only the conversions suffer, you have also invested a lot of resources in the rollout of a new - in the long term useless - design.

To save you this fate, we have summarized the five most annoying mistakes in A | B testing. Check out this article and learn how to avoid them.

Results are believed to be correct too quickly

You finally set up your first A | B test and have been waiting longingly for a significant result for days. The data from your CRO tool shows that your modified landing page works better than the original. The probability that your test will pass is 75%. That sounds pretty good, doesn't it?

No! Seeing results as true too soon is one of the most common mistakes in the world of A | B testing.

A significance of 75% means that every fourth test your new landing page won't perform any better than the original. So before you invest a lot of money and time in redesigning your website, you should wait until the probability is at least 90%. However, 95% and more are even better. Only then can you say with certainty that your result will stand up to all doubts.

The problem of too small a sample

It can also happen that your test results are incorrect because the sample is too small. So you simply haven't collected enough data to build a resilient population. After all, what good is it if the first five users provide all matching data? This value is so small that it cannot simply be transferred to the general public.

The setup of an A | B test

Especially when the difference between the variation and the original is very large at the beginning, decisions are often made too hastily. In such cases, many well-known testing tools often decide too quickly on a winner. Here it is up to you to adjust the settings so that you can rule out such errors.

If you decide too early on a variant that may be even weaker in the long term, you will damage your company. This is exactly why it is so important that you rely on high significance and large samples. This approach reduces two dangers at the same time:

  1. Calling out a winner who isn't actually a winner (False Positive)
  2. Not recognizing a winner even though there is one (False Negative)

It is human and normal that you feel connected to your tests, especially in the early stages. Everyone wants the time, resources, and effort put into a test not to be wasted. And everyone wants their tests to lead to a true result as quickly as possible. Unfortunately, this is also the main reason why many A | B tests are aborted too early. Therefore, as a responsible conversion optimizer, you should always be aware that it is not just about creating a better landing page or a nicer design.

It's about finding the truth. And the truth depends on two things - the significance of the data and your hypotheses.

The test without a plan or: the missing hypotheses

Do you know that, too? You are so excited to finally be able to start a new series of tests that you jump into the adventure head over heels. It must have happened to everyone before. What then falls by the wayside is a resilient plan on the basis of which the test should take place.

Because if you start an A | B test without first giving it a lot of thought, you won't get any helpful results in the end. In their enthusiasm, people often underestimate the importance of creating meaningful hypotheses before an A | B test.

Many simply assume that in the end enough interesting data will come out for a targeted analysis - but that is a misconception. Only the correct hypotheses make a test informative. And only they help to work out the strategies that really lead to a better conversion rate.

"The quality of the results of an A | B test stands or falls with the tested hypotheses."

This means that formulating robust hypotheses is one of the most important steps in the whole process (you can find out what the whole process looks like here) - it is the basic requirement for the success of the next steps. In any case, you have to keep an eye on the overall concept of your website - because there is no point if you only stick to the one current test.

And if something goes wrong in a test - for example because it doesn't fit into the bigger picture - that's no problem either. A | B testing is an ongoing matter. You can also learn important lessons from failed tests that will undoubtedly help you in the further course of the optimization.

Preparation is therefore a central part of successful A | B tests that you should not underestimate. Only if you know what you want to test and what impact the results will have, you can really continuously improve your website.

Don't forget to segment

Also, don't forget to segment your users before testing. Would you like to collect information on all visitors - or only on new or returning users? Would you like to test all end devices or do you differentiate between desktop and mobile? You also have to keep in mind the impact these decisions have on the size of your sample. Because, as I said, this must always be large enough to ensure a significant result.

Segments in Google Analytics

If you always lump all users into one pot and see them as a uniform mass - i.e. forego segmentation - you will never be able to see what is going on in the individual segments. It is possible that a test that is considered to have failed in its entirety has delivered very helpful results for an individual segment. Because new visitors behave differently than returning visitors; and age, gender, origin and the device used also play an important role. It would be a shame if you overlooked these results and wasted potential with them.

External factors are ignored

As I said, in order to complete a test with a significant result, you need a certain number of participants and a certain number of conversions. If your site has a lot of visitors, a test can sometimes have a winner in less than a week. But one thing you should never forget about A | B testing is the fact that external factors can have a significant impact on the test results.

Most people live to a certain rhythm. Regular activities such as work, study, leisure activities, and family life determine what we do on certain days of the week. This rhythm has a great influence on the conversion rate of your website and must be taken into account in the A | B test. Holidays or events (such as winning a soccer world championship) also have an impact on user behavior. In many cases it makes sense to take a look at the calendar in order to carry out the test at a “normal” point in time. Things like school holidays, which can distort the result, are usually known in advance and can therefore be taken into account in good time.

In order to minimize the influence of external factors on your A | B tests, you should let them run for at least two weeks (and thus also two weekends) - and in a time span that is not very different from other intervals in the year . It is important that you also take seasonal features and the weather into account. Because depending on your industry, user behavior in snowstorms may be different than in bright sunshine.

Only if you allow all these imponderables to flow into your knowledge will you know for sure that your test is also armed against any fluctuations caused by external factors.

Conclusion: Better safe than sorry

So don't forget to keep these things in mind when doing your A | B tests. Nothing is more annoying than investing a lot of time, money, and energy into a test that, in the end, cannot use the results. Therefore, it is better to be safe than sorry.

Remember: every website is unique and therefore there is no checklist for the perfectly optimized website. So do extensive research with regard to your industry and target group (e.g. with the help of the Limbic® Map) instead of blindly plunging into a poorly prepared A | B test. Defining resilient goals also increases the likelihood of a useful result.

Start by asking the people who are already using your website regularly. Heatmaps and eye tracking are ideal here to obtain in-depth information about the problem areas on your site. With the help of this qualitative data, you will be able to understand the problems of your website and, based on this, formulate the hypotheses for your tests.

Otherwise you only have to be patient. So always plan enough buffer in case unforeseen events mean that you have to extend your tests. If you observe these points, you have already eliminated the most avoidable sources of error for your A | B tests.