As part of their incubation at Look Forward, our start-ups benefit from our partnership with A/B Tasty, who offers them its services and hosts workshops in order to help them improve their website or application performances. But what is A/B testing exactly? When should you use it? Is there any best practice? So many questions we are going to answer right now.

· What is A/B Testing?

A/B Testing is the fact of testing a version A versus a version B on a significant traffic. It helps measuring a feature performance, on a website or an app, on two different panels: one on which we display the feature, and another one on which we don’t. The collected data on both panels’ reaction enable one to state whether this feature should appear or not. It could be a color, a button, a wording, and so forth. The operation final goal is to improve the website and optimize the conversion rate.

· Why using it and when?

A/B Testing is used to take an enlightened decision, based on reliable data, and not only on intuition. Testing a feature on a small part of the traffic and getting concrete data on its performance, allows to reduce the margin of error. Before A/B testing, there was no mean to concretely measure whether changing an element on the website would have a positive or negative impact. Of course, there were rules on ergonomics to build websites, but they would not be based on reliable information.

Because a large traffic is needed to get a reliable feedback, A/B tests usually take place once the company reaches a certain maturity. However, before A/B testing, it is recommended to realize a general study on user experience, meaning to analyze visitors’ navigation, where does it start and end, or where do you lose them for example. These insights then allow to implement an A/B test on the facts observed during the study, to confirm or disconfirm them, and take a proper decision.

· Are A/B Testing campaigns expensive?

Many factors play a role: the number of websites or apps concerned by the test, or whether you operate on several countries for example. The cost is proportional to the company size. A company with only one website available in one country will be able to A/B test for a rather cheap price. The more the company will be developed, the more expensive it will get.

· Should A/B testing be a priority for a start-up?

As far as the website or app has an important enough traffic that provides reliable data, A/B testing can be done pretty much from the beginning. It can also be carried out on a landing page, to test wording, esthetics, and so on. To sum up, whether at the really beginning or in the middle of your start-up adventure, starting an A/B test campaign should be determined with regards to your specific case.

· Is A/B test relevant to any activity? Is it more useful for B2B or B2C businesses?

During a campaign, you can really set different objectives, for any content. A/B testing is therefore relevant for content website, as well as for e-commerce, or landing pages. As an example, medias use A/B testing plenty. Their KPI will be more on the amount of articles shared, newsletter or member subscriptions. Regarding e-commerce, more business oriented, the emphasis will be more on conversion rate. A/B testing can be used for any kind of activity. However, when a B2C website will get feedbacks quite quickly, a campaign on a B2B website will take longer since the traffic is lower.

· DO’s and DON’Ts

Within Look Forward incubator, Karim Mauger, Marketing analyst and Audrey Ngoue, User experience analyst at Showroomprivé, put their expertise at our start-ups’ disposal by providing pieces of advises together with participating to the workshops organized on the matter. In exclusivity for Look Forward’s blog, they provide us with their vision of DO’s and DON’Ts with regards to A/B testing.

DO’s :

• Tests on wording: these campaigns don’t ask too much work and may have a huge output in terms of subscription rate, or sales rate for companies debuting. Typically, « Sign up for free » will have a more positive impact than just « Sign up ». These changes are rather cheap to implement because they don’t require new process or new suppliers, and they are essential to begin with.
• Keep track of every tests you may have done. After a while, you may forget what have been realized, and have not. You may also want to go back on the results of a test. Always keep in mind that someone may collaborate with you, or you may leave at some point: these persons will need to understand what have been going on in the past.

DON’Ts :

• Engage your customer too much when calling to action. Words « Order » or « Buy » tend to have less success than « Add to the basket » or « Save for later ». The first two are too engaging, when the last two imply that there are still steps and time ahead of any purchase.
• Stuck to « Best practices » : there are always exceptions to the rule. For example, some colors may seem unexpected in terms of graphics, but totally work because they are reassuring, or eye-catching. A/B testing also is a consumer psychology and behavior study. That is also why it is an interesting tool when working internationally: every culture have its own references, and testing different hypothesis allow to understand them better.
• Test everything, and go too fast: there are a lot of implications behind each test, it is important to think deeply about the target, the benefits to get from the test, and plan ahead of time.
• Jump to conclusions too quickly. Once again, the more traffic you get, you more reliable the collected data will be. We may think that trends we identify at the beginning reflects the truth, but some time is necessary for results to smooth and provide a correct insight.

• Some advises to start with A/B testing:

To conduct an A/B test means respecting some steps :

1. Analyze your website and how users navigate to get some tests’ ideas.
   Realize a benchmark, and ask your collaborators in-house if they have any remark.
2. Surround yourself with the best: get a solution that answers to your specific needs. As an example, start-ups need more support and advices. There is a whole panel of solutions available, such as our partner A/B Tasty, implementing tests as well as providing support.
3. Go from a detailed hypothesis in order to stay focus when it will be time to analyze. In that sense, you should define a KPI, think about the test’s constraints and limits.
4. Check the test process yourself, to insure it actually works.
5. Your test should not last more than two to three months (depending on the traffic), so that users are not considered twice in the results and alter them.
6. Start the analysis only after getting reliable data.
7. There is no failure. Since it is a test, you are never sure it will succeed. Even negative campaigns teach you a lot. For example, you may lose in terms of quantity but win in quality.

 

Nothing is set in stone, and A/B testing is a good mean to better perform on the long-term, while listening to your website’s users. We would like to thank Karim Mauger, Marketing analyst, and Audrey Ngoue, User experience analyst for their insights on the matter!