# A/B Testing Automations

When it comes to engagement, even small variations to your automation emails can have a big impact. With A/B testing, you have a way of determining what your participants are positively responding to, giving you better insights to optimise your automation emails.

### What Is A/B Testing? <a href="#what-is-ab-testing" id="what-is-ab-testing"></a>

**A/B testing is a way to compare two versions of a single variable** (in this case, the subject line of an email)**, by testing a Subscriber’s response to variable A against variable B.**&#x20;

In layman terms, it means that you create two subject lines for an email (Version A and Version B), and ReferralHero will send a random version to each subscriber. This ensures that the test is randomised and holds statistical significance.

### How to run an A/B test <a href="#turn-on-ab-testing-in-your-convertkit-account" id="turn-on-ab-testing-in-your-convertkit-account"></a>

**ReferralHero allows you to A/B test the Subject Line of your Automation emails**.\
To enable A/B testing for an Automation email:

* Go to your campaign dashboard > Edit Campaign > Automations > email you want to use for the A/B test (e.g: the Welcome email)
* Switch the toggle **A/B Test Subject Line**
* A second field for Version B of your subject line will appear
* Enter an alternate subject line
* Save your changes

<figure><img src="https://363135598-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LsuqexOLPOWiUrWg_Ko%2Fuploads%2F5DPmpIFCprGidY2kT6Ee%2Fimage.png?alt=media&#x26;token=e506f9a5-a95e-4ad3-896a-e49f1d2dc699" alt=""><figcaption></figcaption></figure>

### How to stop an A/B test

To stop an A/B test, simply switch off the toggle **A/B Test Subject Line** and save the changes.

{% hint style="warning" %}
**IMPORTANT**\
When you stop a running A/B test, all the stats about the test will be lost. Before you cancel it, make sure to check the reports located in your account, Analytics -> Automations.
{% endhint %}

### How to determine a winner

After you start an A/B test for an email, ReferralHero will automatically begin collecting stats. You can view the report at any time by navigating to *Analytics > Automations*.

<figure><img src="https://363135598-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LsuqexOLPOWiUrWg_Ko%2Fuploads%2FKWwb2DH3IoBMMVQ42T7M%2Fimage.png?alt=media&#x26;token=acc27e43-6cdd-42b4-98a1-894741edd528" alt=""><figcaption></figcaption></figure>

Since we are only A/B testing the subject line, the number we are interested in is the open rate. In this case, Version A has a 77.8% open rate versus a 60% open rate for Version B. Version A is the winning version. You should stop the A/B test and use Version A’s subject line as your main subject line.

**When should you be confident enough to pick a winner?**&#x20;

It's really up to you to decide when an A/B test produces significant results, but we think you can confidently pick a winner when you have *sent at least 1K emails* for each version and the *difference between the open rates is at least 10%*.