# A/B Testing Automations

When it comes to engagement, even small variations to your automation emails can have a big impact. With A/B testing, you have a way of determining what your participants are positively responding to, giving you better insights to optimise your automation emails.

### What Is A/B Testing? <a href="#what-is-ab-testing" id="what-is-ab-testing"></a>

**A/B testing is a way to compare two versions of a single variable** (in this case, the subject line of an email)**, by testing a Subscriber’s response to variable A against variable B.**&#x20;

In layman terms, it means that you create two subject lines for an email (Version A and Version B), and ReferralHero will send a random version to each subscriber. This ensures that the test is randomised and holds statistical significance.

### How to run an A/B test <a href="#turn-on-ab-testing-in-your-convertkit-account" id="turn-on-ab-testing-in-your-convertkit-account"></a>

**ReferralHero allows you to A/B test the Subject Line of your Automation emails**.\
To enable A/B testing for an Automation email:

* Go to your campaign dashboard > Edit Campaign > Automations > email you want to use for the A/B test (e.g: the Welcome email)
* Switch the toggle **A/B Test Subject Line**
* A second field for Version B of your subject line will appear
* Enter an alternate subject line
* Save your changes

<figure><img src="/files/pvlts9rGkrkAlUaoj6vX" alt=""><figcaption></figcaption></figure>

### How to stop an A/B test

To stop an A/B test, simply switch off the toggle **A/B Test Subject Line** and save the changes.

{% hint style="warning" %}
**IMPORTANT**\
When you stop a running A/B test, all the stats about the test will be lost. Before you cancel it, make sure to check the reports located in your account, Analytics -> Automations.
{% endhint %}

### How to determine a winner

After you start an A/B test for an email, ReferralHero will automatically begin collecting stats. You can view the report at any time by navigating to *Analytics > Automations*.

<figure><img src="/files/bRJc8TjeswaDUFUkB14d" alt=""><figcaption></figcaption></figure>

Since we are only A/B testing the subject line, the number we are interested in is the open rate. In this case, Version A has a 77.8% open rate versus a 60% open rate for Version B. Version A is the winning version. You should stop the A/B test and use Version A’s subject line as your main subject line.

**When should you be confident enough to pick a winner?**&#x20;

It's really up to you to decide when an A/B test produces significant results, but we think you can confidently pick a winner when you have *sent at least 1K emails* for each version and the *difference between the open rates is at least 10%*.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://support.referralhero.com/campaign-builder/automations/a-b-test.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.