# A/B Testing

# 1.What is an A/B test?

A/B test is an experiment method that compares two or more different strategies, designs, or variations to determine which one has a greater impact on users or business. A/B test can help identify which group receives better user feedbacks, and has higher conversion rates, or better business performance.

In A/B test, the objects being tested will be randomly and equally divided into two (or more) groups: a control group and one or several experiment groups. The control group uses the current strategy or design, while the experiment group uses the modified strategy or design. Users in both the control group and the experiment groups are exposed to respective experiences at the same time.

By comparing user feedback, behavior, or business metrics between groups, it is possible to determine which group is more advantageous. Experiments can be used to validate your content changes, such as different push notification contents, different prices for game bundles, ways of reaching users, or functional changes.

The goal of A/B test is to make decisions based on actual data and user feedback. Through continuous testing and optimization, it gradually improves user experience, increases conversion rates, and enhances business revenue.

# 2. Overview

TE Engage Module supports A/B Test. By configuring multiple groups of content strategies for the same operation task, it helps operation specialists make scientific decisions on the optimal strategy to improve the target conversion effect.

# 3. Usage Instructions

# 3.1 Turn on A/B test

In the "Tasks" module, click on the "+ Create Task" button to enter the campaign creation interface.

At this point, you can click on the "Turn on A/B Test" button in the Configuration section, which will enable the split testing settings for this task.

# 3.2 Choose Experiment Type

You can choose the type of experiment, Split-flow test or Horse-racing test.

# 3.2.1 Split-flow Test

Randomly assign the target audience into multiple experiment groups in proportion, and push simultaneously to compare the effect of the strategies.
The split-flow test is adapted to the scenarios which only need to compare the effect of pushing stategies by group, and is not adapted to the ones that need subsequent full audience pushing.

# 3.2.2 Horse-racing Test

Currently, only triggered tasks support horse-racing tests.

Before pushing with the final strategy, you can run the test with multiple strategies on a small scale to find the best strategy, and then push to all audience with it.
Horse-racing test is suitable for the scenarios with full users pushing demand after the experiment.
The horse-racing tests support setting the automatic pushing strategy before and in the middle of the experiment, so that users can push the content of the winning group to all audience after the winning group is obtained by the experiment.

# 3.3 Split-flow Test Configuration

Sampling Ratio

After turning on the A/B test, you can manually set the test sampling ratio, also known as the gray sampling ratio. This refers to randomly selecting a certain percentage of users from the target audience to participate in the A/B test, while the remaining users will not be reached.

The sampling ratio can be selected as any integer value within the range (0, 100]. You can set it by dragging the scroll bar or entering the value manually.

Experiment Schedule

The experiment schedule of split-flow tests is consistent with the task schedule.

Winning Metric

To continue, you should also set the winning metric for this A/B test. You can either align it with the "primary goal" of the operation task, or select "Click-through rate". Please note that for "click-through rate", you need to configure the "clicked" event in the Funnel Setting of the corresponding channel in channel management.

Experiment Groups Configuration

After completing the above settings, you will also need to configure the push contents for the control group and experiment groups in Message.

The control group will use the current strategy or design, while the experiment group will use the modified strategy or design. Both groups of users will receive different versions of the content at the same time.

Control Group

Each A/B test typically includes one control group, and you can choose whether to reach users in the control group.

When you choose to reach users in the control group, the control group will run the task simultaneously with the experiment groups, which allows you to observe feedback from users on different contents during the same period.

On the other hand, if you choose not to reach users in the control group, it can be used to evaluate the effectiveness of your operation strategies on users. You can observe whether the experiment group shows a significant improvement in target conversion compared to the control group.

Experiment Group

You can set up to 4 experiment groups. You can click on the gear icon on the right side to configure experiment groups. This allows you to add or delete experiment groups and customize their names. Additionally, you can use the scroll bar to allocate sample distribution to each experiment group. The system will estimate the number of users of each experiment group based on your target audience, experiment sampling ratio, and sample distribution allocation.

# 3.4 Horse-racing Test Configuration

Sampling Ratio

The sampling ratio can be selected as any integer value within the range (0, 100]. You can set it by dragging the scroll bar or entering the value manually.

Experiment Schedule

Horse-racing tests are generally conducted during the prior period of the entire task schedule.

The horse-racing tests will start consistent with the tasks , and the end time of the experiment shall be set manually. The end time of the experiment shall not be later than the end time of the task. When the horse-racing test is over and the winning experiment group is obtained, the user can use the winning strategy to push to all audience in the remaining task schedule.

Winning Metric

Experiment Groups Configuration

After completing the above settings, you will also need to configure the push contents for the control group and experiment groups in Message.

Control Group

Each A/B test typically includes one control group, and you can choose whether to reach users in the control group.

Experiment Group

Select the strategy to push to all audience

Users can choose to turn on the switch to automatically select the strategy content and push to all audience with it. When the switch is turned on, at the end of the horse-racing tests, if the winning group is obtained by the experiment, the winning strategy will be automatically pushed to the all audience. If no winning group appears, the audience will be pushed with the default strategy set by the user.

However, if the switch is not turned on , you can go to the "A/B Test Report" page to set it manually after the experiment.

# 4.Experiment Status

Tips: The "Operation Task Status" is the parent status of the "A/B test Status". The status of the operation task will affect the status of the test. For example, if the operation task is paused, the test will also be paused. If the operation task is finished, the experiment will also be finished, etc.

The operation task status includes 'Draft', 'Pending', 'Running', 'Paused', 'Finished', and 'Rejected'.
The test status describes the current state of the A/B test and includes the following:
- Not Started: The experiment in which the operation task is in a non-finished state and has not yet been pushed.
- Experiment Canceled: The experiment that has been closed before being pushed when a operational task becomes 'Finished'.
- In Progress: The experiment is still within the target conversion window while the operation task is in the 'Running' status.
- Paused: The experiment also enters the 'Paused' status when the operation task is in the 'Paused' state.
- Data Collecting: The operation task is in the 'Finished' status, but the target conversion window has not ended yet, and test results are still reporting and being collected.
- Report Completed: The operation task is in the 'Finished' status, and the experiment has reached the target conversion window. The A/B test report is completed and the experiment will be officially closed.
In addition, in the "horse-racing test", if the user turns on the switch to automatically pushing, the selected strategy will be pushed to all audience immediately after the experiment. If the user chooses to do it manually, the experiment will enter the status of "Push waiting" after the expiration of the experiment.

# 5. A/B Test Report

In the A/B test report module, users can visually compare the result data of the control group and each experiment group. This not only allows the team to determine the winning strategy based on statistis but also enables the team to dynamically observe the data trends throughout the entire experiment.

# 5.1 Report Access

Once the operation task for the A/B test has started, users can view the test result data on the "A/B Test Report" tab of the operation task details page.

# 5.2 Report Introduction

# 5.2.1 Basic Information

The basic information in the report includes the experiment duration,status, winning metric, push succeed users, and number of experiment groups.

Experiment Duration: The time from the start of an experiment to the completion of experiement report.
- Default Start Time: The start time of the experiment.
- Default End Time: Today, you may set the end time as late as the report completion time.
- The time filter takes effect on the full experiment report, including experiment group data, winning group determination, and the line chart.
- The experiment duration cannot exceed 90 days.
Experiment Status: to describe the current status of the A/B test. For more details, refer to the experiment status description.
Winning Metric: the key metric for identifying the winning group in the experiment.
Push Succeed: The total number of successfully pushed users covered by all push groups, including the control group and experiment groups, in the A/B test.
Number of experiment Groups: The total number of control and experiment groups in the A/B test.

# 5.2.2 Experiment Statistics

Push Succeed: The total number of successfully pushed users for each experiment group.
Number of Achieved Users: The number of users in each experiment group who achieved the 'winning metric'.
Metric Achieved Rate: The ratio of Number of Achieved Users/Push Succeed.
Improvement Rate: This rate refers to the rate of change in the experiment group's metric compared to the control group's metric.
Experiment Winner Determination Rule: The experiment group with the highest improvement rate is determined as the winner. If there is no experiment group that significantly improves the achievement rate compared to the control group, there is no winner in the experiment.
Confidence Interval of Improvement Rate: This refers to the range in which the true value of the improvement rate may fall. A two-sided test with a significant level of 5% is used.

Calculation Logic:

Experiment Group Metric: M(experiment), Experiment Group Sample Size: U(experiment)
Control Group Metric: M(control), Control Group Sample Size: U(control)

First, calculate the standard deviation for the samples of the experiment and control groups. Since it is a ratio metric:

Calculate the improvement rate and lift:
<div class="equation" data-eq='Improvement Rate = {M(experiment) \over M(control)} - 1

Based on the above calculations, we can obtain:

Here, 1.96 is a variable that is converted based on different confidence levels, with a default of 95% confidence level.

Conclusion:
If the lower limit in the experiment group is negative and the upper limit is positive: Not significant.

If the lower limit in the experiment group is positive and the upper limit is positive: Significantly increased.

If the lower limit in the experiment group is negative and the upper limit is negative: Significantly decreased.

Further Actions
Supporting viewing the push content of the experiment groups.

When the experiment report produces a winning group, you can directly create a new operation task for pushing to all users by clicking the "Copy" button behind each group.

# 5.2.3 Trend of Achieve Rate

In the data details, you can see the push details and target achievement status for each experiment group from a batch perspective.

And it also supports viewing the trend chart of the achieve rate on a daily basis.

# 5.2.4 Configure Push Strategy for Horse-racing Test

In the task setting of horse-racing tests, whether the user turns on the switch to automatically select the strategy will affect the way of pushing after the experiment.

Adjust Strategy

Before the experiment enters the Data Collecting / Report Completed status, you can adjust the strategy and re-select the group that will finally be pushed to all audience.

Tip: The strategy cannot be adjusted for experiments that have ended.

Auto Pushing

When the user chooses to turn on the switch to automatically select the strategy, after the horse-racing test, the full audience pushing will be automatically carried out according to the strategy set by the user.

Push To All Audience

If the user does not turn on the switch switch to automatically select the strategy, after the horse-racing test, the task will enter the state of "Push waiting", and the user needs to select the experiment group and manually push to all audience.

← Task Effect Analysis Creating Campaigns →