Running experiments in marketing and digital can be one of the most effective ways to improve your campaign, your product, your customer experience and your business, but it requires a structured approach to ensure reliable results. This guide will take you through a detailed process for “test and learn”: designing, conducting, and interpreting marketing and digital experiments, ensuring you and your team can make data-driven decisions with confidence.
1. Defining Your Objective 🎯
Before you start, you need a clear objective for your experiment. This objective should be tied to a specific business goal, such as increasing conversions, understanding which offer might lift more sales, or drive greater trial.
Example Objective:
• Objective: “Increase the conversion rate on our landing page from 3% to 4%.”
Why It’s Important:
A well-defined objective keeps the experiment focused and ensures it aligns with broader business goals. It also makes it easier to measure success.
2. Formulating a Hypothesis 💡
A hypothesis is a clear, testable prediction of what you expect to happen and why. It’s based on observations or insights from past performance data.
Example Hypothesis:
• Observation: “Customers seem to drop off when there’s too much text on the landing page.”
• Hypothesis: “If we reduce the amount of text on the landing page, then the conversion rate will increase because users prefer quick, digestible information.”
Why It’s Important:
A good hypothesis provides a clear focus for your test and sets expectations for the outcome. It should always be specific, measurable, and directly related to your objective.
3. Identifying Variables 🔍
In any experiment, there are two critical types of variables:
• Independent Variable (IV): This is the element you change in your test.
• Dependent Variable (DV): This is the outcome you measure.
Example:
• Independent Variable: The amount of text on the landing page (current vs. reduced).
• Dependent Variable: The conversion rate (percentage of users who complete a purchase or sign-up).
TIP:
AI tools like ChatGPT can help you brainstorm multiple hypotheses and/or Independent Variables (IVs). For example, input your goal and Independent Variable, and ask for alternative ways to approach your experiment. This can uncover test ideas you might not have considered, like “If changing the colour scheme doesn’t work, what layout modifications could I test to reduce bounce rate?”
Why It’s Important:
By identifying the independent and dependent variables, you can measure the impact of the change and ensure the results are meaningful.
4. Setting Up Control and Test Groups 🧪
To properly measure the effect of your changes, you need a control group (the group that sees no changes) and a test group (the group that experiences the change). The key is to compare how these two groups perform.
• Control Group: This group sees the original version (e.g., the current landing page).
• Test Group: This group sees the modified version (e.g., a sign-up button with a clear value).
Randomisation:
• Users should be randomly assigned to each group to avoid bias. Random assignment ensures both groups are equivalent, and the only difference is the change being tested.
Example:
• Control Group: 10,000 users see the original landing page.
• Test Group: 10,000 users see the landing page with reduced text.
TIP:
Use AI-powered tools like Google Optimize or VWO, which automatically split traffic between control and test groups, eliminating manual setup errors.
For multivariate testing, you can input variables into ChatGPT and ask it to suggest combinations of elements (e.g., CTA, headline, image layout) most likely to affect your conversion rates based on best practices from other industries.
Why It’s Important:
Having a control group lets you understand what would have happened without the change, providing a baseline for comparison. Randomisation ensures fairness and reliability in your results.
5. Determining Sample Size 📊
To ensure your results are reliable, you need a large enough sample size in both the control and test groups. A larger sample size reduces the effect of random variation and ensures your results are more accurate.
Factors to Consider:
• Traffic Volume: How many users visit your landing page during the test period?
• Significance Level: Aim for a p-value of <0.05, meaning there’s a 95% chance the result isn’t due to random variation.
• Power: High power reduces the risk of failing to detect a real effect.
Example:
• Sample Size: 10,000 users per group to detect a meaningful change with sufficient confidence.
Why It’s Important:
If your sample size is too small, you might not be able to trust the results. Testing on too few users can lead to random variations that don’t represent real changes in user behaviour.
6. Choosing a Testing Method 🛠️
Your testing method will depend on what you’re trying to achieve:
• A/B Testing: Compare two versions of the same element (e.g., original vs. reduced text).
• Multivariate Testing (MVT): Test multiple elements at the same time (e.g., text and CTA button changes). This requires more traffic but can give deeper insights.
Example:
• Testing Method: A/B Testing – comparing the original landing page (A) with the reduced text version (B).
Why It’s Important:
The right testing method ensures you’re isolating the effect of the change and getting reliable data. A/B testing is typically best for straightforward comparisons, while multivariate testing is more complex.
7. Setting the Testing Duration ⏳
The length of your test should be based on your traffic volume and how long it takes to gather enough data to reach a conclusion.
Factors to Consider:
• Traffic Volume: How many users do you need to reach the target sample size?
• Buying Cycle: How long does it typically take users to convert after visiting the page?
Example:
• Duration: 2 weeks, ensuring that enough data is collected over different days and user behaviours.
Why It’s Important:
Running the test for too short a time could give unreliable results while running it too long may introduce external factors (e.g., seasonal changes) that could skew the data.
8. Implementing the Test 🚀
Now that everything is designed, it’s time to set up the experiment in your A/B testing tool (e.g., Google Optimize, Optimizely, or Adobe Target).
Steps to Follow:
• Set Up Variants: Create the new version (B) and leave the control (A) as is.
• Assign Traffic: Randomly split traffic between the control and test groups.
• Launch the Experiment: Start the test and monitor for any immediate issues.
Why It’s Important:
A properly set-up test ensures that data is collected accurately and that the results will be actionable and reliable.
9. Watch Outs: Ensuring Robust Results ⚠️
To ensure the results and recommendations are robust, there are a few common pitfalls to avoid. These “Watch Outs” will help you make accurate inferences and avoid misleading conclusions.
1. Causality vs. Correlation
Just because two things happen at the same time doesn’t mean one caused the other. For example, if both conversion rates and a website’s ad traffic increase, it doesn’t mean the ad traffic caused the conversion increase—it could be an unrelated factor.
Solution:
Focus on isolating the independent variable (the change you made) to ensure it’s the most likely cause of the outcome. Avoid drawing conclusions from coincidental patterns.
2. % Change vs. % Action
It’s important to distinguish between the percentage change and the percentage of total action. A 10% increase in conversion rate sounds impressive, but if your original conversion rate was only 1%, that’s just an increase to 1.1%—not as impactful as it seems.
Solution:
When reporting, always clarify whether you’re talking about the relative percentage change or the absolute percentage action to give more context to your results.
3. Basis Points vs. Percentage Increase
Basis points (bps) are another important distinction. A change from 2% to 3% is a 1 percentage point increase, but it’s also a 50% relative increase. Using basis points helps avoid exaggerating results.
Solution:
Use basis points to avoid inflating the perceived impact when the changes are small. For example, report a 100 bps increase instead of a 50% increase when moving from 2% to 3%.
4. Consistency of Reporting
Inconsistent methods of calculating and reporting metrics can lead to confusion and misinterpretation. Ensure that all reports use the same baseline, methods, and time periods for comparison.
Solution:
Standardize your metrics, ensure the same time frames are compared, and avoid changing methodologies mid-test. This keeps reporting consistent and easier to interpret over time.
Why It’s Important:
Addressing these issues will prevent flawed recommendations based on misleading data, keeping your experiments credible and actionable.
10. Analysing the Results 📈
Once the test is complete, it’s time to look at the data. Focus on comparing the key metric between the control and test groups.
Example Analysis:
• Control Group Conversion Rate: 3%
• Test Group Conversion Rate: 4%
Step-by-Step Process:
1. Assess the Impact: Is there a meaningful difference in your key metric (e.g., conversion rate)?
2. Statistical Significance: Is the result statistically significant (p < 0.05)?
3. Avoid Common Pitfalls: Ensure you’re not confusing correlation with causation or misinterpreting small sample size results.
TIP:
With platforms like Mixpanel, AI can segment user behaviours and create reports predicting how slight changes in your experiment could impact your key metrics, like conversion rate or time on page.
Why It’s Important:
Data analysis ensures you’re drawing accurate conclusions from your experiment and not jumping to false inferences.
11. Communicating Your Findings 🗣️
Once the experiment is analysed, you must effectively communicate the results to your team and stakeholders.
How to Communicate:
• Summarise the Experiment: “We tested whether reducing text on our landing page would increase conversions.”
• Report the Results: “The test group showed a 1% increase in conversion rate compared to the control group.”
• Offer Recommendations: “Based on these results, we recommend applying the reduced text format to all landing pages.”
Why It’s Important:
Clear communication ensures that everyone understands the experiment’s results and what action should be taken next.
12. Final Tips for Success 🏆
• Keep It Simple: Focus on one change and one metric at a time.
• Be Patient: Let the experiment run its course before making conclusions.
• Iterate: Use the results to refine your approach and plan future experiments.
• Welcome a broken heart 💔: Being wrong is great. That way, you’ll learn. Don’t be in a hurry to be right.
Remember:
Marketing experiments are about learning and improving. Even if the test doesn’t deliver the expected results, it still provides valuable insights that can guide your next steps.
Free Download / Resource: I’ve created a simple Marketing and Digital Experiment Tracking Table in xls format for free download. Enjoy!
