One of the interview questions for data analysts: Explain the Central Limit Theorem.
Let's start with a sad fact - if you don't want to know about CLT, you don't need a data analyst in your company, because accidents are really accidental for you. In this scenario, statistics will be a sort of magic for you, and you won't ever understand how your changes made a real statistically significant impact.
But also, you're losing responsibility and can manipulate data as you want. You don't have to approve your words; you can take any comfortable date range and interpret the A/B testing results - nobody can prove you wrong: the world is manipulated by God, Destiny, or Luck.
But this is not the way we, data analysts, see the world. And the main understanding comes from the Central Limit Theorem and the Law of Large Numbers.
Example Scenario: Gym Equipment
Imagine you visit some gym and see a picture attached to one of the exercise machines. It seems many people use weights between 40 and 50 kg. You want to watch if this is true, choosing samples of random people. You mark the exercise machine with invisible inks to see which weights they prefer (let's also consider they don't change their weights for simplification, but in the end it doesn't matter).
Generating Normal Distribution
Let's generate a normal distribution of the exercise machine usage with:
- Mean (Average): 45 kg
- Standard Deviation: 15 kg
- Sample Size: 1,000,000 people
Average: 45.000705
10 first people results: [50 30 55 60 15 25 45 40 45 30]
First Observation: Small Sample (3 people)
Results: [50 30 55]
Average: 45.0
The average matches our expected value, but let's check the next group...
Second Observation: Next 3 people
Results: [60 15 25]
Average: 33.3
Problem: Some mess is coming - the results are anomaly different. Let's take more people in our sample.
Sample of 10 people
Results: [45 40 45 30 60 55 45 60 50 30]
Average: 46.0
Next 10 people
Results: [50 30 60 45 40 35 65 45 40 40]
Average: 45.0
It seems our averages are closer now, and one of them even has the average of 45. What if we take even more people?
100 people sample
Average: 44.4
Next 100 people
Average: 44.7
Law of Large Numbers
Key Insight: It seems our average is getting closer to our hypothesized mean of 45 – and this is the Law of Large Numbers.
It states that the average of results obtained from a large number of independent random samples converges to the true value, if it exists. So, the more people we include in our experiments, the closer our sample mean will be to the true mean.
Central Limit Theorem in Action
But what will happen if we take random samples of 100 people and put their means on a frequency chart (X axis represents the value, Y represents the frequency every average appeared)?
Observation Results:
- 10 samples: Distribution starts taking shape
- 100 samples: More bell-shaped distribution
- 1,000 samples: Clear normal distribution emerges
Central Limit Theorem Conclusion
You see, our distribution of averages is becoming more bell-shaped, symmetric, and concentrated around the mean, like a normal distribution.
This is the Central Limit Theorem. It states that, under appropriate conditions, the distribution of a normalized version of the sample mean converges to a standard normal distribution.
PPC Marketing Example: Basics of A/B Testing
Now, let's connect this to something more practical – for example, PPC marketing. Imagine you are running Google Ads campaigns and want to know whether changing the color of your "Buy Now" button will increase your conversion rate. You can't just guess – you need data, and this is where A/B testing comes in.
A/B Testing Setup
Suppose you have two versions of a landing page:
- Version A: Original button (control)
- Version B: New button color (treatment)
You randomly split your website visitors so that half see Version A and half see Version B. After a week, you collect the number of conversions for each version.
| Version | Visitors | Conversions | Conversion Rate |
|---|---|---|---|
| A | 500 | 50 | 10% |
| B | 500 | 60 | 12% |
The Critical Question
At first glance, Version B seems better – but is it statistically significant? Could the difference of 2% just be due to random chance?
How CLT Helps in A/B Testing
This is exactly where the Central Limit Theorem (CLT) helps. The CLT tells us that if we repeatedly take random samples of visitors and calculate the conversion rate for each sample, the distribution of these sample means will approach a normal distribution. This allows us to:
- Calculate the standard error of the conversion rate.
- Build a confidence interval around the sample mean.
- Test hypotheses – for example:
- Null hypothesis: Version B is no better than Version A.
- Alternative hypothesis: Version B increases conversion rate.
Drawing Conclusions
If the confidence interval for Version B does not overlap with Version A, we can conclude that the improvement is unlikely due to chance – it's statistically significant.
In simpler terms: A/B testing + Central Limit Theorem = knowing whether your PPC change actually works, rather than just guessing.
Key Takeaway
So next time you tweak ad copy, button color, or targeting settings, remember: CLT is your friend. It turns random website clicks into actionable insights and prevents you from being misled by small random fluctuations in your data.