Building my own A/B testing toolkit

Why I did this

I wanted to get comfortable with the real questions that come up in experimentation:

• How do you explain results to people who are not statisticians?

• How do you know your method isn’t producing too many false positives?

• What do you do when your metric is “messy” (like revenue)?

• How do you decide how long to run a test?

• How do you build trust in your experimentation platform?

Also: I wanted a project that could stand on its own as a portfolio piece ... something you can open, run, and understand without needing a long explanation.

The plan :

I planned the work as two connected projects, each with a distinct job:

1) A/B Experimentation Toolkit ( ab-experimentation-toolkit)

This is the “toolbox.” It includes functions that take data from two groups (A and B) and return results in a consistent format.

My goal here was simplicity:

• a beginner-friendly structure

• clear documentation

• predictable outputs

• tests and CI from day one

2) Simulation Lab (experimentation-sim-lab)

This is the proof. It runs many simulated experiments and checks how the toolkit behaves.

This was the key idea:

It’s easy to write A/B testing code that seems correct. But, it’s much harder to prove it behaves correctly across many situations.

So instead of trusting formulas blindly, I used simulation to validate things like:

• How often do we get false positives when there’s no real effect?

• How often do we detect a real effect when it exists?

This second repo is what turned the project from “I implemented some statistics” into “I understand experimentation like a platform problem.”

The Execution

I treated this like a mini product build rather than a notebook exercise.

Step 1: Set up professional scaffolding early

Before writing much functionality, I set up:

• a clean package structure

• CI to run lint + formatting + tests

• pre-commit hooks so mistakes are caught early

It felt annoying at times, but it paid off. Once the workflow was in place, it became easy to make small improvements without breaking everything.

Step 2: Build the toolkit in small, testable pieces

I implemented the toolkit gradually, focusing on what real experimentation teams use day-to-day.I started with simpler metric types and built toward harder ones:

• “average value” type metrics

• conversion metrics

• ratio metrics (the stuff product teams love and stats people fear)

Each time I added functionality, I added:

• a result object that shows effect + uncertainty

• unit tests

• documentation written for a beginner

Step 3: Add platform guardrails, not just analysis

This was a big mindset shift for me. In real experimentation platforms, the job is not only to compute a p-value. You also need checks like:

• “Did we split traffic correctly?”

• “Are we running too many tests and fooling ourselves?”

• “Can we reduce noise so tests run faster?”

So I added features that make the toolkit feel like a tiny experimentation platform rather than a stats demo.

Step 4: Build a simulation lab to validate behavior

Once the toolkit existed, I built a separate repo that:

1. generates synthetic A and B data,

2. runs the toolkit’s analysis,

3. repeats this thousands of times,

4. measures how often results look significant.

This gave me a clear way to validate assumptions and build confidence in the toolkit.

What I learned (the real takeaways)

1) “Statistical significance” is the easiest part

The hardest part is everything around it:

• communicating results clearly

• building guardrails

• planning experiments

• handling messy metrics

• avoiding false confidence

2) Simulation is underrated

Simulation gave me a way to sanity-check my own thinking.

It also made the project feel grounded: not just math, but “does this behave the way a platform needs it to behave?”

3) Tooling discipline matters

CI, formatting, and tests are not just for show. They let you move faster without constantly breaking things.

And honestly, having clean repos made the whole project more enjoyable to work on.

Closing thoughts

This project was about building a system you can trust ... technically and culturally. If you’re learning experimentation and want a project that forces you to understand what’s going on under the hood, I highly recommend building something like this. The learning curve is real, but the payoff is even bigger.

If you’d like to explore the code:

• Toolkit repo

• Simulation lab