Building my own A/B testing toolkit

For a long time, I treated A/B testing like something you “run” and then “check the dashboard.” If the p-value was small, you shipped. If it wasn’t, you moved on. That approach works ... until it doesn’t. The more I read about experimentation in real product teams, the more I realized there’s a big gap between: • knowing the basic idea of A/B tests, and • building systems and instincts that make experiments reliable at scale. So I set myself a project: build a small A/B experimentation toolkit from scratch, and then build a simulation lab to validate it. This post is a high-level story of what I built, why I built it, how I planned it, and what I learned.