Simulate A/B-test

This calculator allows you to model A/B test outcomes by simulating a synthetic user population. It is designed to show how unobserved differences between user segments impact the outcome of your A/B-test experiment.

By running multiple simulations under varying demographic assumptions, you can see exactly how your hypotheses impact the final results. Beyond simulation, the tool use AI to automatically summarize your findings.

Ultimately, it serves as a validation tool: by comparing real-world A/B test data against the predicted outcome based on your assumptions, you can see where they align or diverge. If your actual test results don't align with your initial assumptions, simply head back to the drawing board, adjust the parameters, and simulate a new scenario.

It’s the perfect sandbox to stress-test your hypotheses before you ever go live.

Instructions:

• This tool simulates an A/B test using two variants: a control and a variant 1. The simulation models a population of n users, which you can adjust via the sample size setting below. In the control group, the baseline performance is determined by the "Metric baseline" parameter, which is also customizable.
• You can configure up to six distinct user segments by defining how the new feature impacts the metric. Simply set the score to "Love" if you expect a positive reaction, "Neutral" for indifference, or "Hate" if you anticipate a negative response.
• You can weight each segment to accurately reflect its proportion within your real-world user base.
• Once configured, click Run Simulation to execute the synthetic A/B test and view the AI-generated summary of the results.
• Be patient; the A/B simulations will take some time, and then AI will summarize the results for you.