Also called “split testing” or an “online experiment” is a live experiment conducted on a portion of a user base. Each version is called a “variant”.

• Defend a change by using data within the confidence interval. Preferably in the positive range, unless you think the experiment performed poorly than the design, and see the results of the change by actually releasing the feature.
• Beware of p-hacking—or fishing for correlation. If you are conducting a test for specific parameters, don’t be alarmed if you see a favorable result of another thing. That is, looking for values of stuff we never tested.