Statistical significance

Statistical significance is a judgement about whether the difference between two variants is larger than random variation comfortably explains. Teams use it as the stopping rule for tests, so a shaky grasp of it produces confident rollouts of changes that do nothing. Two practical cautions: checking results daily and stopping at the first significant reading inflates false positives badly, and a significant result on a tiny effect can still be worth less than the work of shipping it.