The popular argument: a popup costs you money because it hands a discount to people who were already going to buy. If that is true, hiding the popup should protect revenue per visitor. Simple Modern put it to a real test.

A true 50/50 holdout ran across Simple Modern's traffic for 30 days, across hundreds of thousands of sessions. Half of all visitors saw the popup. The other half saw nothing, with no way to subscribe through a popup at all. Mobile and desktop were measured separately. Order conversion rate reached 99% statistical confidence.

Visitors were split 50/50 between the popup and the holdout. We compared the two groups on:

• Order conversion rate: orders per visitor, the headline metric
• Revenue per visitor (RPV) and average order value (AOV)

The popup won, and it wasn't close. Order conversion rate +10.4% overall (desktop +14.2%, mobile +9.2%) at 99% confidence, and revenue per visitor +29.5% desktop, +8.9% mobile. Put another way, hiding the popup would have cost desktop roughly two-thirds of its same-session revenue. The discount cost is real but small: the popup group's AOV ran about 6% lower early on, settling to roughly flat on mobile (~0.2%), while desktop AOV came in higher (+12.6%). Even carrying that discount, the popup still won on conversion and revenue per visitor, before a dollar of the future subscriber revenue this same-session test never counted.

List growth is the other half of the win. The holdout captured zero popup subscribers, by design. The popup group opted in at 3.8% email and 3.0% SMS, so every one of those subscribers is pure incremental list growth, future revenue this same-session test never counted.