The Power of Experiments: Decision Making in a Data-Driven World by Michael Luca & Max H. Bazerman

Author:Michael Luca & Max H. Bazerman [Luca, Michael & Bazerman, Max H.]
Language: eng
Format: epub
ISBN: 9780262043878
Google: l4LPDwAAQBAJ
Publisher: MIT Press
Published: 2020-03-03T00:00:00+00:00

From Data to Decisions

Should Alibaba continue the discount program? The answer depends in part on whose perspective we take. Sellers offering targeted discounts should be happy to see the discounts increase revenue (if not always profit). From their perspective, it is probably better to continue the program (though sellers might benefit from a no-discount regime, if it reduces competition).

Things are considerably murkier when we take the perspective of the platform. After all, the promotion doesn’t seem to increase overall spending; it changes the way the pie is split rather than increasing the size of the pie. The longer-term implications of the experiment are also unclear. As in many tech experiments, Alibaba measured short-term effects of an intervention, but it cares about long-term value. If customers’ strategic behavior (putting items in a cart and waiting for a discount) were to increase as awareness of the discount spread, the strategic behavior might undermine the intended targeting and cannibalize full-price sales.

Based on this experiment and internal discussions, Alibaba decided not to expand its shopping cart discount program. It’s great that Alibaba was experimenting (and that the experiment was well implemented), but there’s an important limitation to the map from results to the managerial decision—which is that Alibaba seemed to be asking an incomplete question. Alibaba’s experiment essentially asked, “Should this program exist?” rather than “How should this program be designed?” This is an important distinction, since the shopping cart discount may have seen small effects simply because it was poorly designed rather than because such discounts are inherently flawed.

Specifically, there are several design elements of the shopping cart program that Alibaba might have thought more about. First, the promotion suffered from relatively low exposure—one-third of users did not revisit their shopping carts before the coupon expired (coupons expired on the day they were given), so they never knew the discount was being offered. This low-key placement of the discount (users weren’t sent emails letting them know about it) meant that the discount program could have been easily overlooked. Alibaba could have varied exposure by doing more to raise awareness of the discounts, which might have dramatically changed users’ behavior.

Second, the current program allowed sellers to choose the size of the discount they offered. This prohibited the experimenters from understanding the causal effect of discount size on purchase behavior. In their summary of the results, the authors of the experiment note that the “shopping-cart promotion program did not often give out large discounts (the average and median discount rates were 17% and 13%, respectively).” The program might have been more effective if Alibaba had used some of its available data to help make better recommendations about how much of a discount to offer.

Third, the impact of the discount program on user satisfaction and long-term engagement with Alibaba remains unclear. The research team is transparent about this, noting that their “experiment was not designed to focus on detecting the long-term effects of the promotion program.” (The authors do find that users in the treatment condition search for more products in the following months, though this effect goes away within a year.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.