Using science to evaluate marketing campaign effectiveness

Oct. 25, 2023

3 min read

This is part of a four-part series highlighting work selected for presentation at the 2023 Grace Hopper Conference. The following includes insights from Walmart technologists Q Zhang and Renfei Gao.

Experimenting in marketing is common, with strategies being evaluated based on using audience research and gut. At Walmart, we use advanced testing techniques to assess marketing efforts, like promotions and price changes, allowing us to make decisions based on data. This improves engagement, conversions and customer behavior, helping us find and optimize areas for better results.

Walmart’s scientific approach to evaluating marketing campaign effectiveness is centered on the ‘causal inference’ method, which determines if a change in one variable causes a change in another. It differentiates between correlation (two events occurring together) and causation (one event causing another). For instance, if a business has an ad campaign followed by increased sales, causal inference methods help establish if the ad caused the sales increase.

The most effective method in marketing is Randomized Controlled Trials (RCTs), where two random groups are assigned with one receiving a promotion and the other not. This ensures unbiased marketing impact measurements. If not feasible, we use statistical techniques like Matching, Difference-in-Differences and Causal Impact to minimize biases.

The infographic depicts how RCT works. On the left, we have the test group who were shown paid ads, and on the right we have the control group who weren't shown paid ads. If the conversion rate of paid ads is Y and regular conversion rate is X, then the effectiveness of paid ads is calculated as: (Y-X)/X

An illustration of how RCTs help drive informed decision making in retail

Where full randomization is unattainable, using a quasi-experimental method called Propensity Score Matching can be used to evaluate the marketing effort's impact. Propensity Score Matching is a way to compare groups in a study by matching people with similar backgrounds and features. This helps create a fair comparison, as if they were randomly chosen, and allows researchers to find the real effect of a treatment or change.

Causal inference methods show the true effects of marketing actions. This helps marketers make strategies based on real cause and effect, not just observations. By testing these factors, marketing becomes smarter and data-driven.

Intuition still matters, but knowing beats guessing. Quasi-experimental designs and RCTs offer data-backed insights for better marketing decisions.