Measuring the real effect of your loyalty program
“Members spend more”: true almost everywhere, and it proves almost nothing. Your best customers sign up first. To know what the card really adds, you need a control group. Here is how it works, with a worked example.
By the HoorayPass teamPublished September 30, 20267 min readJump to sources
On this page
The trap: your best customers sign up on their own
Loyalty program vendors often publish flattering numbers. According to Toast, a point-of-sale vendor for restaurants, bringing a guest into a program can move their return rate from about 7% to nearly 30%. The number is real. But it mixes two things.
Who signs up for a loyalty card first? Your regulars. The people who already came often, who know your team, and who would have come back anyway. Comparing members with other customers means comparing your best customers with everyone else. The gap is big, but the card isn’t its only cause.
Researchers have measured this. In 2007, a Dutch team (Jorna Leenheer, Harald van Heerde, Tammo Bijmolt and Ale Smidts) studied supermarket loyalty programs with a sample of households. Once this self-selection effect was corrected, the real effect of membership on share of purchases was positive, but seven times smaller than a naive comparison between members and non-members suggested.
What really changes: occasional customers
The good news comes from another 2007 study, by Yuping Liu, published in the Journal of Marketing using data from a convenience store chain. The heaviest buyers at the start claimed their rewards without changing their habits. Occasional and moderate customers, on the other hand, gradually bought more often and became more loyal.
For a business, the lesson is twofold. The card doesn’t transform your regulars: it thanks them. Its real effect shows among customers who came now and then. That is exactly what a control group lets you see.
The control group, plainly
The idea comes from clinical trials. Before sending a promotion, you set aside, at random, a small share of the customers who would have received it. This control group gets nothing. Everyone else gets the promotion.
Because the draw is random, the two groups look alike: same regulars, same occasional customers, same weather, same season. After a few days, you compare the share of customers who came back in each group. The gap between the two is the effect of the promotion. The rest would have happened anyway.
A worked example
The numbers below are made up for the example. They show the method, not a result to expect.
- Members who receive the promotion
- 900
- Members set aside (control group, 10%)
- 100
- Came back within 14 days, group that received it
- 162 of 900, or 18%
- Came back within 14 days, control group
- 12 of 100, or 12%
- Visits due to the promotion (6% of 900)
- about 54
Without a control group, you would have concluded that the promotion brought back 162 customers. In reality, about 108 of them would have come back without it. The promotion added roughly 54: that is the number to weigh against the cost of the promotion.
The formula: visits due to the promotion ≈ (return rate of the group that received it − control group rate) × number of customers who received it
Reading results without fooling yourself
A small business doesn’t have thousands of customers, and that is a real limit. With 100 people in the control group, a few returns more or less change the result. A few tips to still draw the right lesson:
- Add campaigns up. A single promotion says little. Ten promotions of the same kind, each with its control group, draw a trend.
- Keep the same window. Always compare over the same period, for example 14 days after sending.
- Be wary of small gaps. One point of difference in a small group can be chance. A gap that shows up campaign after campaign, much less so.
- Count in money. Extra visits are only worth their margin, minus the cost of rewards and messages.
And without a control group?
Two other benchmarks help, even if they are less solid:
- Compare the same customer before and after signing up. If they came once a month and now come twice, that’s a clue, especially for an occasional customer.
- Compare the same period year over year. December compares with December, not with February.
Neither removes self-selection. They are better than nothing, but not as good as a control group.
To know how many extra visits you need to pay for a loyalty card, see our calculation by trade. Read the article on the visits that pay for the card
Before you measure, check
What HoorayPass does
Each campaign can keep a control group: it doesn’t get the promotion, and its visits are compared with everyone else’s. You see the real effect, not an impression. The dollar report then does the math: your members’ sales, minus the real cost of rewards, minus the subscription.
Sources
- Leenheer, van Heerde, Bijmolt and Smidts, 2007, “Do loyalty programs really enhance behavioral loyalty?”, International Journal of Research in Marketing (academic study)(opens in a new tab)
- Liu, 2007, “The Long-Term Impact of Loyalty Programs on Consumer Purchase Behavior and Loyalty”, Journal of Marketing (academic study)(opens in a new tab)
- Toast, Regulars Report 2026, reported by Fast Casual (vendor data, US restaurants)(opens in a new tab)
