How to work with a control groups?

A control group is a powerful tool for evaluating your communication performance. It helps you measure the monetary business benefits of email or sms campaigns. In this article, you will learn what control groups are, how to work with them and how to respond to all your colleagues who’ll decide to critique you and steer away from the idea of establishing them.

Control groups: what is it?

Simply put: a control group in communication is a group of subscribers excluded from all direct communication for a given period of time. These are people who meet the campaign criteria (under regular circumstances your communication would reach them) but were deliberately excluded from it. As a result, we have two types of subscribers: ‘reached’ and ‘unreached’. By comparing behaviours between these two groups (e.g. number of transactions and basket value), we can identify the effect of direct communication.

Control group

The control group should be formed of a representative selection of subscribers. It is important to exclude the control group from any direct communication that takes place during this time, i.e.:

  • regular newsletter campaigns;
  • communication based on consumer behaviour (triggered communication), e.g. email sent after purchase, response to an abandoned shopping cart, etc.

This applies to all direct channels: SMS, emails, postal services and telephone calls.

Size of the control group

To be statistically significant, a control group must be large enough (statistically significant), meaning large enough for the results to be representative and extrapolated to your entire database. This is assumed to be a minimum of 10 000 subscribers, regardless of how large your country’s database is. If you are just starting your direct marketing journey and your subscriber base is not very large, you can determine the size of the control group in percentages. A minimum value for a control group is 1–3% of all subscribers you plan to send your communication.

Key principles for setting up a control group

  • To compare your ‘reached’ subscribers with a control group, both of them must be representative, so the selection of participants needs to be randomized.
  • Subscribers who have opted out of communication should not be included in the control group, as they are not representative of the subscribers with whom we communicate.
  • The control group should be created at one point and maintained for the entire period selected (e.g. one quarter). After this period, the group will be replaced by a new control group for an upcoming period.

One or many control groups?

It might be a good choice to create a single quarterly control group to measure the results of direct communication, especially if you are just starting out. In that scheme, once a quarter, you draw a representative group of subscribers to whom you do not send communication and then compare how the excluded group and the communication group behaved. This will provide you with indispensable insight – which communication triggers are working well and which can be improved. Working with a single, control group on a quarterly basis should be sufficient, especially if your business is limited to cyclical, mass newsletter campaigns or if your client database is not very large. The more often you communicate with your customers, the shorter the period for which you establish a control group should be.

Along with the growing number of subscribers and adding new push&pull campaigns, it’s worth considering introducing new rules, i.e.

  • Create one quarterly overall control group to measure the overall impact of your communication on the business. You draw one general control group from the entire subscriber base and exclude them from communication activities for a certain period (quarter).
  • Create separate control groups, one for each specific type of communication. The purpose of the additional control group is to test the effectiveness of a specific type of communication, e.g. a discount coupon campaign or cross-sell / up-sell communication after a purchase.

Critique of control groups

Control groups are a very powerful tool, and it is difficult to undermine it because they clearly show the effectiveness of direct communication activities. Sometimes, unfortunately, no impact on sales can be noticed at all (or in extreme cases, a negative impact). But even in such circumstances, it is always valuable feedback and it is very difficult for a company to develop and improve without it.

And, while it is difficult to argue with the facts, there are several arguments to the detriment of the control groups:

Control groups reduce the campaign reach
My personal experience is that this is the most common argument, mostly put forward by the sales manager, to show that control groups are a waste of time. After all, it is obvious that if you exclude some subscribers from a campaign, the number of messages will be lower – in other words (or in the words of the sales manager), you will earn less. It is therefore important to keep the sizes of the control groups as low as possible (and never more than is statistically necessary). In mature communication processes, my experience also shows that people who have been excluded from communication for a short period of time, often are more engaged after they are back in the game. This helps to compensate for this temporary loss. And to reinforce this effect, you can prepare a special action (promotion, interesting content) for those you are bringing back after the ‘quarantine period’.

We do not always want – or it is not in our interest – to exclude anyone from communication
An example? A welcome email for people who have just joined your newsletter or signed up for your loyalty programme. If someone trusted you to share their data and consented to the communication, they should not go unnoticed. You should at least welcome them and offer a series of engaging messages.
You can find more examples of this within communication strategies. Common sense should prevail. It starts with the fact that, within the control group draw, the number of new club members will be marginal (the larger the base, the greater the proportion of long-term subscribers). If the number of new subscribers outweighs those with longer tenure, the solution may be to offer a separate communication process for 'newbies’ (e.g. a separate welcome communication sequence independent of ongoing campaigns). This is how to welcome everyone the way they deserve it, without compromising on data quality.

It is not always possible to exclude someone from communication for legal reasons
It is no longer a question of what we want or what we don’t. Consumer law sets out rules related to the different treatment of customers, profiling possibilities, etc. Sometimes there is a need to reach the entire base, regardless of the communication plan you have prepared (e.g. change of regulations). In such cases, it is always advisable to consult the legal department and check whether the exclusion of a certain group from ongoing communication is justified for that particular communication.

Those receiving offers are more likely to show their card at the cash
Sometimes you can see this with the customers who received an offer, especially if it includes a discount voucher for a wide range of products (e.g. x% discount on the entire range). These customers may have a higher propensity to show their cards in-store more often or log in when shopping online. They have a vested interest in doing so. They have to do this to get a discount, while those excluded from communication do not have such a strong trigger to show the card. As a result, you will notice an ‘artificially’ lowered number of transactions done by the customers from the control group as compared to those who received the communication. It is worth bearing this in mind and checking periodically if it is the case. This can be achieved through exit polls (similar to the ones taking place during elections – you ask those leaving the shop if they have taken advantage of the offer and whether they belong to a club / have shown their card today). The most important thing, however, is to work systematically on appropriate benefits, communication, and staff motivation to alleviate this negative effect.

You exclude someone from communication for too short or too long
None of these options is actually a good one. What if you set up monthly control groups, but the purchasing decision in your industry takes several weeks on average? People excluded from communication in one month can shop the next month and it will change your score. On the other hand, if you exclude some of your customers for too long, they may forget about you and move away to your competitors. What is the ideal period? It largely depends on your industry, your communication strategy, etc. You should look into that with an analyst, observe typical customer behaviour and adjust the timing to suit your situation.

If we change the rules for the control groups too often, we get strange results
Sometimes it is better to have non-ideal assumptions than to constantly change them in search of the perfect ones. You should be guided by the motto 'better done than perfect’. The analysis of the control groups is based on certain assumptions. It is an estimate, a mathematical prediction, and not an irrefutable theory. Therefore, it is better to maintain fixed rules, bearing in mind that they are not flawless. This is because every change makes it hard to compare a period to period and all the knowledge you have gained so far needs to be revised. And this usually takes more time than quarterly reporting in our corporations…

If you enjoy my blog post

Please consider buying me a coffee

Get inspired