Experimentation has been a key tool for marketers to test effectiveness of advertising for many, many years – most often used in the form of A-B testing and matched-market geo-testing. The basic form of experimentation involves defining a test group and a control group, and measuring the impact of specific advertising by calculating the difference in conversion between the group exposed to the advertising and the group that is not exposed. In today’s world, with the challenges of walled gardens, cookieless execution, and incomplete consumer path understanding, experimentation has taken on a renewed importance as a key method to isolate true incrementality.
The need to move past last-click attribution has led to the growth of experimentation as a critical tool in the measurement toolbox, and several companies have succeeded in building platforms that allow for experimentation at scale to provide consistent insights for media optimization. If you are a business with easily trackable conversions, experimentation is an excellent path to understanding as it provides some significant benefits including:
- True incrementality for some key channels that are testing
- Potential for deep granularity understanding within each tested channel (e.g. placement, campaign, etc.)
- Potential for creative insights (based on test design)
- Privacy compliance
- Easy to understand results – which drives adoption and more informed optimization
However, experimentation does have its limitations. It requires an intrusive set-up that requires changes to media plans and can limit the ability to make the most of every paid media dollar as it requires holding back some potential impressions. It also works best with online conversions and specific media channels that are easy to turn on-and-off. As an example, while a TV match-market geo test can provide some very important insights, it can take months to set-up and run, and is difficult to scale. If brands are trying to scale, it is important to ensure that only limited budgets are used to test new media tactics.
Good marketers will often implement a 80-15-5 rule – meaning 80% of spend is devoted to driving the business with established channels, 15% is spent on emerging channels, and 5% is spent on testing the effectiveness of new channels. Ultimately, while experimentation is helpful for any brand, using it as a stand-alone tool is only suitable for brands that spend on limited channels.
The good news is that brands can get a complete understanding of both strategic and granular incrementality by combining experimentation with Marketing Mix Modeling. The two approaches are extremely complimentary and can be used to calibrate and validate each other – providing marketers with critical insights across all media tactics and sales channels. At Leavened, we encourage this type of measurement integration, and through a partnership Measured, the market leading incrementality testing company, we provide marketers with a true 360-degree view of performance and opportunity for optimization.
