Research Outline

Ad Testing Pitfalls


Gain an understanding of potential pitfalls and issues surrounding creative testing such as is used for ads, in order to support a client presentation regarding how testing results and actual market impact are not always aligned.

Early Findings

Data Integrity

History & Selection Effects

  • Specific challenges to A/B creative testing include history effects and selection effects.
  • History effects occur when testing is run between two different times without accounting for the impacts of timing, such as 20 hours on a weekday one month and then 20 hours on a weekend another month, or during a holiday season peak vs. low season.
  • Selection effects involve testing only one element of ads, without accounting for the impact of other marketing strategies being run simultaneously.
  • For example, testing one ad, while simultaneously running a pay-per-click ad on another platform that is directing traffic to a website, and not differentiating the results could cause data issues that make it hard to determine the impact of a campaign.
  • Sampling distortion effects can also be an issue, which are primarily caused when the sample size is not large enough to overcome random chance.

Selecting Testing Approach

  • Because there are so many potential testing strategies, it's often challenging to determine which type of tests will yield the most accurate or useful results.
  • For example, Facebook ads offer a split-test feature, but for localized or custom ads, a more complex approach may be more effective.
  • Knowing what to test within an ad is also often a challenge, because involves marketers being able to determine which changes will both be easy to implement and create a measurable impact.
  • Ad testing methods need to be selected in a way that generates the most information on why the customer behaves the way they do that the marketer can apply the knowledge and have a foundational understanding of what motivated the behavior.