The Pitfalls of A/B Testing

Direct response advertising aims to prompt users to take immediate action online, such as making a purchase or submitting an application. This time, we spoke with Kosuke Takeshita from the Web Integration Department at Dentsu Digital Inc. and Yohei Harada from the Production Direction Group at Nextedge Dentsu about the ideal form of direct response advertising creative and the pitfalls of PDCA implementation.

*Nextedge Dentsu Inc. became "Dentsu Digital Inc." on July 1, 2016.

Direct response advertising and branding advertising are not mutually exclusive

──Please tell us about your work.

Takeshita: I handle web communication production, direction, and actual production. I often collaborate with Nextedge Dentsu Inc. on projects. We work together, with Nextedge Dentsu Inc. handling planning for external customer acquisition strategies and the subsequent website strategies as well as overall creative direction.

Harada: I handle the planning, composition, and production management for banners, landing pages, and other performance-based ads run by Nextedge Dentsu Inc. I propose creative strategies utilizing data from market research, ad reports, and access analytics to meet our clients' expectations.

──Please tell us about direct response advertising.

Harada: Direct response advertising refers to ads designed to directly prompt users who see them to take actions like making a purchase, registering for a membership, or submitting an application. In contrast, ads primarily aimed at enhancing product, brand, or company awareness and image are called brand advertising.

Digital marketing creative encompasses banners, landing pages, and the text used in search-linked ads. Nextedge Dentsu Inc. specializes in creating direct response advertising creative within the digital domain.

──What distinguishes direct response advertising from brand advertising?

Takeshita: Beyond the role differences mentioned earlier, another distinction lies in whether the focus is on achieving short-term marketing results or on expecting effects that include mid-to-long-term outcomes. However, while direct response advertising and brand advertising are often discussed as "opposites," I don't see them as mutually exclusive. Both approaches are important as "places where user-brand contact and brand experiences are created."

Driving actions like purchases is just one of the goals of direct response advertising. Similarly, brand advertising isn't just about appealing to easily grasped images like "cool" or "stylish." Good brand advertising defines its purpose within the broader marketing context and achieves that purpose.

──So you're saying direct response advertising and branding advertising shouldn't be separated?

Takeshita: Ideally, yes. However, in the initial communication planning phase, separating them makes the process easier to organize and clarifies the purpose and role of each communication strategy. The problem is leaving them separate. Skilled marketing companies effectively use both, linking the image built through branding to the results of direct response advertising.

For example, a cosmetics company used celebrities in terrestrial TV commercials to convey the product's worldview. Meanwhile, their 120-second commercials on BS and CS channels featured relatable users discussing the product's appeal, successfully making viewers think, "I want to pick this up myself" or "I want to try it." It was precisely because both types of communication existed that they achieved results.

However, building a brand image primarily through advertising requires significant time and money. Therefore, many companies choose to concentrate their management resources on direct response strategies during the initial phase.

Button colors, fonts... Could widely circulated winning formulas actually damage the brand?

──Are there any points to note regarding the creative for direct response advertising?

Harada: We must ensure the creative doesn't damage the brand image.

This is because direct response ad creatives often follow specific techniques and trends, like "using XX increases click-through rates" or "XX boosts conversion rates." For example, suggestions like using green or orange for buttons, or sans-serif fonts. Blindly applying these trends to creatives can sometimes undermine your brand's unique atmosphere and image.

Of course, I understand the desire to boost numbers even slightly. However, rather than chasing metrics alone, it's crucial to calmly assess why a particular expression is being used, how it aligns with the brand, and whether it should truly be adopted.

Takeshita: It's true that such techniques tend to take priority. The "why" behind "why this expression gets clicks" gets left behind, and only the technique is adopted. This can result in creative work that doesn't align with the brand. By the way, in this context, "brand" doesn't refer to an image of being cool or stylish; it's better understood as meaning "the essence of that product."

If you misunderstand this and proceed with the preconception that "branding = stylish, cool," placing visually appealing photos on small-sized banners won't effectively communicate to users.

To communicate more instantly, you need to combine highly visible copy with images. On top of that, just being slightly mindful of things like basing the color scheme on colors the brand values, choosing a color that stands out against that scheme for the parts you want to highlight, or using the same font if one is typically used, can dramatically change the creative. Even for direct response ads, instead of a branding approach that feels careless and doesn't clearly identify the company, you can create an expression that makes people think, "Ah, that feels like that company." However, since various techniques exist depending on the objective, there's no single definitive answer.

Numbers matter. But PDCA driven solely by numbers is meaningless.

──How do you measure the effectiveness of direct response ads and conduct PDCA?

Harada: Similar to the expression discussion earlier, I feel the "why" is often neglected in effectiveness measurement too. For creative PDCA, you need to clearly define "what the objective is," "what you're appealing to," "over what timeframe," and "how you'll evaluate it," then create a meticulous plan. But in reality, many companies can get as far as planning but struggle to maintain continuous execution.

When actually implementing creative work, executing plans with "priority" and "flexibility" is crucial. Priority means considering the potential impact on the business. Without this perspective, you can get fixated on immediate, granular metrics. Even if those numbers improve, the cost-effectiveness might not align. Such initiatives lack meaning, so prioritization must be based on business impact.

Furthermore, while some get bound by the notion that "once a plan is made, it must be executed exactly as written," it's necessary to flexibly adjust plans each time, asking "Why are we implementing this initiative? Does it truly make sense?" This is because environmental changes or unexpected events often prevent plans from proceeding as intended. Initiative plans require constant review.

──Do the key metrics for measuring effectiveness vary by project?

Harada: The metrics to evaluate change based on the project's objective. For example, for banner campaigns aimed at driving purchases or sign-ups, we use "click-through rate × conversion rate" as the metric. For banner campaigns focused on expanding awareness, "new unique users" might be the metric. Often, if the project's objective isn't clearly defined, the campaign won't succeed.

Takeshita: Some people think the PDCA cycle for direct response advertising means repeatedly testing visual changes like button color or size, while others believe there's a "correct" color.

While testing various patterns isn't inherently bad, it's crucial to establish hypotheses earlier on—such as how to communicate this product or service to gain market acceptance, or which target audience will find it appealing—and then evaluate based on those hypotheses.

Harada: As the term "high-speed PDCA" suggests, if you only focus on seeing results quickly and reflecting them back, you might lose sight of what worked well and what you're aiming for.

You need a perspective that shows progress across fine-grained spans, mid-term spans, and even larger spans like one or three years. In digital marketing, we've reached a point where we can run PDCA cycles beyond human limits. Sometimes, our minds can't keep up, and it just becomes "just keep spinning the wheel."

The Pitfalls of A/B Testing

──I've heard there are cases where A/B testing itself becomes the goal.

Harada: Yes, A/B testing is a method that compares multiple creatives to pursue high-conversion appeals and designs. It reduces acquisition costs and streamlines operations, so we use it frequently. However, when solely focused on maximizing conversion rates, you might suddenly notice a decline in the number of acquisitions. This is the pitfall of A/B testing.

Behind the efficiency gains from AB testing, less efficient prospect segments are being discarded. For example, if you have "quality" and "price" as appeal axes, and you create ads only for the "price" appeal pattern because it has better acquisition efficiency, you end up discarding users who value quality. The advertiser themselves is selecting and discarding users, narrowing the market. This is the danger of AB testing.

Narrowing appeal axes limits responsive users and leads to a bottleneck. After a certain point, you need to step back and refocus planning—like re-examining the needs of the discarded segment.

Takeshita: Sometimes, in the pursuit of efficiency, we focus too much on immediate acquisition rates or purchase rates. It's like casting your line at just one school of fish you happened to spot in the vast ocean. Rather than focusing solely on that one spot, we must remember it's just a part of the wider ocean. We need the perspective to search for other schools of fish, and even to cultivate and expand them.

──I see. So with A/B testing, perspective is crucial. In the second part, we'll discuss success stories and the future of direct response advertising.