Observed Signal · May 6, 2026 · Best Practice · Source: https://martech.org/feed/ · Impact: 2/5 · Sentiment: Neutral
Email A/B Test 'Winners' Can Be Misleading
A MarTech contributor warns that declared winners from email A/B tests can be misleading because results are context-dependent. Short test windows, audience variability, shifting inbox context (timing, seasonality, competing messages) and reliance on a single primary metric (e.g., opens or clicks) can produce winners that hurt long-term commercial outcomes. The article gives an example where a higher click-through subject line drove less qualified traffic, lowering conversion rate, average order value and revenue per recipient. It recommends hypothesis-driven testing, multi-variable experiments, evaluating downstream metrics (revenue, AOV, repeat behavior), and building a body of learning rather than treating single-test winners as definitive truths. Published on MarTech (owned by Semrush) on 2026-05-06.
Practical guidance on interpreting email A/B tests affects campaign measurement and revenue optimization for marketers, but it is a tactical/best-practice piece rather than a platform change or major industry event.
Track SEMrush Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your free Explorer workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Article published on MarTech on 2026-05-06.
- A/B test results can be distorted by time, audience variability, context (timing, seasonality, inbox competition) and choice of metric.
- Example case: a variant with higher click-through rate produced lower conversion, lower average order value, and lower revenue per recipient.
- Author recommends hypothesis-driven testing, examining downstream metrics (revenue, AOV, repeat behavior), and using structured multi-element experiments instead of treating single-metric winners as definitive.
- The article notes rollout strategies matter (contrasting a 10/10/80 rollout with a 50/50 split used in the example).
Connected Companies & Entities
1 Entity mappedOntology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Opens and Clicks Are Poor Email KPIs
The article argues that common email engagement metrics—open rate and click-through rate (CTR)—are unreliable proxies for business outcomes. Based on the author’s multi-year A/B test data, open rate correctly predicted the highest-converting or highest-revenue-per-email (RPE) subject line only 20% of the time, was incorrect 10% of the time, and was inconclusive in 70% of tests; overall it misled or provided no useful insight in 80% of tests. CTR performed worse: the highest-CTR variant matched the highest conversion or RPE only 7% of the time, pointed to the wrong winner 36% of the time, and was statistically inconclusive in 57% of tests. The author recommends using outcome-focused metrics—conversion rate and revenue per email (RPE)—and provides formulas for each, stating these directly tie email performance to business goals.
One question to improve every email campaign
The article argues that every email campaign should start by identifying a single, measurable primary goal — the one metric that will determine success. The author illustrates how campaigns with multiple equally weighted objectives (multiple CTAs, competing messages) make design and evaluation harder, using an association's conference email as an example. The piece distinguishes outputs (emails sent, pages launched) from outcomes (registrations, renewals, qualified leads) and recommends completing the sentence: “We’ll know this email was successful if…” before building the campaign. The article was published on MarTech on 2026-07-23 and is authored by Jeanne Jennings, CEO of Email Optimization Shop.
Search Engines Outpace AI Chatbots by 20x
An analysis by data provider OneLittleWeb of traffic across 10,171 websites in June 2026 shows that search engines still dominate website visits, drawing 162.19 billion visits (57.2% of tracked traffic) compared to just 8.26 billion for AI chatbots (2.9%). This makes search engines nearly 20 times more popular than all AI chatbots combined. Google alone recorded 98.19 billion visits, while ChatGPT had 5.32 billion, holding 64% of the chatbot segment. Both categories saw declines from the previous month, with chatbots dropping faster. The study, based on browser visits estimated from Semrush and Ahrefs, excludes app usage, which may undercount AI assistant usage. As Google integrates AI features into search, the line between search and chat continues to blur, but search engines remain the primary gateway to the web.
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.