Three questions for your next website test

Before that, our final strategic workshop of the year is in less than a month.

Deep Dive LDN: Annual Planning

12th November 2026 | London | Invite-Only

Our last Deep Dive of the year is all about annual planning. One hands-on day in a small room of hand-picked brands, and the space to work through your 2027 plan and roadmap.

Here is what you can expect on the day:

• Workshop your 2027 plan: targets and roadmap, built around the problems you bring.
• Roundtables: small-group sessions with brands and experts in the field.
• Breakfast and lunch: on us.

It's free for qualifying brands, and with a room this small, seats go quickly.

Apply for your free ticket.

Why your BFCM Meta budget might be optimising for the wrong customer

Every year, the same pattern plays out. Peak season arrives, budgets increase, yet acquisition costs keep climbing and ROAS drops. Not because brands are doing something wrong, but because everyone is doing the same thing at the same time.

But here's what Clustie found from working with 90+ Shopify brands: the stores holding their performance through peak season aren't necessarily outspending their competitors. They are simply giving the algorithm better signals.

Meta’s algorithm is only as good as the data you feed it.

When you scale into BFCM using broad interest audiences or generic lookalikes, you are asking Meta to find buyers without telling it what a high-value customer actually looks like. As a result, the platform optimises for whoever converts the cheapest, not whoever is the most valuable to your bottom line. Those are rarely the same person.

The fix is straightforward: use a better signal based on your own Shopify data.

Instead of asking Meta to guess, you can tell it exactly what your best customers look like from your Shopify purchase history, buying behaviour, and predicted lifetime value. When Meta learns from your actual best customers, it finds more of them.

Brands making this shift ahead of peak season are seeing an average of +40% ROAS improvement and a 30% lower in CPA versus comparable campaigns, using the same budget and creatives. Just a sharper signal.

The Takeaway: Black Friday and Cyber Monday make paid acquisition more expensive for everyone. The brands that protect their ROAS aren’t the ones spending the most. They’re the ones whose audiences are ready before the auction gets noisy.

Clustie is a Shopify app that connects your customer data to Meta in minutes, without needing a data science team or complex setup.

Start your 30-day free trial.

How to check if a big A/B test result is broken

Decades ago in media research, Tony Twyman coined a rule that holds true for eCommerce today: if a number looks surprisingly good, it's usually wrong.

Say you add a trust badge under Add to Cart. Four days later, revenue per visitor is way up, and the team wants it live by Friday. Before you do, ask three questions:

1. Did your visitors split evenly?

Check the raw visitor counts on each side before you look at revenue. In a 50/50 test they should be almost exactly even.

Even a small gap matters. With 100,000 visitors, 50,800 against 49,200 looks close enough. It isn't. A test that's working properly would almost never produce a gap that size, so something is adding or removing visitors on one side, and the result can't be trusted.

On Shopify, the usual causes are:

• Test pages that load slowly and lose visitors on one side
• Refunds or test orders removed from one side only
• Apps, caching or bot filters that treat the two versions differently
  

2. Did the lift last?

Returning customers react to anything new, and that reaction fades. Look at the result day by day, and separately for new and returning visitors. If the lift shrinks after a few days, or only shows up with returning customers, it probably isn't real.

3. Can you explain why it worked?

If nobody can say why the change made people buy more, a big lift is more likely a bug than a win.

The cause is often in the tracking. Common ones:

• Purchases counted twice when someone reloads the thank-you page
• Bots or preview tools being counted as visitors
• A cookie banner that blocks your analytics but not the test, so some visitors are in the test but never measured
  

A useful health check is to run a test where both sides see the same page. It should show no difference. If tests like that keep producing a winner, the problem is your testing setup, not your site.

The Takeaway

Before you put a test result live, check that:

• Visitors split evenly between the two versions
• The lift held up day by day
• It worked for new visitors, not just returning ones
• You can explain why it worked
  

Anything that fails a check goes back for investigation, whatever the headline number says. A result you can't explain is a result you can't ship.

This week's feature was brought to you by Dayo Samuels, Rainy City Agency.

If you have a result from last quarter that looked a bit too good, share it with Rainy City Agency, and they will tell you whether it holds.

In Other News…

Meta's new AI ad tools and what UK brands can use now: Two features announced on 6 October are open to all advertisers: Customer Lifecycle Strategy, to prioritise new customers in a campaign, and video made from your static images in Advantage+ creative. In-chat checkout is still in testing, with no UK date yet.

What this means for you: if you're short on video, this is a cheap way to test it, and Customer Lifecycle Strategy suits a Q4 focused on new customers.

Shopify starts phasing out script tags: Apps can no longer create or update script tags as of 1 October. Script tags are an older way many apps and tracking pixels load onto a store. Existing ones keep running for now, but Shopify will stop loading them on storefronts on 1 March 2027.

What this means for you: anything still loaded by a script tag, including tracking for your ad platforms and testing tools, will stop working on that date. Ask your developer or app providers which of yours still use them, and plan the switch well before March.