How We Squashed a Browser Bug That Cost an E-Commerce Client 6-figures in Revenue

In today’s fast-paced e-commerce environment, the tiniest details can significantly impact your bottom line. 

For e-com retailers, organic search traffic from Google is a primary driver of revenue.

But even though search engine optimization (SEO) data is easily accessible (via Google Analytics (GA), Google Search Console (GSC), or third-party tracking tools), the numbers can easily be misinterpreted.

Even savvy, well-established e-commerce companies struggle to understand unexpected dips in traffic and revenue.  

Several months ago, one of our clients faced an existential threat when sales unexpectedly plummeted. 

As a dedicated digital marketing agency that always goes above and beyond, we took it upon ourselves to uncover what caused the loss in revenue and help recover lost earnings.

It took all our experience and technical knowledge to reveal the culprit — a bug occurring during checkout after a browser update was released. Welp!

What Caused the Plunge in E-commerce Revenue?

We take client confidentiality seriously. 

For the sake of anonymity, let’s call this client ACE. 

ACE is a successful e-commerce company competing in a highly saturated DTC sector (not a YMYL niche).

SwishDM had worked with ACE for six months before the revenue drop occurred.

In addition to steady organic traffic growth, ACE was crushing its historical revenue numbers after we conducted multiple A/B tests to improve conversion rates —  (skyrocketing from $140k to $200k monthly organic revenue).

In May 2023, they were on pace to break their historical sales record one more time (on pace to exceed the $230k mark by the end of the month), which thrilled both the client and our team.

The end result?

A disappointing $165k.

This significant shortfall immediately rang alarm bells.

As a standard operating procedure, we immediately analyzed GSC and GA to see if there were any differences in the number of clicks/sessions between April and May. 

But we found nothing.

May clicks dropped 1.97% compared to April, but that slight difference couldn’t be responsible for such a precipitous drop in revenue.

What about conversion rates?

The organic search channel conversion rate dropped from 0.78% (in April) to 0.75% (in May), a 4.31% drop in percentage, and a 0.03% drop in raw numbers. 

We would expect this to lead to a slight decrease in revenue (but not to this level).

But here’s the kicker…

Conversion rates for ALL major channels dropped — not just the organic search channel.

This channel-wide drop in conversion led us to believe there must be an underlying issue with the entire website.

When the user experience (UX) hasn’t been intentionally changed, a cohort analysis is an invaluable tool for uncovering unexpected dips in performance.

What Is a Cohort Analysis?

According to Wikipedia, a “cohort analysis is a kind of behavioral analytics that breaks the data in a data set into related groups before analysis. These groups, or cohorts, usually share common characteristics or experiences within a defined time-span.”

Instead of examining the month-over-month data with no direction, a cohort analysis enables us to see how each group of visitors with the same characteristics interact with the site and (hopefully) develop theories on what led to the drop in revenue site-wide.

So, what did we find in the Cohort Analysis?

Yes, we know that’s the not highest-definition image you’ve ever seen.

Unfortunately, Google Analytics doesn’t allow you to change the date range in a Cohort Analysis, so we had to take a screenshot from the explanation video we sent to our client.

via GIPHY

Anyways, back to the data.

Firstly, why did we only look at returning users?

Actually, we looked at both returning users and new users. 

But, because most of our client’s revenue is from returning users (around 74%) — and the new user data was inconclusive — we focused on the segment of returning users.

The numbers you’re looking at above represent the average monthly revenue per returning user. As you can see (hopefully), prior to May, the revenue per user numbers were robust (averaging $12.87 in the five weeks leading up to May in a low-price DTC niche).

But starting May 7th, we saw a steep decline in Revenue per User numbers. 

The precise figures in the following three weeks were $9.21, $7.00 & $8.94, respectively, averaging $8.38. A $4.49 (35%) drop in Revenue per User.

Then, starting the week between May 28th – Jun 3rd, the numbers increased slightly to average around $10.88. Still a $1.99 drop from that $12.87 mark we had before May.

That’s almost inexplicable. 

What could have happened in May?

We broke the stats down further by device type to see if there was any correlation between the browser people were using and the decline in sales.

Mobile users noticeably followed the dropoff in conversions around the middle of May. Even though the numbers weren’t decisive, we still saw a decline in Revenue per User in that period (the lowest point was $1.17 in the week between May 21st – May 27th)

For Desktop users, the trend was much more apparent. 

In the week between April 30th and May 6th, Revenue per User dropped from $12.76 (in the week before that) to $6.53 — nearly a 50% drop — and stayed under the $8 mark most of the time from that point onward.

Tablet users totaled around 2,300 in those 12 weeks, so let’s disregard them for now.

After the cohort analysis, we knew definitively that both mobile and desktop users were experiencing problems with the site. 

But what was the issue? 

What About The Shopping Behavior Report?

Finally, we’re back to high-res screenshots 😉

Armed with the cohort analysis, we examined the Shopping Behavior report. We broke it down by Device Category to see how visitors interacted with the checkout process.

Comparing the data between June (at the time of the audit) and May (the start of the drop in Revenue per User), immediately, we noticed two standouts:

1. Higher “No Basket Addition” percentage in Mobile devices (83.04% vs 76.65%)
2. Higher “Basket Abandonment” percentage in both Desktop and Mobile devices (4.63% vs. 2.1% and 9.2% vs 3.38%, respectively)

Our gut feeling was that some users were encountering issues with the checkout process.

After looking at the Shopping Behavior by Device Category, looking at the different browsers to identify potential browser-specific problems made sense.

That’s when we had our first “breakthrough.” 

The three biggest takeaways from this report were:

1. Basket Abandonment percentages were higher on all major browsers, especially Safari (19.13% vs 3.91%). Raw number-wise, it was a difference of 50 basket abandonments in Safari alone (more than all other major browsers combined)
2. No Basket Addition percentage was higher for Safari users (84.09% vs 73.58%)
3. Check-Out Abandonment percentages are significantly higher for Safari and Firefox users (93.23% vs. 85.7% and 75.56% vs. 68.75% respectively)

At this point, our theories were: that one (or more) of the following user actions had issues:

1. Add To Cart
2. Check-Out on Firefox
3. Safari issues as a whole

Out of the three, the 3rd theory seemed the most likely. 

The dropoff percentages at every step of the checkout process in Safari screamed at us to investigate further.

And that’s exactly what we did.

Let’s Come Back To Where It All Began – The Cohort Analysis

This time, instead of filtering by device types, we created two new segments: ‘Returning Users Using Safari’ and ‘Returning Users Not Using Safari”.

The low-res JPEG returns, but please bear with us.

First, let’s talk about returning users not using Safari.

The trend here aligned with what we knew about the site so far — a drop-off starting from the week of April 30th – May 6th. 

But revenue stabilized in the week of May 28th – Jun 3rd.

What about the Returning Users Using Safari segment?

Aside from the week of April 9th to April 15th, Revenue per User numbers had been relatively consistent  ($7.15, $9.22, $8.05 & $9.68).

• From April 30th and May 6th onward, sales declined sharply, with Revenue per User numbers dropping weekly to a low of $4.23 between Jun 4th and Jun 10th.

With all the data we had accrued, could we say with certainty that Safari was the smoking gun for this whole ordeal?

Not really.

But, it was worth sharing the evidence with our clients at ACE for their development team to investigate further. 

Thanks to our analysis, the developers knew where to focus their efforts instead of diving blindly into the conversion rate optimization rabbit hole blindly.  

We urgently informed ACE about the potential Safari issue and asked their team to investigate the checkout process immediately.

The Aftermath

Here are the key takeaways:

• ACE thanked us profusely for our timely and incisive analysis of the precipitous drop in revenue
• Thanks to our efforts, ACE’s engineering team uncovered issues with the Safari 16.5 update, officially released on May 18th, 2023
• Safari update issues included:
  • Users experienced unusually long load times during the checkout process, leading to higher abandonment rates
  • A third-party tool had a loading error on the new Safari
  • One aged-out core library was incompatible with the Safari update

Since being alerted to the problem,  ACE has conducted many new Safari tests, hotfixes, and optimizations. 

How does the site perform on Safari now?

As you can see, for Returning Users Using Safari and Revenue per Users numbers only started improving from the week between July 16th and Jul 22nd. 

Yes.

Despite the devastating impact on revenue, it took that long to improve the outstanding issues.

Why do we say improve instead of ‘fix’? 

Because not everything is 100% fixed yet. 

ACE’s engineering team is still working on the issues (and squashing newly identified bugs) at the time of this article.

Once a bug takes residence in your website, squashing it isn’t easy

The Takeaways

After all that, what are the crucial takeaways for business owners and SEOs?

1. Don’t just look at rankings and clicks when reviewing a loss in revenue. Looking at available data from a new angle gives you a perspective you would have never otherwise considered.
2. Don’t get overly hung up on pinpointing the issue and when it happened. Look for trends that stick out like a sore thumb and justify further investigation. We’re not doing rocket science here — the goal is to quickly improve a bad situation and get revenue back to normal.
3. User experience can make or break a campaign (or an online business altogether).
4. Maintaining a custom platform takes time and money. BUT, not maintaining a  custom platform can cost a successful business even more time and money. It can easily result in a six or seven figure loss in earnings (or more). Be proactive!

Is this kind of in-depth conversion rate analysis a standard service most digital marketing agencies offer?

No, it’s not. But SwishDM’s team aren’t your standard SEOs.

Our decades of combined technical and content creation experience make us different…

Just check out our testimonials.

Want to take your digital marketing and SEO to the next level?

Set up a call with SwishDM today.