September 2026 update: make the decision process trustworthy before interpreting a lift
Conversion optimisation is an evidence process, not a collection of proven percentages or benchmark claims. The archive record below reflects an earlier stage of the series. A current programme should begin with a real decision: what change is under consideration, what action would count as success, and what harm would make the change unacceptable? If those are not set before exposure, the final report can be shaped around whatever number moved.
Microsoft’s work on trustworthy online experiments recommends practical checks around experiment design, data quality and interpretation. One essential check is sample ratio mismatch. If the observed assignment differs unexpectedly from the planned split, the experiment may have a delivery, logging or targeting problem that makes the result unsafe to interpret. Investigate the mismatch before reading a conversion difference as an effect of the page.
Define the primary outcome before running the test, then keep a small set of guardrail metrics that can reveal an unacceptable side effect. A page may increase one recorded action while creating a problem elsewhere in the journey. Guardrails are not a way to declare every change successful; they are a reason to pause, investigate or reject a change when the broader evidence says it should not ship. Record the eligibility rules, assignment, implementation date and any exclusions so another reviewer can understand what the comparison actually measured.
A test often needs at least 7 days to cover a complete weekly cycle, but 7 days is only a calendar minimum. Low traffic, a small effect or continuing uncertainty can require a longer run, while a data-quality problem can invalidate a longer test as well. Do not stop because a calendar threshold passed. Statistical significance is not a permission slip. The American Statistical Association cautions that a p-value does not measure the size or importance of an effect and does not by itself support a scientific or business conclusion. Consider the size of the observed difference, uncertainty, data quality and whether the metric represents the outcome that matters. Do not revive unproven ROI or cost-per-acquisition benchmarks from the archive as though they were measured facts.
External validity matters as well. An experiment can be accurately measured for the visitors, timing and implementation that were tested while still being a poor guide to another audience, season or site state. Microsoft’s research on external validity and metric pitfalls is useful here: treat the result as evidence with scope, not a universal law. A sound conversion process preserves the hypothesis, checks sample integrity, reviews guardrails and explains the decision in plain language.
Sources: Microsoft Research on trustworthy experiment analysis; Microsoft Research on sample ratio mismatch; Microsoft Research on experimentation guardrails; Microsoft Research on external validity; Microsoft Research on metric interpretation pitfalls; American Statistical Association statement on p-values.
Archive record
Part II in this series covered the importance of the roles of testing and modification in the development of successful conversion optimization. Now we will attempt to make sense of what we convert.
Step 7: Measure
I remember a very energetic web guy back in the dot.com glory days who pitched us on his money-making system with the declaration, “He who has the most data wins!”
I must admit, I thought that was pretty cool and used it myself a time or two to impress people. But experience has shown me that quality data is really the key, because today we can measure things to the point of becoming overwhelmed.
Metrics are everywhere. Just watch a sports show on TV and you’ll see how every aspect of player performance has some sort of statistic. And if you have ever witnessed a presentation of data in a meeting you have probably also witnessed eyes glazing over and slow little pools of dribble accumulating out of the mouths of those present. Try and avoid that.
Quality data starts with the basics first:
- Who – as much as we can easily collect by name and contact info
- What – was it a red squeegee or an SEO scorecard request?
- When – because at some point, we’ll need to know if the weekends are an active time period for us or not.
- Where – Why in the heck are we getting so many sales out of Kentucky?
- How Much – our average sales are below $2.00 – the merchant fees alone are killing us!
- How Many – we took in 10 case forms yesterday, more than we did during all of last week. Why?
Once you have these core metrics you can compare them based on historical data over time and then you’ve got half the battle won already. But there is more to know. And the more you immerse yourself in simple data, the easier it becomes to start to peel out more granular information that turns into actionable data, such as:
- The day of the week to send emails based upon highest open rates
- Testing a new form with modified fields, resulting in a 40% increase in forms received, when previously two out of every three forms were abandoned
- Determining sticky (or confusing) landing pages to increase average minutes per visit
And there are many, many tools of the trade to help you measure such data. Google Analytics is the free giant, but Webtrends, Unbounce and Hubspot also offer tracking metrics and lead measurement within their own subscription platforms.
Step 8:Wash, Rinse, Repeat.
Sorry. Go back to Step 5, modify, and try it again.
Step 9: Understanding Results, or What the Heck is my ROI?
The ongoing name of the game is to increase that conversion rate, whatever you, your business, your agency or your internet marketer have agreed upon is an actual conversion is (see Step 1).
But the holy grail of conversion rate optimization is the Return On Investment.
Because the more you improve that conversion rate, the more you improve your chances to make more money. And the best part about the internet is that it’s so highly measurable. Of course, the worst part about the internet is it’s so highly measurable. So let’s turn this into an SAT-worthy word problem with a common illustration of the ROI formula:
- It takes 50,000 impressions a month to get us 500 clicks which get us 25 appointment requests of which we close 5 deals. The average value of those deals is $1,500. Thus we take in $7,500 (5 x $1,500 = $7,500).
- Not so fast. We spent $5,000 a month to buy those impressions (or those “clicks” if you prefer, or those email addresses, etc.).
- Thus you get 25 leads for $5,000. Using some applied physics with calculus we come up with $5,000 divided by 25 = $200. That is your Cost Per Lead.
- You closed five deals. So your Closing Ratio of 5 out of 25 is 20%
So what we are left with is $7,500 – $5,000 = $2,500 And that’s your ROI.
Right? Yes. In a perfect world. Except that $5K you spent isn’t the only cost you’ll have in it. Please also consider manpower, resources, time and effort, and expertise.
How do these things, often intangible, get calculated in? And remember, not every calculation on cost is as clean. To make it even more challenging, conversion rates get muddled and even outright miscalculated if your funnel is set up incorrectly or your goals aren’t measured properly.
Worst of all, it’s impossible to do a break-even analysis beforehand because you may not know what your actual leads or clicks or impressions are going to cost before you start running.
What we’re talking about here is the total Cost Per Acquisition which is always, ALWAYS greater than your Cost Per Lead.
Still you have to start somewhere. My recommendation is to begin with three manageable and reasonable questions:
- What is a lead worth to you?
This can be the most difficult question of the whole process since the natural inclination is to say, “Well it depends upon what they buy!” But you must make a guess as to what an average sale is for you, or set a realistic target sale price. - What are you willing to spend for a lead?
And we get back to the reflexive answer, “Well, how much will it cost?” But you have to start with what your comfort level is and some kind of framework for reality. If you sell a .99 item, you should not spend $3 for a lead. Even if you believe that over the lifetime of that customer he or she could potentially spend $10. It’s never a sure thing. - What do you need to add onto the lead cost to get a sense of what your Cost Per Acquisition is?
Again, try to be realistic. There are always outside costs. And even if you are doing the work in-house, someone still has to spend man hours on it.
Okay, so putting it all together, what does it all mean?
Here’s a guideline, but keep in mind, it’s just a general rule (because as we learned from the wise and great Obi Wan Kenobi, only a Sith speaks in absolutes):
Once you have your Cost Per Acquisition down, it should not be more than 20% of your final sale. Why? Because you could eat up your profit margins quickly and you will always have some sort level of waste in the constant battle for conversion optimization.
