“Who will Optimize the Optimizers?”–Watchmen Paraphrase
There’s a finesse to all this Optimizing Stuff.
I’m frequently amazed at all the learning and effort that needs to go into it to bring in the data, analyze the data, and pump out creative content that sometimes sounds a siren song to our customers and other times falls short of our optimization goals.
I have heard from far more experienced Conversion Optmizers than I that optimizers only get better in CRO as they continue to test and refine. Your first hypothesis might be rough, you might misread the data, you might pursue ghosts of a client, when in reality you’re missing the actual truth about what your clients want from you.
But the truth is only revealed with testing and iterating, testing and iterating.
It’s the eternal peeling of an onion.
As Life Goes, So Goes CRO
CRO tries to model what is actually happening in life. So It’s fitting that A/B testing is the capstone of the whole CRO process, because in this, it models everything else we do in life as we strive to perfect our crafts and work our arts.
Life is optimization and CRO is brilliant, because it acts as a whole new breed of digital marketing that recognizes this fact. CRO is the constant driven effort to make website design and digital marketing more organic, more like life. And it only makes sense we are truer to those who are in our market the more we try to shift as they shift, like an ongoing dance of customer and digital marketer.
When we look at the usual process of website design, the flat, static unchanging site, that is designed once and forgotten about, we can understand that.
In the physical world, no tactician of marketing and no professional salesman, would ever just sit back and let the customer browse their shelves without learning anything about how to place their merchandise better to make it more attractive to their customers or even what sort of merchandise to retire and what type to sell more of.
That’s not the way to sell anything. Instead, we have to bob and weave with the client, we have to woo them, and in order to do that, we have to understand them over and over and over again.
We need to change our tactics to fit the ideals of these customers. We need to optimize.
But it’s not just our sites that need optimized. If we want to get better at what we’re doing in CRO, we also need to optimize our optimization programs.
We need to make sure we’re getting better as well, more refined, more hypothesis savvy, more on point. If the tests are going to lead us to truth, then we need to get better at testing and better at discovering what the truth actually is.
Optimizing the Optimizers
Like any optimization test we run, to optimize our optimization program, we need to measure our qualitative and quantitative data. We need to take note of what is working and what is not.
Evangelizing for Optimization in an Existing Organization
First let’s look at starting a CRO program in an established company.
One important thing when inititally starting a CRO program is to create an effective team. Merrit Aho, Director of Optimization for Search Discovery, suggests a good inhouse CRO Agency needs many players to staff a properly laid out CRO program. Consider hiring an CRO generalist first and then a developer, a designer, an analyst, and a project manager.
Another thing that Aho says needs to be addressed is the misconception that many in the company-at-large might have about conversion optimization. Testing is not simply about testing buttons or testing everything, as some might incorrectly think.
In order for an organization to wield its CRO program effectively, it’s important that there is an understanding that testing has a central position in modern-day business practices, and that the tests that are run are discerning, led by business goals, and are prioritized so that each test is measured for ease of application and the amount of impact that we think it will make to our company’s bottom-line(s).
Another thing we need to optimize for from the onset is growth. We need to start out winning. Spotting the low-hanging fruits with heuristic analysis and Google Analytics is important and optimizing for them first is important so that others in the company get excited about the CRO program and can see how impactful it can be.
Just like optimizing a site through clarity of value, Aho says that it is important that we are given clarity to our CRO program to others in our organization: clear goals, clear what CRO is and does, and also clarity presenting data and wins to those in our organization that are outside the CRO world so they can grasp its importance and see as a valuable part of the organization.
Optimizing our testing
Tests are important in CRO, but to truly measure how effective our conversion optimization efforts are we need to look at the program with a birds-eye view. It’s not enough to see how many wins we’ve had in the past month, we need to ask ourselves if we’re improving.
Clair Vo, VP of Product at Optimizely, suggests we set goals for our tests, measure our tests regularly, but also iterate improve and she gives some metrics that she uses to help her do that.
- Testing Velocity: This is a basic understanding of how many tests we’re running each month for low traffic sites or each week for high traffic ones. We want to have a goal set to do so many tests within this segmented time period and be able to track our averages.
- Testing Capacity: Test capacity measures what testing velocity should be in a given year, which is also important for setting and measuring our goals.
- Testing Coverage: Again in line with our goals, this determines the percentage of testable days we are running a test.
- Win Rates: This is the percentage of tests that win, lose, or are inconclusive. Vo says that ideally, once we hit a steady testing velocity, the win rate should also stay level or increase.
- Expected Value : The expected value takes into consideration the cost of testing, the win rate, and the average lift when a win happens. This is a very important metric as it allows us to prioritize our tests according to a cost/benefit analysis.
- Annual Expected Value: By understanding our expected value and multiplying this by our annual testing capacity, we can now set a goal of what we might expect for an annual total.
Optimizing our CRO programs by being clear in our goals and values, and backing that up with hard numbers that we’re tracking keeps us measuring refining and testing, and just like our A/B tests we need to expect to iterate so we can stay in line with our company’s goals and keep giving value in our optimization efforts.