Conversion optimization, much like most SEM activities, touches all verticals and various channels. Each company has its own unique experience, and we always emphasize, no two projects are ever the same. It’s that uniqueness which ensures CRO will succeed for that organization. Nonetheless, projects are brought together with a common thread: the documentation and deliverables that are produced as a result of the research and effort are pretty standard.
Much to my own chagrin, most of our potential clients say that what most CRO agencies say they offer sounds pretty much the same. Obviously, as an agency co-owner, I believe what we deliver to our clients to be superior compared to other companies: our consultants are unique, our process is robust, our attention to client needs and details is unbeatable, and our results speak for themselves.
But from a potential client’s perspective: process and deliverables wise, it all sounds just about the same. That makes distinguishing our UVP to potential clients challenging. But that’s for another blog.
Realistically, there are certain common deliverables that every conversion optimization service provider must strive to provide. If they don’t, well stay away from that company.
What are the deliverables of Conversion Rate Optimization Project?
According to Wikipedia:
“A deliverable is a tangible or intangible good or service produced as a result of a project that is intended to be delivered to a customer (either internal or external). A deliverable could be a report, a document, a software product, a server upgrade or any other building block of an overall project.”
In terms of CRO projects, deliverables are generally offered as a sort of tangible documentation that is a record of work, results, and analysis. From company to company, how some of these are presented, and in what format, varies. But there are some classic pieces that everyone provides:
Qualitative and quantitative research results analysis, prioritization and roadmap document, design wireframe and PSDs for optimization changes, usability or AB test results.
Why is it important to understand CRO deliverables?
The reality is that no one CRO agency or project is the same, but there are standards that should be followed. In 2014, after running an extensive survey on CRO, we were shocked to learn that people’s understanding of CRO was still quite elementary. Conversion optimization was too often mixed up with AB testing, as though the two disciplines were somehow the same. Additionally, many SEM agencies who added CRO to their services, were merely pixel moving and template tweaking, yet claiming insane and unbelievable improvements in conversion rates.
“Great processes lead to higher quality tests and to increased quantity.”
Many CROs realized the amount of misinformation was hurting us tremendously. More and more clients expectations for rapid results and transformative improvements made our jobs and our delivery suffer.
So we took it upon us, as one of the pioneering agencies in the field of CRO, to set the record straight. This article, as well as many others, is paving the way to better understanding and embracing of CRO in order to achieve what we call #websiteswelove.
While no company will fully reveal its CRO deliverable mix, certain deliverables are common with all CRO service providers. I personally believe in transparency and will share, in this post, the deliverables based on the different phases of our conversion process SHIP.
A reminder that SHIP is an acronym for SCRUTINIZE, HYPOTHESIZE, IMPLEMENT, PROPAGATE, which indicated our approach and conversion rate optimization process.
Scrutinize
70% of the work of a CRO takes place during this phase. This is the exploration and discovery phase. It is during scrutinize that a roadmap of what you will be doing on the website will be born.
Deliverables during this phase could get messy if one is not careful. There are a ton of data points that will be revealed during this phase, but the question is, how do you compile and package the information so it is easy to consume and doesn’t overwhelm a stakeholder.
What’s also important to consider, whether you’re an in-house CRO or an agency, who is exactly going to see the deliverables, and what matters most to them. Overwhelming a CMO with numbers and stats is likely not the best use of his time, nor will have enough context to make sense of it all.
But let’s see, during scrutinize we conduct:
Experiences and quick customer journey mapping, analytics assessment, polls and surveys, customer interviews, usability testing, expert review, and heuristic analysis, heatmap and video recording observations.
Each of these activities will require post activity to categorize the problems in one of 4 categories:
- Fix right away (FRA)
- Investigate further (IF)
- Research opportunity (RO)
- Instrument (IT)
If you were to take each of those activities and provide documentation of the results and learnings, it would be overwhelming. What we suggest for the scrutinize phase is a combination of many activities, but minimal documentation. How so?
Breaking it down by activity
Experiences
One of the first activity a CRO wants to do is to identify the main customer paths and experiences that would bring a visitor to the site. This is a minor customer journey map activity to help you give you context around the visitor.
“A customer journey map tells the story of the customer’s experience: from initial contact, through the process of engagement and into a long-term relationship. What do they wish to achieve, and what are their expectations of the organization?”
Expert Review
For instance, if I’m going to conduct expert analysis, we will generally collect must of our observations in an excel file:
- Description of problem
- Page
- Screenshot
- Category (fix, investigate, research, instrument)
The excel file is sheet and certainly is not something that you’d share with the CMO or client lead on the project. Internally and for people that are not as high up, the document will be useful and help you plan for the project more effectively.
Analytics
Analytics reporting can be overwhelming. There is a lot of data to consider, and not all of it relevant for the customer at a given time. However, a CRO Analytics Assessment should provide insights and details with actionable next steps based on that data. Remember that data collected during this phase is categorized as well into the: FRA, RO, IF, IT.
An analytics assessment deliverable to stakeholders should not be overwhelming or repetitive. It should contain only the very important numbers and stats segmented of course by device and source type and browser. It should also include important takeaways and next steps based on the data presented.
Heatmaps and Video recording are considered quantitative analysis. But we often create a deliverable with just our findings from video recording and heatmap analysis. The report breaks down each screen, the scroll maps, the areas most clicked, the areas least clicks and summarizes conclusions based on this. This report has proven even more valuable when we combine event tracking data and compare it to click data. We also conclude, based on clicks and events, what the visitor intent is with clicking certain areas and what can be done to improve that experience. The CEO at App Radar,Thomas Kriebernegg explains:
“By using click maps you can remove distractions on the page to streamline the user experience. It answers questions like: Are there distracting design elements that take users away from the main conversion goals? or Are the users missing out on the CTA since it’s difficult to spot?”
The activity of collecting heatmaps and video recording is also valuable during a test launch. SO comparing the data from the variation to the control can show what type of an impact the changes had.
Qualitative research
Each qualitative activity can serve as a tangible deliverable and report. Polls and survey analysis can be compiled into a report. The report would contain the findings, most interesting statistics and trends, and observations and next steps.
Closed questions (yes and no or multiple choice) can be categorized easily, however, open-ended questions are tricky and must be dissected to better understand the takeaways. We typically will read each and every response and then categorize like answers to get a good view of interests or pain points of visitors. From Nielsen Norman Group, Susan Farrell points out:
“When you ask closed-ended questions, you may accidentally limit someone’s answers to only the things you believe to be true. Worse, closed-ended questions can bias people into giving a certain response. The most important benefit of open-ended questions is that they allow you to find more than you anticipate: people may share motivations that you didn’t expect and mention behaviors and concerns that you knew nothing about.”
Customer interviews, similarly, can be compiled into sometimes a one-page categorizing responses and then another with the commentary and observations.
The Conversion Roadmap ™
This is the grand document which highlights what will be tested and when. What we typically do, is highlight the issues along with how the item will be categorized (IF, IT, RO, FRA). But then include everything with a table for easy consumption.
The research opportunities are all included in the prioritization sheet which separates problem identification from possible solutions. It also uses 18 different factors to weight each line item. Finally, it assigns different priorities for different line items.
The factors the prioritization weighs include:
- Potential Impact
- The problem above the fold?
- How did you uncover the problem?
- Was the problem discovered through expert review?
- Was the problem discovered through qualitative research?
- Was the problem discovered through user testing?
- Was the problem discovered through analytics?
- Was the problem discovered through heat maps/Video recording?
- Known top competitor feature?
- Does the test idea enhance primary conversion goal?
- Is implementing the change easy (politically, LOE)?
- Adding or removing an element?
- Does the test idea increase visitor trust?
- Does the test idea reduce FUDs?
- Does the test idea incentivize visitors to act right away?
- Does the test idea increase engagement?
- Type of change
- Type of page
- % of page views
The prioritization sheet is an important part of “the testing program” if that will be happening on the site. It is a compilation of all the research that was conducted (qualitative and quantitative) and based on that research the problems that qualify for testing.
The conversion roadmap, on the other hand, included items to be fixed right away by the client (or agency if that is the agreement), the items that need tagging (IT), and finally, areas that need more qualitative and quantitative research. Both documents are important for the success of a CRO program. Of course, the roadmap document is most important because it includes all.
Other
If you’ve conducted a usability study during the scrutinize phase, the results, like with all documentation, should be broken down, again, highlighting the most pertinent observations. Including
Hypothesize
Once you’ve arrived at this point, your research opportunities, testing ideas, have been highlighted. But usually, during scrutinize phase, just the problem (not the solution) has been gathered.
So now it is time to come up with a hypothesis based on the problem which will lead to a solution. According to Richard Bayston:
“You do not have to test your website. You test hypotheses. If they’re well-constructed and sensible you get reliable results. And if you know in advance which will likely give you the biggest payoffs, you will not wind up with inconclusive results that you can use.”
One problem can have multiple hypotheses and multiple solutions. The key is to do this activity as a group in order to limit the options.
We also find discussing “some” of the hypotheses and solutions with the client in advance makes sense to get their input as well.
Deliverables during this phase
What happens is that the hypothesis with the highest priority moves forward. We send the client a document that highlights the problem which led to the hypothesis we formulated as well as the wireframe or sometimes JPEG image of a solution.
The document is important to clarify the objectives of the test and get buy-in from the client.
Wireframes
The wireframe will generally just allow the client to understand the new layout of elements and/or the look and feel of the newly added functionality. It’s important to have this layout before starting the design, especially when there is a big change that is begin made. It is also important to include in the wireframes how each and every newly added function will work with multiple wireframes from one stage to the next.
PSDs or JPEGs of Design
Sometimes, if the changes are more subtle, we will jump into the design rather than wireframe. The image should be high quality and reflective of any updates made on the site.
Once approval is given, we then create a test plan which is automatically generated through our system Pii. It includes all the information that a developer needs to code the test: the test goals, the test format, the page sections to be tested (if an element is removed, replaced, etc.) and the PSDs of the solution image.
Propagate
Once a test is concluded, the deliverables are not done. An analysis needs to be done with the test data. We will analyze:
- Analytics
- Event tracking
- Heatmaps and video recording
- Test results
Based on this data, we come up with conclusions and next steps based on the test like:
Does the test require us to investigate further (did we have more questions based on results or hypothesis)
Did the test result in other research opportunities for further testing?
Does the test result tell us anything about the customers and site visitors?
This analysis should be shared with the client with a running reminder of the next tests slotted based on the roadmap.
One last thing
We give our clients a weekly status update sheet to remind them what activities happened and where we are at. We generate this update through our system, Pii, automatically, however, you can also create a document that gives them an idea of the progress and how much time and effort was spent on a weekly basis.
Conclusion
There are many deliverables that are produced during a CRO project. However, if there is a way to consolidate some of the information, then do it. Depending on the level of the individual receiving the documentation, adjust deliverables to what is also important to them. Otherwise, nobody will be satisfied and just end up feeling frustrated.
