What Elements Should You Include in an A/B Test?

Khalid Saleh

Khalid Saleh

Khalid Saleh is CEO and co-founder of Invesp. He is the co-author of Amazon.com bestselling book: "Conversion Optimization: The Art and Science of Converting Visitors into Customers." Khalid is an in-demand speaker who has presented at such industry events as SMX, SES, PubCon, Emetrics, ACCM and DMA, among others.
Reading Time: 18 minutes

As you start the process of multivariate or AB testing, one of the critical questions you will face is: What elements of the page should change and test?

This article will suggest many elements that you should consider when creating a test. However, before you jump into selecting elements and creating variations for them, ask yourself: Why are you picking a particular item to test?

The process of selecting elements to test on a page is where real conversion optimization happens.

If you are looking for a long-term strategy, something that will produce a meaningful impact on your bottom line, then you must follow a structured conversion optimization process.

A good conversion specialist should be able to suggest many different elements to test on a particular webpage. The real challenge, though, lies in creating a long-term strategy; creating a testing program that quadruples your online sales; creating a conversion optimization program that spans three, four or five years.

The truth about conversion optimization

We understand that you might be looking for shortcuts. You might be looking for ways to capture the low-hanging fruits first. Most of the quick wins will be a result of fixing usability problems on your website.

But fixing usability problems merely makes your website “usable.” What you should aim for is a high converting website. There is a significant difference between a usable website and a highly converting website. The mindset of visitors and how they interact with each of these websites is different:

  • With a usable website: visitors can easily shop on your site
  • With a highly converting website: visitors  want to shop on your website

The following sections will provide you with different suggestions on elements you should consider when creating your test.

Value proposition

We have seen many companies that think of the value proposition as a single statement on a website (such as the tagline). A value proposition is beyond that. It must be conveyed in the copy, the images, the different elements of the web page, the overall design of the website, landing pages, and campaign.

How should you test the value proposition?

  1. Test different statements that express the value proposition
  2. Test location/placement of the value proposition statement on a web page
  3. Test how different elements on the page can convey your business value proposition

Trust elements

We covered trust in a chapter of our book, Conversion Optimization: The Art and Science of Converting Prospects to Customers.

What symbols, designs, copy do you have to support trust? You should start by:

  • Listing all elements that increase the confidence of visitors in your business
  • The sequence of items regarding their perceived value to visitors

Notice how we use three different areas on our “contact us” page to increase visitor trust:

Invesp contact us page

Since our goal is to give visitors the confidence that we are a trustworthy business that delivers results, we use our clients’ names and logos, as well as the number of successful A/B tests we have conducted.

Notice the main image on the IRCE conference page back in 2016:

IRCE conference page in 2016
  • The image increases visitor trust by showing a large number of attendees.
  • The text below the image focuses on the number of speakers and sessions (trust indicator and an offer feature).

Here is a 2018 image from the IRCE website:

IRCE banner image for 2018

Again notice the following trust elements:

  • The number of unique companies attending
  • The number of attendees

How should you test trust factors?

  1. Test different statements and elements that increase visitor trust
  2. Test location and placement of trust elements
  3. Test copy vs. images of trust elements

The headline

It takes visitors less than 100 milliseconds to decide if they know what your website is about and if they trust it or not.

The headline is one of the first elements visitors will notice on your landing page. It is your opportunity to express your value proposition and appeal to different visitor personas.

Trigger & benefit

A “trigger” in your headline gives readers a reason to continue reading your landing page. Simple trigger words include “Learn,” “Get,” “Save,” or just about anything else that lets visitors know they are going to get something from continuing to navigate on your website.

The “benefit” outlines what visitors will get. For example: Save on Fashions (trigger) and Look Fabulous Too (benefit).

Power words

Try powerful, compelling words. Something like this:

  • Write Killer Headlines
  • Headlines that Boost Your Sales
  • A Fast, Powerful Way to Get Brilliant Results

Keywords

Keywords are the internet’s great gift to marketers. They let you know the language of your customer. There is no denying the value of keywords in your headline – not just in attracting customers, but in helping your SEO too. Try different words in assorted combinations of the other options in this list.

Length

Do any research, and you will find excellent, contrasting advice supporting short, long and about just every length of a headline on a webpage. It should be as long as it needs to be – and no more.

  • Short headlines tend to be punchier and may help your landing page stand out better on a SERP.
  • Longer headlines can help you better qualify the visitors to your page.

It has been said that shorter headlines can get you more traffic while longer ones can get you traffic that is more likely to convert.

Sub-headlines – Make sure you try combining headlines with sub-headlines to help get more information across and improve the chances that your visitors will stay.

Questions, commands & other headline techniques

Here are some samples to illustrate these options:

  • Want to Have More Effective Headlines?

(When you pose a question that your readers want to see answered, they are compelled to read on.)

  • Start Writing Better Headlines Today

(Your readers sometimes need to be told what to do.)

  • 5 Tips for Writing Irresistible Headlines

(Tell your readers exactly what they will get from your landing page.)

  • How to Write Headlines That Hook Your Readers

(Another trigger – benefit formula: “How to _____ (trigger) that will ______ (benefit).)

  • Just Released: The 5 Most Effective Ways to Write Better Headlines

(This headline makes your landing page sound newsworthy like it’s an event.)

Benefits and features

Features are the facts about a given product or service. Features of an outdoor grill might include:

•    Electric
•    Grease tray
•    Non-stick cooking surface
•    Temperature control
•    300 square inches of cooking space
•    Attached working surface

Features do not sell a product by themselves. They do not give the “what is in this for me?” answers, although they do give useful information about your product or service.

Benefits start you on the road to getting a conversion. The benefits of the above outdoor grill include:

Convenience (feature)

  • Electric
  • Non-stick cooking surface
  • Attached working surface

Easy to Use (feature)

  • Electric
  • Non-stick cooking surface
  • Temperature control
  • Attached working surface

Saves Money (feature)

  • Electric
  • Temperature control

Let’s take another example, this time we will use the features of Basecamp using their own copy.

Basecamp homepage talking about product features

We are not listing all of the product features here:

  • Pings: Personal backchannels
  • Unlimited for everyone
  • Work Can Wait
  • Campfires: Gather ‘round and talk
  • To-dos: Assign multiple people + date ranges
  • All New Search
  • Applause
  • Reports

Notice how each of the above items lists a product feature in Basecamp. How do you translate that into actual benefits for the end user? Basecamp explains the benefit to the end user for each of the above features with a paragraph.

Pings: Personal backchannels (feature)

Pings are like instant messages or direct messages. Want to get someone’s take on something before sharing it with everyone else?  Ping them!

Unlimited for everyone (feature)

For the past 12 years, Basecamp has limited the number of projects you could create based on which plan you were on. Maybe you had the ten project plan or the 40 project plan. The only way to get unlimited was to reach deep into your pockets and upgrade to the highest plan. No longer! Now every plan – even the entry-level plan, is unlimited!

Basecamp's ping feature

How should you test benefits and features?

  1. Should your primary copy rely on benefits, features, or both?
  2. How do you list benefits or features (paragraph or bullet points)?
  3. The order of the benefits list
  4. The order of the features list
  5. Should you use images to support benefits list?
  6. Should you use pictures to support your list of features?
  7. What language should you use to express/convey benefits and features (direct, hip, classic, etc.)?

Results

We discussed using benefits and features to persuade your visitors to convert. Results are even better. Results state the value customers get from using your product or service.

Going back to our grill example: what results do your customers want from your barbecue?

They want a grill that offers:

Tasty, Healthy Food Everytime

  • Temperature control
  • Grease tray

Perfect for Easy Entertaining

  • 300 square inches of cooking space
  • Attached working area
  • Easy to clean

A Double Bonus: Environmentally Friendly While Saving You Money!

  • Electric versus charcoal emissions
  • Do not have to buy charcoal for the life of the grill

How can you determine the results that your customers want? By knowing your customers and the reasons they buy your products. Also, you need to understand why other people prefer your competitors’ products. Then, go back to the question “what is in it for me?”

Tell your potential customers the exact results of your product or service. Don’t be shy. This is a case where tooting your own horn is the right thing to do.

How should you test different “results” of using your product or service?

•    Style of listing results (paragraphs or bullet points)
•    Placement of results
•    Order of the results (which result resonates first with customers and which is more important)
•    The language used for expressing results

Web copy

Many experts wrote about the impact of web copy on persuading visitors to convert. Your text as much as your design plays a vital role in conveying a concept to your website visitors.

While many companies invest a lot in creating professionally designed websites, they pay little attention to copy. However, over the years, we have seen the impact of well-written persuasive copy on convincing a visitor to convert.

If you are looking to start testing, then we can only assume that you created personas for your website. Persuasive web copy must appeal to each of your personas with the different traits within the same web page.

Let us assume the following two personas are amongst the ones you created for your website:

  • Joe, a 36-year-old college graduates who is an impulsive buyer always looking out for deals
  • Shelley, a 44-year-old stay-at-home mom who is cautious about spending her money

How do you create copy that appeals to both personas?

Joe is looking for a quick synopsis of what your product has to offer. He doesn’t need nor like to read long copy. He wants to get on your website, check out your product, place an order and leave the site within five minutes.

Shelley, on the other hand, will spend longer time, compared to Joe, reading every little word in your copy. She wants to know that the product will indeed help her and she will get her money’s worth. She would like to see product comparison charts, product features, the complaints customers had with the product, and any warranty information that you can offer.

Creating persuasive copy that appeals to both personas is challenging. It is also what makes the difference between a great copywriter and an average one.

Persuasive copy helps potential customers understand their current state and encourages them to envision how your product and service will transform their state for better, gently guiding them through the conversion process.

Remember that, in most cases, your biggest competitors are not other vendors who offer the same solution or products, but rather the current tools your visitors are using or needing.

Bryan Eisenberg, the leading conversion expert, reminds us of a critical element for a successful copy:

People rationalize buying decisions based on facts, but
 they make buying decisions based on feelings

Persuasive copy appeals first to the visitors’ emotions. It creates powerful imagery in the mind of the visitor about their state of being when they start using your product or service. In his book, Persuasive Online Copywriting,

Joseph Sugarman proposed the following structure:

  • Open strongly by eliciting interest and excitement.
  • Develop the drama; explain why the product or service is different.
  • Explain how to use the product or service.
  • Elaborate on the unique benefits.
  • Justify the purchase; identify the lasting value.
  • Address service concerns.
  • Ask for order.

How should you test web copy?

  1. Focus on money vs.time (or both)
  2. Focus on intellect vs. emotions (or both)
  3. Focus on pain vs. gains (or both)
  4. Focus on style vs. substance (or both)
  5. Style of writing (modern, classic, etc.)

Long copy vs. short Copy

We all have seen both long and short copy web pages. The debate online continues which is better. There will be no right answer ever because it ultimately depends on your product, target market and the role of each page of your site. There isn’t a “one size fits all” rule about anything within the realm of conversion optimization, and that applies to the length of copy as well.

The length of your web copy depends on:

  • the type of offer you have
  • your target market

A good rule of thumb is that the more investment your product or service requires from customers, the longer the copy needs to be. You must also remember that your copy must be persuasive regardless of its length.

How should you test the length of copy?

  1. Long vs. short copy format
  2. Different lengths of copy

The hero image

The right image can persuade your visitor to convert. While few people will disagree on the importance of pictures in supporting the online sales process, figuring out the right image to use can be challenging.

Invesp testing reveals that the right image can increase conversions by upwards of 10%.

As you select images for your landing page, consider the following:

  • Value proposition: your image should convey and support the value proposition of your business, your service or your product. Your visitors should easily connect your value proposition with the image you are using.
  • Continuity: Make sure that there is continuity in the images (color, placement, etc.) from the first touch point a person has with your business on the landing page to other areas of your website.
  • Quality: There is no good excuse for using low-quality images. Remember that your pictures say a lot about your business. Low-quality photos leave the visitor with a negative connotation of your business.
  • Uniqueness: When companies moved online 20 years ago, it was enough to get a high-quality stock photo and use it on the website. Visitors expect more nowadays. If you have the budget, hiring a professional photographer and designer to create unique images for your site is a huge plus.

How should you test images used on your website?

  1. Should you use symbolic or literal images?
  2. Should you use images of people or objects?
  3. Should you include people the image, then how many?
  4. Should you include people the image, where are they looking?
  5. Should you include people the image, what is the state of mind (happy, serious, sad, etc.)?
  6. What is the mix of people appearing in an image?
  7. In what location should you place the image?
  8. Image size
  9. Image color

Video and images

We discussed above the importance of images on increasing conversions, but, in many cases, well-produced videos outperform images in improving conversions. The video or images impact on conversion varies from one industry to the next.

Blumenthal conducted a study on the impact of having author pictures appear in search results vs. author videos when users searched for lawyers.

They concluded:

Authorship snippets have a greater positive impact on CTR’s for specialty lawyer searches than video snippets. Video snippets have a greater positive impact on CTR’s for specialty lawyer searches than having no media snippet, but less of a positive impact than authorship snippets.

Invesp’s research indicates that video had the most significant impact on e-commerce stores when retailers included videos of their staff demonstrating the use of products. The conversion uplift from such videos averaged 17%.

How should you test the use of images and videos on your website?

  1. The user of product videos vs. images
  2. The type of videos used (product demonstration, features, customer reviews, etc.)
  3. Length of video

Testimonials

Testimonials are powerful in persuading visitors that you are a credible and trustworthy business. The more testimonials you have, the better your conversion will be. Keep in mind that anonymous testimonials or testimonials with a first name only don’t increase visitor trust in your business and might actually reduce it.

Testimonials should tell the visitor that you are a credible business and that doing business with you was an excellent experience to the point that a previous customer is willing to put his name publicly to thank you. If you have video testimonials, that will be even better.

Notice how we use a testimonial from one of our long-time customers to increase conversions on our website. Not only are we publishing what the client says, but also adding the use of his business logo, 3M, to improve the buyer confidence and trust:

testimonials

Notice how Salesforce uses testimonials to persuade visitors to sign up for its product:

Salesforce using testimonials to get users to sign up for its product

Salesforce uses five different elements in designing a great testimony:

  1. Written testimonial from the customer
  2. Title of the person who gave the testimonial
  3. Video testimonial from the customer
  4. Ability to view other testimonials from other customers
  5. Emphasis on the company of the customer who provided the testimonial

How should test testimonials on your website?

  1. Written testimonials vs. video testimonials
  2. Location of the testimonials
  3. Design of the testimonials
  4. Frequency of testimonials
  5. One page testimonials vs. incorporating them into the web page copy

Eye path

People view or scan pages in a “Z pattern.” There are two important areas on a web page above the fold (highlighted in red in the image below):

Z eye pattern
  • The upper left corner of the page
  • The lower right corner of the page

Alternatively, when reading pages, people use an “F pattern.” Usability expert, Jacob Nielsen, published a study showing the “F pattern”:

F reading pattern

image credit: Nngroup

How does that impact you?

  • Use the top section of the page for your value proposition using a strong headline. Use the left side of the page for navigation.
  • Use bottom portion of the F pattern for the CTA.
F reading pattern

How should you test the visitors’ eye path?

  1. The appearance of different page elements
  2. Vertical vs. horizontal page and offer layout
  3. Placement of the CTA on the page

Live Chat

Many website visitors prefer to conduct business online in complete anonymity. However, having a live chat feature available to visitors to interact with you is a better option than forcing them to call your business. Several A/B tests reveal that adding live chat on some websites increases conversions by close to 12%.

The team at Monetate posted the following results of one of their clients:

“In the end, live chat was a winner. Version B, which included the chat widget, increased Average Order Value by 3 percent, earning our client over $20,000 during the duration of the test, and over $130,000 in projected annual revenue. Not bad for a little live chat widget.”

Remember that only a small percentage of your visitors will use the live chat feature (typically less than 3%). However, visitors who use the live chat are more likely to convert if you can adequately answer their questions. These visitors show a high level of motivation by clicking on the live chat and putting the effort to talk to your team.

To ensure live chat’s success:

  • Ensure live chat it is staffed with the right person who can adequately answer visitor’s questions. Keep in mind the challenge of international visitors who do not have full command of English.
  • If you have enough resources, we recommend starting with live chat for 24 hours/day, seven days a week. After collecting enough data, you can reduce live chat hours to specific times when there is high volume.
  • Limit the fields you ask visitors to fill out before starting the live chat feature.

How should test live chat on your website?

  1. The use of live chat vs. not using live chat
  2. The design of the live chat widget
  3. How fast does the live chat widget appear?
  4. The location of the live chat widget
  5. The text that appears on top of the live chat
  6. Time availability of the live chat widget
  7. The fields you require from visitors before starting the live chat

Form fields

Reducing the number of form fields a visitor must fill out increases the number of times the form is filled. On the balance of that, your staff would be better equipped if the visitor fills out more information.

We recommend starting with the list of fields that you must have from every contact. These will probably include a name and the best way to contact them. If you need additional information from the visitor, then cover it on a second step form which the visitor can fill after submitting the initial request.

How should you test web forms?

  1. Location of the form
  2. Design of the form
  3. Form headline
  4. Form sub-headline
  5. Number of fields in the form
  6. The design of the form CTA
  7. Privacy statement

Privacy and security

We expect a certain level of privacy and security as we navigate different websites and as we provide our information to them. However, emphasizing that you “value your customer privacy” and that you “will not share or sell their contact information” helps you increase visitor’s trust. The right place to do that is in close proximity to where you are asking the visitor for information (contact information, credit card, etc.). Emphasizing the protection of customer data is critical, especially when there is news about data theft.

Notice how Salesforce understands the importance of privacy, but improperly places the privacy statement and security icons outside the contact form:

Old Salesforce contact form showing the improper placement of the privacy statemtent and security icons.

As bad as the Salesforce contact form might be, it is still light years better than the SAP contact form that leaves the visitor confused. It is almost as if SAP does not want you to contact them and they leave the visitor with the feeling that no one will respond to your contact request:

contact-request

Notice how Macy’s places security icons in the payment section of the checkout. They are merely thrown there with little consideration for the visitor eye path:

Macy's contact form

How should you test elements of privacy and security?

  1. Placement of the security icons
  2. Placement of the privacy statement
  3. Design of the security icons
  4. Design of the privacy statement

Price-based incentives

Incentives are designed to encourage your visitors to act right away. As the headline suggests, price-based incentives encourage visitors to take action by offering a discount, freebie, or some bundling.

The challenge is figuring out how much sales you will gain by offering an incentive. If providing a 10% discount increases conversions by 20%, then this is a no-brainer. However, if giving the 10% discount increases sales by 5%, then you are getting more sales transactions but less revenue.

Notice how the category page from newegg.com uses price-based incentives to encourage visitors to act.

newegg category pages

Newegg’ approach, however, presents two main problems:

  • The design does not emphasize the lower pricing
  • No copy asks visitors for immediate action

The category page from Tigerdirect.com, a competitor of Newegg, does a better job:

Category page of tigerdirect
  • The discounted pricing is apparent with the use of the red color.
  • The copy of “save xxx instantly” emphasizes the price incentive.

Could you think of ways to improve the copy of TigerDirect?

How should you test price-based incentives?

  1. The amount of discount you should offer
  2. The design of the discount
  3. The copy used for the discount

Urgency-based incentives

Urgency-based incentives rely on time limitations to encourage visitors’ action right away. In this case, you set a deadline, and you offer an incentive to visitors to act before the deadline.

PubCon is a must-attend conference for online marketing professionals. Just like other conferences, there is an early bird discount for early registration. Unfortunately, the registration page does not emphasize the discount and the time limitation:

urgency-based-incentive

Amazon uses urgency-based incentives to increase buyer conversion:

Amazon urgency based incentives

What can you test with urgency based incentives?

  1. The incentive expiration date
  2. Urgency design
  3. Urgency incentive copy
  4. Placement of urgency incentive on different pages

Scarcity-based incentives

Scarcity-based incentives rely on quantity limitations to encourage visitors to act right away.

Let’s take a couple of examples:

  • If the visitor knows that the website has only two items left in stock, then he is more likely to act
  • If the visitor knows that a consulting firm can take one more client, then he is more likely to respond right away

Quantity limitations could be created/managed, by ordering lower quantities from suppliers, or naturally occurring, by visitors buying so many items that the product runs out. In either case, emphasizing the scarcity of an offer through copy and design can have a positive impact on conversions.

What should you test with scarcity based incentives?

  1. The design of the scarcity incentive
  2. The copy to express the scarcity incentive
  3. The placement of the of the scarcity incentive

Freemium vs. a free trial subscription model

Can you offer potential customers a free trial, a demo or an evaluation version of your product?

The goal is to have a low-barrier entry for customers. In the fermium model, you offer your product for free with no time limit. Additional features of the products are available at a price. Free trial model, on the other hand, offers the customer a free version of the software for a limited period.

How should you test the subscription model for your website?

  • Free trials vs. fermium model
  • Length of free trial (14 days, 30 days, 60 days, etc.)

Guarantees

You can use different guarantees to assure visitors that doing business with you is safe and that they will not regret it. The goal of using guarantees is to reduce the customers’ risk. Of course, every transaction in business has some risk involved in it, so by reducing the customer’ risk, you will have to carry that burden.

A good guarantee requires both parties to invest in the transaction. The customer pays a certain amount (customer investment), and in retu,rn the business promises a 100% satisfaction guarantee.

There are different types of guarantees: satisfaction guarantees, performance guarantees, money back guarantees, etc.

Follow these steps to determine the best guarantee you can offer in your business:

  1. Ask your current and potential customers to list the top five concerns or obstacles they have about doing business with you or with your competitors
  2. Assess what your competitors are offering regarding guarantees
  3. Determine your ability to offer a guarantee to reduce one of your customers’ risk
  4. Determine the business risk of offering the guarantee
  5. Test different guarantees to see what resonates better with customers

You can fill this table to help you assess your guarantee:

Business riskImpact on potential customersPossible guarantee to deal with the riskGuarantees offered by competitors to deal with this riskBusiness risk in providing the guarantee
Poor product quality
Long delivery time

A strong guarantee has the following four elements in it:

  • It is relevant to the customer’s highest risk
  • It considerably reduces the customer’s risk
  • It is specific
  • It has a long period

The stronger each of these four elements is, the stronger the guarantee will be. Strong guarantees provide a more significant change in converting visitors to customers.

Which do you think is more powerful:

  • “30-days” vs. “60-days” vs. “one year guarantee.”
  • “Satisfaction guaranteed” vs. “100% Satisfaction guarantee.”
  • “100% Satisfaction guarantee” vs. “unconditional, no questions satisfaction guarantee for one year.”

How should you test guarantees?

  • Different types of guarantee
  • Design of the warranty
  • Time used with the guarantee
  • The copy used to present the guarantee
  • Placement of guarantee

Call To Action (CTA) Buttons

Calls to actions prompt visitors to take immediate action. Supported by persuasive copy, visitors respond to your offer by clicking on the call to action, which might be either buttons or lines of text.

As you create CTAs for your page, keep in mind the following guidelines:

  • Including a no-obligation statement increases the chances of visitors clicking on your CTA.
  • Simple, straightforward CTA text beats smart/complicated CTA.
  • Tell visitors what to expect after they click on your CTA.
  • Ask visitors to take action immediately.

How should test the “call to action” on your page?

  1. Using Button vs. text as call to action
  2. The text used in the CTA
  3. The color of the CTA
  4. The size of the CTA
  5. The design of the CTA
  6. The placement of CTA

It is important to note that as long as CTAs are clear on the page, their impact on your overall conversion is minimal.

Placement of business logo

Most companies do not pay close attention to where they place their logo. Over 80% of major e-commerce websites set their logo in the upper left-hand corner of the site. However, in some instances, the location of the logo impacts the overall conversion rate.

The placement of the business logo is a quick test to implement using point and click AB testing software. While different placements might impact your conversion rate, the average uplift is minimal.

How should you test the placement of the logo on your website?

You should testing three different locations for the logo on your website:

  1. The upper left corner of your site
  2. The upper middle section of your website
  3. The top right part of your website
Share This Article

Join 25,000+ Marketing Professionals!

Subscribe to Invesp’s blog feed for future articles delivered to receive weekly updates by email.

Khalid Saleh

Khalid Saleh

Khalid Saleh is CEO and co-founder of Invesp. He is the co-author of Amazon.com bestselling book: "Conversion Optimization: The Art and Science of Converting Visitors into Customers." Khalid is an in-demand speaker who has presented at such industry events as SMX, SES, PubCon, Emetrics, ACCM and DMA, among others.

Discover Similar Topics