CXL Institute Growth Marketing Minidegree | Week 4 | REVIEW
Week 4 of the cxl Institute Growth Hacking minidegree.
https://cxl.com/institute/certificate/cxl-growth-marketing/
This week was all about conversion.
Qualitative research is mostly about learning who the customers are, what they want, the language they use. This is critical for copywriting, understanding friction, learning what matters to them about the products you sell, and so on.
Conversions are all about relevancy – if what you offer and how you present it matches their state of mind, you have gained a customer. If your customer is “everybody”, you’re making it extremely difficult for yourself – nobody will identify with “everybody”.
Whom to survey
Survey people who still freshly remember their purchase and the friction they experienced in the buying process. Only talk to your recent first-time customers (who have no previous relationship or experience with you that might affect their responses).
You want to filter out repeat buyers or people who bought a long time ago. If you ask somebody who made the purchase 6 months or more ago, they have long forgotten and might feed you with false information.
How many people to survey
While best online surveys are qualitative (open-ended questions), we still need a good number of responses in to get an adequate overview. If you only survey 10 people some loud voices can skew the picture, and you it’s easy to identify false patterns.
I’ve found that the best quantity is somewhere between 100 and 200. You don’t need more than 200 as the answers tend to get repetitive, and don’t offer additional insight. Remember – this is a qualitative survey, not quantitative (like opinion poll). Any less than 100, and there might not be enough answers to draw conclusions from.
If you have less than 100 people who recently bought from you, then you do with what you can get. 10 responses is better than zero.
What do the textbooks say about the minimum number of respondents?
The recurring answer to the question ‘how many’ is ‘it depends’.
You primarily are running the survey for qualitative purposes, and you are not going to need to make numerical comparisons between two data sets. This means you can feel reasonably comfortable with fewer responses. The fewer sessions you gather, the wider your margin of error becomes. For example, at 90% confidence, here’s how the margin of error looks for various sample sizes less than 1000:
If you want to increase conversions, you have to figure out who exactly is your primary target audience, what they want, what matters to them and what are the sources of friction for them. If you say your target audience is “pretty much everybody” or “anyone interested in my services”, you don’t have much of a chance to boost conversions.
As you can see, with 200 responses your margin of error is close to ±6%. That’s perfectly OK for qualitative surveys. It’s mostly about spotting patterns and learning about the voice of the customer, not exact percentages.
You can also see from the table that the relationship between the number of sessions and margin of error is not linear, and there is a point of diminishing returns. Also, the more responses you get, the more time you need to allocate for analyzing the responses. Since the questions you’re asking are all free form, it’s a lot of work.
What to ask
What you want is to get in the heads of your customers, learn why and how they buy, identify sources of friction. Responses will also help you craft customer personas that are based on actual customer (as opposed to your market team’s idea of customers).
My experience is that the sweet spot is around 7 to 10 questions. More than 10 and the number of people who take the survey goes down. Less than 7 and you might not capture as much information as you could.
I recommend asking the following questions as they give the best insight (all open-ended, free format), adjust the wording as you see fit:
What can you tell us about yourself? The goal here is to see if there are any trends you can spot (e.g. generational). If you’ve got a B2B business, ask about their industry and position in the company (and who makes the decision!). If demographics matter – respondent’s age, sex and income matter for some reason – then you might want to ask this information via multiple choice questions. Don’t ask it if it’s just “nice to know”, and not crucial.
What are you using [your product] for? What problem does it solve for you? Here you want to make sure you understand their problem, identify use cases. You might discover some unintended uses as well.
How is your life better thanks to it? Which tangible improvements in your life or business have you seen? This will tell you the end-benefit your product provides in the words of your customers. If some say really nice things, hit them up for testimonials or case studies.
What made you sign up for our product / buy from us? What convinced you that it’s a good decision? You want to know what’s working for you in your current website + identify some advantages you might want to emphasize more.
Did you consider any alternatives to our product / buying from us? If so, which ones / how many? You want to know the intensity of comparison shopping, and who people compare you to. People always comparison shop, but in some cases way more. This is absolutely critical to know in order to make a compelling case to buy from you. You can also use this information to build a ‘compare’ page where you compare yourself to the competition and make a case for your advantages,
Which doubts and hesitations did you have before completing the purchase? Identify main sources of friction, and address them (or fix them if they’re usability problems).
What’s the one thing that nearly stopped you from buying from us? This is about identifying friction again, coming from a different angle.
Which questions did you have, but couldn’t find answers to on the website? 50% of the purchases are not completed due to insufficient information. This helps you identify some of the missing information your customers want.
What was your biggest challenge, frustration, or problem in finding the right product? This helps you learn about the way people would like to buy.
Anything else you would like to tell us? Leave room for feedback you don’t know to ask.
Feel free to include some that are specific to your business.
Make sure the information you collect is actionable – don’t ask questions just because you’re curious. Once you have written your questions, go through them and ask yourself: “What am I going to do with this information once I have it?” Make sure each question contributes something unique and is necessary.
Keep it neutral: try to use language that doesn’t lead the customer any particular way. Imagine that you are taking this survey as a person with a particular set of answers. And then go through again with a different (or opposite) set of responses, and see if the question is easier or harder to answer – then adjust the wording so that it is neutral.
Avoid multiple choice (most of the time)
Note that the majority of the answers should be free-form, not multiple choice. You want the customers to be able to express themselves without constraints. You don’t know what you don’t know, and multiple choice will never reveal those things. Never ask questions like “how happy are you with X, rate from 1 to 10” – that’s useless for our purposes.
Remember, you’re after insight! It’s not about ticking the box or creating reports.
Pressure and incentivize
When sending out surveys, remember to put some time pressure on them (“fill this out in the next 3 days”) to get data faster, and remember to reward each and everyone who completes the survey (free product or service, Amazon gift card etc).
Since you want to get up to 200 responses, you want to keep it cheap / free (e.g. digital download). If you don’t offer any candy, the response rate will suffer.
Analyzing survey responses
First off lets be clear: there is no general consensus among qualitative researchers concerning the process of qualitative data analysis. There is no single right way to go about it.
What I’m telling you here is a process that has worked for me and many of my CRO peers + is advocated by some of the researchers.
The process
It’s all manual labor.
Be clear about the goals and what you are looking for
Conduct an initial review of all the information to gain an initial sense of the data.
Code the data: organize it into some manageable form. This is often described as ‘reducing the data’, and usually involves developing codes or categories (while still keeping the raw data)
Interpret the data.
Write a summary report of the findings.
While these are steps 1 to 5, I want to stress that the process of qualitative analysis is not a linear but rather continuous and iterative. It is perfectly normal and expected that you jump between all these steps, go back and forth.
Be ready to spend at least 4 hours on this, or even a couple of full working days. Don’t be afraid to put in hours to find insights.
1. Goals
Our main goal is to learn about customers. Typically we’re seeking to learn the following:
Who these people are? What are the common characteristics? Can we form some hypotheses about different customer personas?
What are the problems they are solving for themselves? We can use this in our value proposition when we state the problem we’re solving.
What’s the voice of the customer like: how they word things? Your website has to speak the same language your customers do. Notice how they describe the problem, the solution, the desired benefits.
What are the main sources of friction: doubts, hesitations, unanswered questions? Once we know this, we can take action to reduce the friction.
How would they like to buy?
Do they comparison shop? How much? This is important – if they shop around a lot, we need to stress more on our unique benefits and need to be visibly better/different from the competition.
Any insights about their emotional state?
2. Initial review
In this phase, you go in and look at the responses question by question. Some questions can be grouped together (doubts & hesitations and unanswered questions are both about friction, and “who are you” and “which problem were you solving for yourself” are both about customer personas), and thus looked at together.
The goal here is to identify trends and patterns and create a “code” for each trend. The code is usually a word or short phrase that suggests how the associated data helps us reach the goals we set in the previous steps. Make sure you write the codes down!
Coding enables you to organize large amounts of text and to discover patterns that would be difficult to detect by reading alone. Codes answer the questions, “What do I see going on here?” or “How do I categorize the information?”
In the next phase, you go in and attach codes to as many responses as you can.
NB! Beware of your bias. It’s very human to identify a couple of trends right away (at least a perception of a trend), and then only start looking for information that proves the trend while ignoring everything else. Know that this will happen to you, and self-correct when you become aware of it. If needed, take a break and come back to the data the next day. Or better yet, have a second pair of eyes
NBB! It’s also typical to only pay attention to the first 50 or so results and then skim over the last 150. First responses are in no way more important than later ones. If you get tired and notice that you start skimming over the responses, take a break and come back to it later.
3. Codification
Now that you have a list of codes, go back and attach codes to as many responses as you can (significant part of the data). Not all responses can be labeled. It’s perfectly fine to tweak, add and eliminate the codes as you get a better grasp on the data at hand. Eliminate less useful ones, combine smaller categories into larger ones, or if a very large number of responses have been assigned the same code, subdivide that category.
The goal is to link elements of the data that are conceived of as sharing some perceived commonality.
For instance, I had a client whose product was ” vegan healthy meal plans”: weekly grocery shopping list and recipes for each breakfast, lunch and dinner for 7 days.
When I read through the answers the first time, I noticed that there are 3 typical use cases:
Busy mom – too busy to think about what to shop and what to cook
Overweight or sick people – want to get healthy by following the meal plans
Vegan / people with celiac disease – people who bought it because of the gluten-free and vegan thing
So these 3 became my codes – during the 2nd reading, I went in and added comments / notes “busy”, “overweight” or “vegan”. I counted the number of responses per code to get an idea of the distribution.
This influenced how I prioritized the order of content on the sales page.
4. Interpret the data
Now that you’ve read the data so many times, what are the patterns you’re seeing? There usually things that stand out.
Write down what you can about hypothetical personas (as many as you can spot)
Count how many responses per code you have to prioritize issues
5. Summary report
Write down the key learnings (your memory is not as good as you think) to always keep them at hand for formulating hypotheses (when comparing to other sources of data). It’s also essential for your teammates and clients.
Voice of the customer
Besides identifying trends, pay attention to their language. How do they phrase the problem? Often I copy and use the exact wordings from a survey answer in a value proposition or other key part of the website copy. It tends to works extremely well.
Word clouds
Sometimes it helps to generate word clouds from the responses. Most common words will stand out, and can serve as additional insight. Note that this is NOT as useful as reading actual responses.
Here’s a word cloud of a survey I did for a Paleo diet website, and the goal of the survey was to figure out “what people want”. Look at the cloud, and see if you get any insights (without knowing anything else)?
Can you name 3 things that people seem to want?
If you’re thinking that…
their goals seem to be about weight loss, and they want to accelerate the process,
they want specific how-to guides,
and recipes for a healthy diet,
… you’d be spot on.
Now imagine if you could compare the word cloud with the actual form responses – it could serve as (in)validation to the hypotheses you have until now.
Here’s another Word cloud, formed from the answers to question “What are you using our service for?”. The goal of the question was to uncover needs and user intent, so we could adjust the value proposition and sales copy accordingly.
The service in question is about healthy 7-day meal plans.
So – any thoughts about what they’re looking for in this service?
Stuff that seems to be in your face:
vegan,
plan,
recipes,
healthy,
husband?
I would now go compare these with my notes, and if needed go back to the raw responses and look up what’s going on there with the vegan stuff and husbands.
Tools to use
Word of caution: never use word clouds as a way of analyzing the surveys without actually analyzing the responses. You’ll start to make stuff up. It’s mostly meant to summarize and double check your findings – if there’s a large keyword you don’t have a code for, might be a good idea to go back to the survey and look up responses containing that keyword.
Tools to use for conducting surveys
I absolutely love Google Docs forms. It’s simple and free. (But some don’t since it doesn’t have bells and whistles, and looks kind of plain).
Typeform. Most beautiful surveys.
Comments
Post a Comment