Being data-driven and running analytics and data science and predictive modeling and machine learning and setting up a data lake and hadoop and pig and hive and... You know what? Come to think of it. All of that just makes me want to take a nap. Of course we all need to be more data-driven in our approach to running a strategic customer education operation, but analytics is intimidating. Especially for those of us who did not double major in statistics and computer science.
The good news is that we don't need to be data scientists to take a more data-driven approach. There is a simpler way. A doable way for us mere mortals. I learned that this is possible from Maria Manning-Chapman, VP of education services research at TSIA. We ran a webinar with her recently. In the webinar, she shared some data that showed her members (technology companies who run education services teams) have customers who use their product more often, use more features, and work more independently because they completed training.
Naturally, we asked her how she got this data. Manning-Chapman told us that she helped her members formulate some questions and her customers simply called their own customers and asked about their product use.
I thought. "You mean, I don't have to ask for our data scientist to run a project for me to find this out?" We don't even have a data scientist.
Sometimes, you just have to call a customer and ask. And that is your data collection plan.
This mind-blown moment inspired me to help others run a simple analytics project to answer the question that is either on our minds or the question that management keeps asking us:
What affect does training have on customers and the business?
In this blog, I describe a minimally viable analytics project that you can use to answer the question above. You don't need access to your customer database. You don't need to have a data scientist at your disposal. You don't even need a data lake (Yes, data lake is a thing). You can just do a simple project, do it yourself, and then have my permission to call yourself "data-driven" when you are done.
Step 1: Ask the right question
For me, the hardest part about taking on a research project is knowing what to research. And knowing what to research comes down to asking the right question. Until you ask the right question, you will never be able to get the data you need. Asking the right question is hard for two reasons. First, you have to actually know what you want to know. Sometimes we just don't know that. Second, if we know what we want to know, the question we ask needs to be clear and specific.
Let's start by talking about how to ask the right question so that it is clear and specific. Using an example, I think this will help you with the first part...actually knowing what you want to know.
For us, asking the right question begins with finding out broadly how our training courses affect some element of our business. This assumption automatically narrows down the scope of what your question will be because we probably want to know how our training courses affect customer satisfaction or renewal rates or product revenue or product adoption or how much time our product saved our customers. Whatever it is, we want to know how our training improved that.
With that in mind, here's how I'd create a question. I would use the format:
What affect does ___________________ (training) have on ________________ (business result)?
Using this format, our question could be:
What affect does training have on customer satisfaction?
A better question would be:
What affect does a particular training course have on customer satisfaction?
Maybe you don't measure customer satisfaction at your company. Maybe your company cares about renewal rates. No problem. Just swap customer satisfaction for renewal rates and write, "What affect does a particular training course have on renewal rates?"
You can use this format to swap variables and always be asking the right question. Well, at least most of the time.
The question you ask will also tell you where to look for the data. If your question is about renewal rates, the data will be in your CRM. If your question is about how much time or money you save your customers, you will simply need to ask your customers these questions. At least you will know where to look.
Step 2: Create a spreadsheet
Before you start collecting the data, you need to set up a simple tool so you can visualize where you are taking this project. In order to know whether your training courses have an affect on a business outcome, you need to compare two groups of customers (You could do a before-and-after for a single group of customers. There are many ways to do this). The first group is the list of customers who completed your training course. The second group is the list of customers who have not completed training. When we collect data for each group, we will compare how the data is similar or different, but we'll get to that later.
For now, you should create a simple spreadsheet with one column for customers who completed training and another column for customers who did not complete training. Keep it simple.
Your spreadsheet may look like this.
|Group A: Completed Training||Group B: Did Not Complete Training|
|Customer 1||Customer 1|
|Customer 2||Customer 2|
|Customer 3||Customer 3|
|Customer 4||Customer 4|
|Customer 5||Customer 5|
|Customer 6||Customer 6|
|Customer 7||Customer 7|
|Customer 8||Customer 8|
|Customer 9||Customer 9|
|Customer 10||Customer 10|
The point here is to create a tool for comparing the results of two groups.
Step 3: Collect the data
Now it is time to collect that data. You probably have easy access to the list of customers who completed training. You may or may not have the list of customers who did not. If you do not, go ask someone from sales or marketing. The make the request easy on them, ask for a small list. Or ask for the list of new customers during a past 30 day period. The time period will matter depending on your research question. For example, if your question is about understanding the affect of training in renewal rates, you should not use a list of new customers because new customers have not even gone through a renewal cycle yet. So, there are some details to consider about how to get the right list. But you want to get a list of the right customers who did not complete training.
Once you have both lists, you can start collecting the data. If your research question is about customer satisfaction, as in the example above, you need to get the customer satisfaction numbers and enter them into your spreadsheet for each customer. If you measure NPS, enter the NPS score. If your company uses a lengthier customer satisfaction survey, pick the score from one of the questions (and only one) and use that data point. You probably want to start with the question about overall satisfaction. Find that number for each customer and enter it into the spreadsheet.
Simple as that.
If you start with a small number of customers, say 10 to 30, this will not take you very long. And I recommend starting small so you actually start. Your friends who studied statistics in school will tell you about sample sizes and statistical significance and validity and reliability, and they would be right. But we are going for a minimally viable research project for starters. So just ignore your friends for now, and take their advice after you get good at these small analytics projects.
Depending on your research question, collecting data could involve just calling or emailing the list of customers and asking them your question. If your question is about the affect training has on product use, you could call customers and ask them, "Since you completed training, are you getting more out of the product, Yes or No?" You may find that 75% of customers you called tell you they are getting more out of the product since completing training. That would be good, right? My point here is that depending on your research question, collecting data may be as simple as calling customers.
Step 4: Analyze the data
Once you have the data entered into your spreadsheet for each column (group of customers) take a simple average of the numbers. Then look at the average for each column. Is the number different? How different? If there is a sizable difference in the numbers, there is a chance training is making a difference. At the very least, you may find that training is related to the business outcome. For example, if the average NPS for customers who completed training is 59 and the average NPS for customers who did not complete training is 49, then you may have something. If the two numbers are very close, you may need to go back to the drawing board, get a larger list of customers to collect data on or ask a different question. You are looking for a difference.
I am not going to debate statistical significance because that is above my pay grade, but understand that just because the difference is large does not mean that there is a statistical difference or otherwise that the difference is valid. It just means you found a difference and that difference is worth talking about.
Wow, NPS is 10 points higher for customers who complete our training. We should figure out how to get ore customers to complete training. Maybe we can help raise their NPS scores.
Step 5: Communicate findings and recommendations
FInally, you need to communicate your findings and recommendations to managemet. If you kept this simple, you may only have one data point to share. So you don't need to create a 74 slide presentation and schedule a formal meeting with 12 stakeholders in the big conference room. It could just be an informal meeting in the hall. However you communicate, you want to following a format like this:
- Hypothesis and recommended action
It might look like this.
- Finding: We did a little analytics project and found that customers who completed one of our training courses had an NPS 10 points higher than customers who did not complete training.
- Method: We generated two lists of customers. One of enterprise customers in year 2 of being a customer and who completed this one course during their first 90 days as a customer. The other list was enterprise customers in year 2 of being customers who never completed training. We compared the NPS score for each and found that 10 point difference.
- Hypothesis and recommendation: We believe that we can use this training course to raise NPS an average of 10 points if we can get more customers to complete training. I'd like to work with the customer marketing team and professional services to run a campaign to promote training to this particular set of customers. And do this in Q2 this year.
An MVP analytics project
Remember, this is an MVP analytics project. It is designed to get you started and to start thinking in terms of using data to make decisions. This process also provides you a method for communicating to management using data as a means for discussion and decision-making. Even if management challenges your results, they will undertand and appreciate tha you are taking a data approach. At the very least, they will respect that you took a data approach to doing something to improve the business.
Most people don't do this.
If management likes your research question, but challenges the results, you have an opporutnity to ask for support from people at your company with more analytics chops to run a real study. Of course this will make more time and effort. But at the very least, your MVP analytics project is a means to have this conversation.
The process in this blog might not hold up to high scientific standards of statistical significance, validity, and reliability, but it is data-driven. It will provide useful data. And it will help you improve how customer education can positively affect customer outcomes and business results.
[Webinar Recording] Software Adoption Crash Course for Customer Education Leaders
Maria Manning-Chapman, vice president of education services research at TSIA, will be our guest on our upcoming webinar. She will talk about why customer education is ideal for driving adoption and how to do it. Manning-Chapman will talk about the research she has been conducting, and how you can leverage your customer education progams to drive customer adoption. Watch now.