If you run your customer education business with profit and loss responsibility, you have a revenue and profit margin target to hit each period. Revenue and profit are just about the most objective metrics to know whether your customer education team is performing well. After all, if your training offerings are not good, or you were not effective at selling training to customers, you would not make your revenue or profit numbers, and you would be under performing.
However, if your priority is not making a margin, you likely struggle to demonstrate whether your programs are making a direct impact on the business in a positive way. After all, there are numerous ways training can be measured.
And that is the problem.
You could measure training effectiveness with learning surveys, quiz scores, product usage (if you have the tools to do this), or survey customers and ask them if they think the training was effective. This is a short list. There are many other ways.
If you really want to measure the effectiveness of your customer education program, you want to figure out whether training is having some positive impact on one or more of the top priorities of the company as a whole.
Of course, different companies have different priorities. Some are focused on growth and new customer acquisition. Others on retention. Still others are in process and efficiency-mode. Customer education teams need to understand which priorities to focus on and then create a process for determining how training is making an impact on what the business cares about most.
Undertaking any project to evaluate effectiveness is a challenge because it involved data, and multiple systems (usually), and some understanding about what to look for. This article is designed to help you create a project plan for performing an analysis of your customer education programs to determine whether it is helping the organization achieve its goals.
For purposes of clarity, we will walk through a six step process for running an analytics project and use a practical example that attempts to discover whether customer education has any impact on product renewals.
Step 1: Ask the right question
The first thing you need to do before starting any analytics project is to ask the right question. A question provides vision for where to go. It provides guidance and acts to limit the scope of your project. And limiting the scope of an analytics project is probably the most important part of it. If you do not impose constraints on yourself, you could be all over the place. Focus is key.
If we want to find out whether training has an impact on renewals, we could ask the following question:
What is the relationship between training activity and renewal rates?
This is a good start, but what does "training activity" mean? A better question would be:
What is the relationship between enrollments and renewal rates?
Enrollments is better because it is an objective measure. You could probably run a report right now to list the total number of enrollments in your training courses in the last month.
I know what you are thinking, "Wouldn't course completion be a better number to use than enrollments?" Maybe it is. That is for you to decide. We will talk about that choice in the next section.
Step 2: Define the data points
In our question, there are two data points: 1) enrollments; and 2) renewal rates.
What is the relationship between enrollments and renewal rates?
Each needs to be clearly defined to the level of an objective metric. For example, enrollments is objective. You could choose attendance. You could choose completions. Or any other metric. As long as the metric is objective, and you can get the numbers. If you do not measure completions, then you should not choose than number. So you can use enrollments or attendance or completions, so long as you can get the numbers.
For purposes of our example, we will stick to enrollments.
The second data point is "renewal rates." Number of renewals is different than renewal rate. One is a number and the other is a percentage. You could choose either, depending on what number you measure. In our example, we will choose renewals, instead of renewal rates because it is easy to run a report in Salesforce that shows all accounts that renewed within 12 months of the contract date. We would run another report that shows all accounts that do not have a renewal within 12 months of the contract date.
Notice that our question needs to change slightly to:
What is the relationship between enrollments and renewals?
Our two data points are: 1) enrollments; and 2) renewals. Now that we have our terms defined, it is time to collect the data.
Step 3: Collect the data
Since the result we are aiming to impact is renewals, we will start be collecting renewal data from our CRM. For this report, we want to make two lists. The first list shows all accounts that renewed within 12 months of the contract date. This assumes a one year contract term with renewals occurring annually. If you have different terms, you will need to adjust your report.
The second list shows all accounts that have renewal date greater than 12 months from the contract date or has no renewal. This parameter indicates the account did not renew within the contract terms.
You want to create two lists because you want to understand the different between what accounts that renewed did and what accounts that did not renew did. In other words, did the accounts that renewed enroll in training more often or have more enrollments? We would assume so, but of course, we do not know until we run these numbers.
Let's now turn our attention to the enrollment data. Now that you have two lists (renewed accounts and not-renewed accounts), it is time to collect all of the enrollment data for these accounts. For this, you will run a report from our LMS to produce that list.
Step 4: Analyze the data
In step 4, you put together the two data points. Unless you have a good integration between your LMS and CRM, this will require spreadsheets and a bit a manual work, but since we kept our question simple and defined our terms clearly, it is manageable. In a spreadsheet, you can add the account enrollment data to the list of accounts that renewed and do the same in the column for the accounts that did not enroll.
Once you have the renewal and enrollment data together, you want to find out if there is a difference between the number of enrollments for customers that renewed and for those that did not renew. This may require counting manually. If you are good with Excel, you are likely thinking about the magic of filters and pivot tables to compare these two data sets.
The simplest way to run your first analysis is to take the average number of enrollments for the renewed accounts and the average number of enrollments for the non-renewed accounts and compare the difference. This method will make a statistics major cringe, but it gives you an idea about whether there is a difference.
The next step is to figure out if the data means anything enough to help you drive changes.
Step 5: Derive insights
Now that you have the numbers and have analyzed whether there is a difference, you can dig in further to determine of you can find out why there is a difference, whether the difference is significant, and what other patterns you might see.
There is no magic to this process. It might involve looking at several accounts to see what actually happened with their renewal or enrollments. Likely the best place to start is with the extremes. For example, look at the non-renewal account with the least amount of enrollments and find out:
- Who is the account contact?
- Were invitations to training offered?
- What was the reason for not renewing?
- Is there a renewal in negotiation?
- How many possible users does this account have?
- Did any take training? Which ones?
You could take a similar approach with the renewed account with the most enrollments?
You want to look for patterns. Once you start to see patterns, you can formulate plans for making changes.
Step 6: Make recommendations
If, after this process, you believe that enrollments have a positive impact in renewals, you should create and recommend a campaign to convince the accounts with low enrollment numbers to enroll in more training. The best part about your recommendation is that it will be based on a data-drive approach and will make your job easier in convincing management to fund your proposed changes.
Collecting and analyzing data gets some people excited and drives others dizzy. If you are the latter, the good news is that you do not need to be a data scientist to create a project like we described above. I recommend you get very good at steps one and two at a minimum. Know how to ask the right question and clearly define the data points you want to analyze. Then, ask for help with the rest.
If you are good at asking the right question and clearly defining the data points, you can get a savy analytics person to help you with the rest.
Analytics is a difficult task. This blog is one small attempt to make the process less intimidating. One another attempt, we are hosting a webinar to go into more detail on how to perform a process like this using your LMS and Salesforce. Join us on Tuesday, October 25 at 10am PT. Our Learndot product manager, Jesse Miller will walk through three examples for how to use Salesforce reports to demonstrate the value of your customer education efforts.