As L&D professional one of the main challenges you might be facing is demonstrating the value of a L&D initiative like a training or e-learning within the organization. While e-learning promises cost-effective mechanism of training in the long run, how do you successfully evaluate the effectiveness of the e-learning? While the simple answer is to calculate the e-learning program’s Return on Investment (ROI) , the difficult part is to establish a linkage between the employee’s/learner’s performance and e-learning. However, an attempt to establish a correlation between improvement in performance and e-learning is quite doable and can definitely support the demonstration of measuring the Return on Investment (ROI) of an e-learning program/s.

Hence, let’s look at the popular ways of evaluating or measuring the effectiveness of e-learning or how to go about calculating e-learning  ROI.

What Is RoI And What Is The Importance Of Considering RoI?

ROI is the return on investment that an organization gains (ROI = Gain or Return/Cost).

ROI calculations are critical to the operations of any enterprise, as they allow them to evaluate performance and determine the profitability of an expenditure or initiative.

Below are some of the reasons that make every organization to calculate the ROI of a training program.

•To justify the training budget (to check and allocate the funds towards training initiatives).

•To determine the effectiveness of training

•To evaluate the training method used

•To determine if there was a change in time, cost, or behavior

•To provide evidence to management/stakeholders

•To gain credibility with executives

•To identify areas for improvement.

So.. How can we measure the e-learning ROI ?

While there have been efforts made to measure e-learning ROI, there is no established or time tested, model which does it. However one of the most popular ways to measure e-learning ROI is to use the Kirkpatrick’s model

Dr. Donald Kirkpatrick came up with his model for evaluating the training effectiveness in the 1950s. Dr. Kirkpatrick in his model highlighted the need to consider the evaluation at four different levels. Let’s quickly look at these four different levels

1.Level I Reaction

The degree to which participants find the e-learning favorable, engaging, and relevant to their jobs. It talks about understanding the response, opinion and behavior of the learners like –

  • What are the feelings and reactions of the learner towards the e-learning course?
  • To what extent is the learner satisfied?
  • Was the e-learning relevant?
  • What was the level of participation?

Here are some ways to measure the learners’ reaction to e-learning

To analyze the learner reaction, an organisation must focus on two primary factors i.e-

The number of learners who attempted the course and the number of learners who completed the e-learning. The organisation can easily track it through features in the LMS.  

For example, when a new compliance e-learning course is launched, an organisation can set parameters like –

•The average total number of days it will take for the learners to attempt and finish  the e-learning

•Expected percentage of learners who will complete the e-learning

•Expected percentage of the learners who will at least attempt the e-learning

•Expected dropouts from the e-learning

•The number of attempts made by the learner to complete the particular e-learning.i.e if the learner had completed it in one single attempt or multiple attempts .

Once the learner completes the e-learning, an organisation can measure the reaction by comparing the actual value with the set parameters. For example, an organisation can expect a 60% completion rate for compliance e-learning within one month of the launch.  Now this can be measured with the actual completion rate within a month to understand and gauge the learners’ reaction towards the e-learning.

Feedback is another popular way to measure the reaction of learners toward e-learning. But I don’t recommend simple feedback forms or questionnaires as it can just be a ‘tick box exercise’ as the learner might not give genuine feedback. Hence, if you are choosing to include feedback, make sure that there are open-ended questions too and they are included in the relevant section during an e-learning.

2.Level II Learning

The level I talked about taking the feedback and opinion of the learners. But the learner reaction doesn’t always reflect what they have learned in real.

The most popular ways of measuring learning are:

•Including e-learning assessments to test the knowledge of the learners after regular intervals during e-learning itself. For example, after the completion of each learning topic, the organisation can include an e-learning assessment. The assessment can be in the form of fill in the blanks, drag and drop, MCQ’s  and even game based assessments and e-learning scenarios.

• Conduct post e-learning assessments after a period of time to evaluate the success of the knowledge transition. This can be done both online and offline. The organisation can use an online question bank to test how much the learner can recollect and retain. I recommend to include case studies and use examples that reflect the situations that happen in a real work environment to test the knowledge of the learner.

•Apart from this, an organization can also take feedback(quantitative and qualitative) from the superior or boss of the learner to understand the knowledge level and performance of the learner.

3.Level III Behavior

Level III discusses the degree to which participants apply what they learned during training to the job.In other words putting the learned things into practice. Behavioral changes are very difficult to assess and it’s reflected only when the employee applies what is learned.  Sadly, the fact is that behavior changes can be only induced among the learners by including it on the performance evaluation mechanism. Hence, I recommend to make the learners aware of the expected behavior change and is included in their performance evaluation mechanism.

In e-learning, an organisation can use the strategies like e-learning simulations and scenarios to know how much are the learners able to put the learned things into practice.
By simulating a work environment online, an organisation can allow the learner to put the learned things into practice and test his/her knowledge and skills acquired and measure it with the expected behavior change.

4.Level IV Results

The next step is to determine how behavioral changes affect the organization. Measure and analyse the impact and effectiveness of the implemented e-learning initiative. For this calculate the total cost incurred as well as the total benefits derived out from the training program. To make it simple here is a general formula

ROI %= Total Program Benefits – Total Program Costs / Total Program Costs *100

Any result over 100% means that the e-learning has a net benefit and any result below 100% means that the e-learning has a net cost. While it is comparatively easy to estimate and arrive at the costs associated with the e-learning course, it can be too challenging to estimate the real benefits derived from it.

Let’s analyze the various costs of e-learning  development and implementation:

Content creation/purchase: Include the cost of the e-learning content. For example, if you are buying the online course then it includes the billed price and the cost of hours you spend on customizing to the requirement. So it will be content price + (hours spent on customization * your hourly rate).

Software purchase/development: The cost involved in purchasing or developing a software to create as well as run the course. For example, do you need to purchase a new authoring tool to create the e-learning course If you don’t already have an LMS, perhaps you need to buy or build one? This cost should also be included in the calculation.(If the LMS cost is just a one time cost, the organization can decide to spread out the cost by averaging with the planned numbers of e-learning program over a period of time say 5 years)

The cost of time : It is very important that we also include the cost of time as it should also be reflected in the total costs. For example, while discussing the content or at the development stages you’ll need to involve the management and sometimes SMEs’. Remember that the time spend is not free and it comes as a cost to the company. Hence, all these hours should also be accounted for (your time * your rate in hours+ management/SME time *rate in hours ) will give an estimate of total time cost.

Lost production time: Not many organisations consider this as they feel that the time trainees spend in training is usually the time taken from their work. But I recommend considering the lost production time and it is easily calculated by multiplying the number of employees taking the course by the number of hours it takes to complete multiplied by the hourly rates. (number of employees taking the course *number of hours it takes to complete * hourly rates.)

While the cost of e-learning is a rather simple parameter as it seems quite measurable but the real challenge.is to calculate the benefits of the           e-learning especially in numerical terms when it’s non tangible?

How do you ensure that the benefits are the result of the e-learning program alone?

What if some external elements like the market factors influenced the results?

For example,a good season time for the sales or a successful marketing activity can influence the sales results and give a boost to the sales revenue chart but how much of it can be attributed to a salesperson undergoing an e-learning program is difficult to determine. Hence, it’s very important to ensure that such factors are either accounted for or eliminated from while measuring e-learning ROI. (The organization can depend on the historical sales chart to average out and arrive at a normal sales figure for the sales period)

To determine the benefits derived from the e-learning program one need to set and implement KPIs (Key Performance Indicators) even before the implementation of the e-learning program and decide on how much they should change after the learner completes the e-learning.

To summarize the process of projecting the benefits it involves the following steps:

•Identify and agree on the KPIs

•Agree on how much you want them to change after the training program (%)

•Calculate how much monetary gain this change will bring

From our experience, identifying the KPIs is not the job of the L&D department alone Instead it requires conducting an in-depth discussion with the respective departments and stakeholders to identify and fix the right KPIs for the particular training program.

For example, if it is a sales related e-learning, the KPIs can include the following:

•Monthly Sales Growth pre and post e-learning

•Monthly Sales Bookings pre and post e-learning

•Sales Opportunities pre and post e-learning

•Quote To Close Ratio pre and post e-learning

Once the learner completes the e-learning, the organisation can compare the projected benefits with the post-evaluation measures i.e

•Calculate the change in the KPIs(comparing with projected KPIs growth)

•Calculate how much monetary value it brought(compare it with the projected monetary value)

•Ensure that you eliminate/isolate the effects of the other external parameters to make sure the change is caused by the e-learning and not some other circumstances.

Defining and evaluating KPIs can be one of the ways to measure the total benefits derived from the e-learning and now we are ready to put the figures into the formula to obtain the ROI.  Does this sound doable? I would like to hear from you i.e how do you measure the ROI in your organisation. Feel free to share your suggestions and you can reach me at hello@playxlpro.com