evaluate training effectiveness

Can you evaluate training effectiveness?

To evaluate training effectiveness you must consider two aspects:

What are the cost benefits?

Has the training met your stated aim?

In most cases senior management will want to know the impact of training on the bottom line. That is, was the training more than just an expense to the business. This is not an easy task.

In this article I shall look at how we can calculate a return on investment in training (cost benefit) and then consider various means to measure how effective your information assurance (IA) training was in achieving your aim.

I hope to demonstrate that you can provided acceptable data to management on both counts.

Recapping Purpose of IA Training

In an earlier blog article I addressed the question of why an organisation should provide IA training to all levels of staff.

In that article I defined the aim as:

“To ensure users are aware of information security threats and equip them to support your organisation’s security policy in the course of their normal work.”

I concentrated on looking at educating your people in your IA requirements as one of the many security controls you can put in place. You should gear your IA training to affect staff behaviour towards security in support of your organisation’s security culture.

I discussed the purpose of IA training and looked at what constitutes appropriate training. It is important your IA training is relevant to a person’s role and experience within your organisation.

After looking at various delivery methods for your IA training the article concluded with a brief consideration of the benefits to the organisation.

The following quote from Kevin Mitnik sums up the need for training and education succinctly:

“The methods that will most effectively minimize the ability of intruders to compromise information security are comprehensive user training and education. Enacting policies and procedures simply won’t suffice. Even with oversight the policies and procedures may not be effective: my access to Motorola, Nokia, ATT, Sun depended upon the willingness of people to bypass policies and procedures that were in place for years before I compromised them successfully.”

Having briefly considered the benefits of IA training I stopped short of looking at how we can determine the effectiveness and cost benefits of the training to the organisation.

This article looks at those issues in more detail.

So, having delivered your IA training at the appropriate level to your staff ask yourself the following questions:

Can I demonstrate the cost effectiveness of my IA training to management?

What is the return on my investment in training?

How do I evaluate training effectiveness?

How do I know if my training has met my immediate needs?

How do I know my training has been effective in contributing to or creating my security culture?

In the following paragraphs I shall build on my original article and answer the three questions above looking at ways in which you can demonstrate the effectiveness of IA training.

First, let us look briefly at return on investment (ROI) in general.

So, what is ROI?

Traditionally, ROI has been a simple tool by which an organisation assesses the most cost effective solution to a problem or the most profitable investment.

For example, a factory needs to update some plant to improve output.

Procuring new plant comes with a price tag of £500,000 and the production manager estimates the improved output would generate revenue of £2,000,000 over 5 years. The expected revenue would be 4 x the investment over the 5 year period.

The senior management can now decide if that would be a sensible return on their investment.

This simple example illustrates the general principle of ROI.

Does ROI work for Security?

But does this model of ROI work for security? Basically, it does not because security is not a thing with inherent value, unlike the new plant in the factory example above.

In the majority of organisations it is difficult to convince senior management of the value of security investment other than in terms of insurance against bad things happening. Management often considers security a necessary evil with an associated expense.

A better view might be that the investment in security enables the organisation to operate efficiently and expand into new areas of business.

But what about estimating the return on security investment (ROSI)? Is it possible?

In short yes it is possible but involves a thorough understanding of the organisation’s exposure to risk. (Incidentally, the organisation’s risk appetite also comes into play.)

The risk assessment process takes into account the value of the assets to be protected (tangible and intangible), as well as the probability of the risk being realised.  The organisation then calculates the Annual Loss Expectancy (ALE).

Balance the ALE against the annual cost of security, so that the cost of security is in proportion to the potential damage to the organisation.

As a simple example, after conducting a risk analysis a small e-commerce company calculates their annual risk exposure to a certain cyber attack to be £10,000. That is, if an attack was successful it could cost the company £10,000 worth of lost business over the year.  Their ALE is £10,000.

The company investigates their options for hardening their system and learn that an upgrade costing £5,000 would mitigate the risk exposure by 75%. That would indicate a ‘return on investment’ of £7,500. This indicates to them that the upgrade investment could reduce losses due to that attack to £2,500.

So the organisation recoups the cost of the upgrade through the reduction in risk exposure.

This is a simple illustration but, for a more in-depth look at ROSI, ENISA published a useful paper here which looks at how CERTs can demonstrate returns on investment.

On the other hand, a Bruce Schneier article from 2008 provides a rather more pessimistic view of the subject of ROSI:

“ROI” as used in a security context is inaccurate. Security is not an investment that provides a return, like a new factory or a financial instrument. It’s an expense that, hopefully, pays for itself in cost savings. Security is about loss prevention, not about earnings. The term just doesn’t make sense in this context.”

In his piece Bruce discusses the issues surrounding ROSI looking at spending on security problems weighed against the benefits. That is, not spending more on security measures than the value of the business benefits they bring.

Bruce also suggests the data from cyber security incidents is not really good enough to make accurate calculations of ROSI. Businesses tend to end up relying on the figures provided by vendors.

“This (kind of thing) is why most ROI models you get from security vendors are nonsense. Of course their model demonstrates that their product or service makes financial sense: They’ve jiggered the numbers so that they do.”

As Bruce states in his article “Caveat emptor”.

So, having looked at ROI and ROSI, we can see it is possible to derive meaningful numbers in support of security investment.

Arguably, calculating the effectiveness and cost benefit of IA training should follow similar lines to ROSI.

Let’s take a look.

Calculating Return on Training Investment (ROTI)

Please remember my context for this discussion is IA training. Other subject areas might need to address different, or additional, elements of analysis.

As with ROI and ROSI, ROTI is effectively a measurement of the cost of training compared to the benefits accrued.

The true cost of providing training is often taken for granted. Frequently, management and clients will express surprise at the price quoted for a training course.

This surprise is generally the result of a lack of understanding of all aspects that go towards developing and delivering training. So let us look at what contributes to the true cost of training provision.

So how do we calculate the costs of training and place a value on the benefits accrued?

Elements of Training Cost

Delivering training is comprised of several activities each of which must be considered carefully:

  • Design and Development – Are you developing the training in-house or are you using external specialists? Are you buying the training off the shelf?
  • Promotional – How much internal effort are you expending? Any external help? Cost of promotional materials both internal and external (posters, pamphlets, etc).
  • Adminstration – this element of cost is easy to overlook. Apart from admin support to the training development team there are the costs of support to the students attending training; eg joining instructions, registration, billing.
  • Training staff costs – staff costs will be predicated on student numbers and training hours for the programme. Style of training, classroom, CBT, one to one will affect staff costs. Do not forget preparation time and expenses.
  • Materials – student training materials, books manuals, etc. If you are buying off-the-shelf licence fees must be considered.
  • Facilities – the cost of your training internal facilities might be part of your general overheads but you should break out the costs of own facilities.
  • Student costs – more complex than might be first thought. You need to consider the cost to the business of a student’s attendance on training. If the student undertakes training in what would be considered normal productive time the true costs become more complex to calculate – lost production, lost sales. Otherwise this element simply considers travel and expenses costs per student.
  • Evaluation – consider the costs of training evaluation including the ROTI calculation.

Financial Benefits

The most difficult part of ROTI calculation is to put a cash value on the benefits of a training programme. As we shall see later the real benefits of our training are in changed behaviour and improved performance – both of which are difficult to quantify financially.

Some areas which might enable us to put a financial value on the benefits of training include:

  • Labour savings – staff retention.
  • Productivity – reduced down time due to security errors.
  • Other savings – improved sales prospects if security accreditation is a key element of a pitch.

Calculate the ROTI

Calculating the ROTI as a percentage is relatively simple:

ROTI = benefits/costs x 100

You must decide over what period you wish to accrue benefits.

Let us say the cost of some IA training for a number of staff is £10,000. Considering all aspects above we estimate that over 24 months we shall accrue benefits valued £20,000.

The ROTI would be 200% over 2 years. Now management can judge whether the training investment provided an acceptable return.

Management could also calculate how long it would take for the training to payback their investment.

Period = costs/monthly benefits

In our simple example the training would pay for itself in 13 months.

One argument against ROTI is that it is effectively a single snapshot whereas an organisation is really interested in evaluating the benefits over a period of time.

Therefore, we need something more than a simple ROTI calculation. We must ask:

How effective was the training?

Did we achieve our aim for the training?

Let us look at how we can evaluate training effectiveness especially regarding IA training.

Can I Evaluate Training Effectiveness?

Generally, what we are attempting to achieve through IA training is a positive change in staff security behaviours. We need to develop a security culture as a core part of our business strategy and operations.

We achieve our security culture through education, training and awareness.

Organisations adopt many ways of evaluating training effectiveness a selection of which follows:

Very often organisations evaluate training on the basis of the quality of the material presented. The assumption here being if the training material and presentation are of high quality then so must be the learning.

‘Happy sheets’ are a common method of assessing quality. These are basically questionnaires that students complete at the end of their course to rate how content they were with the standard of training provided.

Tests at the end of a course of training have long been the traditional method to evaluate training effectiveness. The stock favourite assessment has been the multiple choice ‘vote for Joe’ exam. However, this only tests short term retention of knowledge and not necessarily understanding.

Feedback from management at staff performance reviews can help identify whether the training was effective in delivering the required behavioural changes and new skills.

Ultimately, we must ask whether the training has met our objectives and achieved our aim. We need to know whether the training has delivered the improvements in staff behaviours we seek.

We must identify the real impact of the training in the long term. Observation of staff in the course of performing their duties is a good way to do this.

One sure way to determine training impact is through students demonstrating their new knowledge and skills.

John Eades, CEO LearnLoft, suggests 3 ways for measuring training effectiveness in his article 3 Ways Organizations Can Improve The Way They Measure Training Effectiveness :

Visual confirmation where students submit a video or audio recording, or maybe a screen shot, as proof of a completed task or operation;

Social ownership whereby students can demonstrate their full understanding of a subject by teaching others. This could be in workshops or at their normal place of work. This route will help to engage staff in training through peer-to-peer teaching and learning;

Skill assessment through a before and after training review. For example, comparing staff members’ approach to selecting passwords before they receive training in the organisation’s security policy to that after they have completed training.

Getting students to demonstrate their understanding and new learned skills at some point after completing training can be an effective route to evaluating the lasting impact on your security culture.

This demonstration of understanding can be viewed as an evaluation of a learning outcome rather than assessing the process or inputs.

Focussing on learning outcomes is a key finding of recent Chartered Institute of Personnel and Development (CIPD) research into learning and development (L&D) evaluation.

“An immediately obvious implication of our L&D evaluation research is the need to focus on learning outcomes, which may be broadly defined as some permanent or long-lasting change in knowledge, skills and attitudes, and which is an output or outcome, rather than on any training itself, which is an input.”

This statement is particularly apposite to IA training where we are not just teaching new skills but seeking to change behaviour in support of our security culture.

Ultimately, an organisation must align IA training with its security policy framework. There must be traceability between the IA training programme and the organisation’s security policy which must in turn be integrated into business operations.

In this way any IA training delivered links back to business operations.  This link to business operations is ultimately the positive argument for the training to take place.

Conclusion

At the start of this article I posed the question, “Can you evaluate training effectiveness?”.

By looking at return on investment calculations, some common training evaluation methods and touching on evaluation of learning outcomes, I believe the answer is a qualified yes.

Getting the training ‘right’ is vitally important otherwise why do it in the first place?

It is possible to align IA training with business objectives and operations as long as a sound security policy framework is in place.

We can evaluate training effectiveness in financial and learning outcome terms in ways that senior management can understand and appreciate.

However, the sad fact is training is often the first casualty when the going gets tough and organisations trim budgets. So perhaps at those times it is more important to be in a position to prove the positive contribution IA training brings to the business rather than worry about the financial numbers too much.

In this article I have looked at just a small selection of approaches to the evaluation of training effectiveness.

<span “>There is much advice, guidance and academic research available on the subject.

So you can do it. How you choose to do it is up to you and your organisation.

Contact me here if you would like help with any aspect of IA training for your organisation.

References

Return On Security Investment (ROSI) – A Practical Quantitative ModelWes Sonnenreich 2006

Assessing the ROI of Training – Clive Shepherd, Fastrak Consulting Ltd 1999

Evaluating learning and development – CIPD 2015

Posted in Opinion, Training.