CIPR Skill Guide Internal Communications: Measurement & Evaluation Applying the PRE Cycle to Internal Communications

Introduction Measurement & evaluation is at the heart of successful, strategic internal communications (IC). Without it we cannot possibly demonstrate the value we add, understand whether our programmes and campaigns are hitting the mark, or assess return on investment. And yet it is an area where few communication departments excel. There are a number of reasons why communicators have not always embraced measurement & evaluation. One is a lack of understanding - driven in part by the perceived complexity of the subject (most of us are, after all, more comfortable with words than numbers and analytics). Another is budget restraints, though the truth is an awful lot can be achieved on a shoestring. Others believe that their money is better spent on delivery - though one would argue there is little point in that if you do not know whether your tactics are working! The reality, of course, is that internal communication can be measured, often quite simply and inexpensively. This guide provides a short overview of the CIPRs own Planning, Research and Evaluation (PRE) process and applies it for the first time to internal communications. This established and widely regarded tool was originally developed for the evaluation of externally-focused PR activity, but it can be applied equally well to employee communications. Introducing the CIPRs PRE cycle PRE is a simple five-step process which sets out how and where to use research and evaluation techniques to inform your planning, set objectives, track effectiveness during a programme and evaluate success at the end of it.
AUDIT Where are we now?

1 5
RESULTS & EVALUATION How did we do?

2 3

OBJECTIVE SETTING Where do we want to be?

ONGOING MEASUREMENT Are we getting there? STRATEGY & PLAN How do we get there?

PRE involves a constant cycle of research and evaluation, rather than seeing it as something you do only at the beginning or end of a project. Its an integrated planning process with measurement and evaluation at its heart. As such it should be used to inform and shape your plans, track progress and, in due course, evaluate the success of a programme in delivering its stated outcomes.

Step 1: Audit Gathering information and conducting research to build a solid foundation for your IC strategy. The audit stage - which culminates in the development of a clear project brief - is the critical first step towards developing a robust internal communication strategy. It basically involves gathering as much information as you can about the issue or challenge youre facing, the organisational context and the communication requirements. An audit should be conducted before you start to think about implementation, creative execution, channels or campaigns. The brief itself should be developed in conjunction with your internal client and should cover the following areas: Overview of the organisation - description, background and history, vision and values, strategic goals, business objectives, structure; Internal communication requirement - current state, audience breakdown, key messages, desired response/objectives, potential measures; Internal comms background - history and previous activity; Supporting material - examples of existing channels/materials, style guide, case studies, etc; Integrated communication - what else is going on inside the organisation, planned announcements, change programmes under way; and Process - budget, timings, etc

Step 2: Objective setting Setting clear, meaningful and relevant communication objectives The starting point for developing any internal communication strategy is to establish what it is you want your audience(s) to think, feel or do to identify your desired communication outcomes. As we will discuss in the next section, outcomes are a very specific sort of objective. Good internal communication objectives have four key ingredients, and if one of these is missing your objective is not complete: 1. Audience in identifying your audiences you need to be as specific as you can the more accurately you can describe your audience, the more effective your message will be. 2. Outcome what is the business goal you are aiming to support and what action do you want people to take as a result of the communication? Here are some examples of tasks you might want to accomplish through a communication programme: Increase awareness; Build understanding; Change perceptions; Increase participation; and Encourage or discourage a particular behaviour;

3. Target a clear, measurable level of achievement you are aiming to deliver (from X to Y) and how you will measure success or failure. 4. Timeframe realistically by when do you hope to reach the desired level of achievement? E.g. 90 days, 6 months, by end of financial year. As you will now discover, the work undertaken during the audit stage will help enormously when it comes to objective setting. From your knowledge of the audience you will be able to clearly define who you need to reach. From your discussions with your colleagues or clients you will have an understanding of the outcome youre seeking, how it supports wider business objectives and what needs to happen when. From your review of existing data you will understand where you are now (having identified a benchmark measure) and where you need to get to.

Remember, the more precise your objectives, the easier it will be to quantify the outcome and determine the return on investment (ROI) further down the line. But be careful, one of the commonest mistakes in the evaluation process is to confuse outcome (opinion and behaviour change) with output (evidence that a message was delivered). Remember that, unless agreed with your internal client at the outset, the mere delivery of a conference or publication, even with 100% attendance or readership, is unlikely to be viewed as a successful outcome. Step 3: Strategy & plan how do we get there? Step 3 is about deciding what types of research you will need to use to develop your plan (the document detailing how you will deliver your stated objectives), to track progress and to measure the effectiveness of your programme or campaign along the way. A key part of developing your plan is to determine how it will ultimately be measured. There are four types of measurement to be aware of here: Input Whats already out there? Background information and research to inform your initial planning. Some of this information could provide benchmarks against which to measure later. This is the sort of information you made use of during the audit stage. Are your messages getting to people? Used to measure WHAT we did message exposure and audience reach. A valuable quantitative measure focused on the efficiency and effectiveness of your processes. It can analyse message exposure but it cannot explain to what extent peoples opinions or behaviour has been influenced. Are your messages getting through to people? Used to measure THE IMPACT of what you did the extent to which the audience is aware, has understood and remembered and their likely reaction and response. Out-take research is often carried out during a campaign or programme to check whether messages are getting through to people. Are your messages changing attitudes or behaviour? Used to measure the TANGIBLE RESULT of what you did the degree to which comms changed awareness, opinions and behaviours. The most valuable form of measurement, it provides concrete proof that a change in employee behaviour can be traced back to an internal comms programme, activity or message. It is also the basis for calculating ROI.

Output

Out-take

Outcome

A wide range of tools and techniques are available to evaluate IC effectiveness at all four levels. Your choice will depend on a number of factors particularly how much depth and detail you require, your time availability and your budget. A simple menu of research methods is shown below, according to degree of sophistication. Externally sourced research will generally be more reliable, but there is a huge amount of valuable information that is likely to exist already inside the business and that you should be able to access quickly and easily. Basic Communication team can source, undertake and analyse internally. Desk research and information gathering Collecting demographic data Telephone interviews Pre testing of messages Intermediate Managed by communication team but may require specialist external input. SWOT analysis Focus groups Pulse surveys Content analysis (channels) Audience profiling Advanced Relies on specialist external advice, managed by senior member of communication team. Key driver analysis Telephone surveying Major employee surveys

External benchmarking Developing a brief

Creating a communication dashboard Creating a communication scorecard

Qualitative and quantitative research When considering what techniques to deploy you should also be aware of the differences between qualitative and quantitative research. Qualitative research involves investigating participants' opinions, behaviours and experiences from their point of view. It differs from quantitative research in that it does not rely on mathematical-based modelling and analysis, but instead uses logic and gut feel to decipher gathered data. Its downside, compared to quantitative research, in that it is more resource-hungry and therefore expensive and, because it usually relies on much smaller sample sizes, is less reliable scientifically (conclusions are based on interpretation rather than fact). Qualitative techniques that are popular with communicators include interviews and focus groups. By contrast, quantitative research deals with numbers and anything that is measurable. It usually involves the use of structured questions where the response options have been predetermined and typically touches a large number of people. Example quantitative techniques include surveys and questionnaires, which can be delivered in a variety of ways on paper, electronically or face-to-face. The annual employee survey is an example of a popular quantitative technique. The measurement pyramid The illustration below shows how the four levels discussed earlier inputs, outputs, out-takes and outcomes can be combined to create a robust PRE structure.
Degree to which IC has changed awareness, opinions and behaviours

Outcomes

Results versus objectives Hard evidence (quantitative sales, financial, etc) Soft evidence (qualitative focus group feedback, etc)
Extent to which audience is aware, has understood and remembered; their reaction and likely response

Out-takes

Number and type of questions submitted Pattern of intranet hits Awareness measures- is the message getting though? Likely intentions of audience
What message went out; exposure and audience reach

Outputs

Material issued number of newsletters distributed, publication of related

stories on intranet, etc

Events who and how many attended Frequency of exposure (how often did we communicate key messages to
each audience?)

Background information and audit research

Inputs

Communication brief Desk research (demographics, survey data, content analysis of channels, etc) Qualitative research from interviews and focus groups

Source: Based on Top 10 Research methods for planning & evaluation in How to Measure PR and Corporate Communications published by MASS Communication Group. Jim R Macnamara.

Step 4: Measurement are we getting there? Internal communication is sometimes about communicating simple messages quickly, but more often its role is to influence employee opinions, change attitudes and spark behaviour change. Because those results often take time, it is vital to check on progress regularly at key points throughout the process, as well as at the end. This enables remedial action to be taken and plans to be adjusted if necessary. Again, you can use a wide variety of research techniques, qualitative and quantitative, to track your progress. See what the findings suggest you continue, stop or start doing and examine the reasons behind any surprising or unexpected results before taking action. There is also very often a need to prove to your leadership team or client that what you are doing is having an impact - perhaps in order to secure additional budget for the project or to back-up the business case for further investment. Step 5: Results & evaluation - how did we do? If you follow the PRE process properly and, importantly, if you develop a number of clear, meaningful and measurable objectives at the outset (stage 2) you should be able to make a direct comparison between those objectives and the results you actually achieve. This will enable you to determine the overall success (or failure) of the communication programme. Depending on the extent to which you included financial measures and/or linked your objectives back to wider organisational objectives, you may also be able to determine the ROI. For more information about the PRE cycle see the CIPR PR Toolkit: Media Evaluation Edition, which is available to members priced at 35 and non-members at 86. A series of best practice case studies on this topic can be found in the information centre along with other skills guides.

By Lee Smith FCIPR Lee Smith FCIPR is an award winning communicator and one of the UKs leading bloggers on internal communication. In 2006, he co-founded the UK-based internal communication agency, Gatehouse (www.gatehousegroup.co.uk), which provides employee engagement and behavioural change advice and consultancy to organisations ranging from large businesses through to the government sector and charities. He holds an MSc in Corporate Communication & Reputation Management and is a Fellow of the Chartered Institute of Public Relations.

Downloading and reading this skills guide is worth 5 CIPR CPD points. www.cipr.co.uk/cpd