Developing Simulations

Situational Exercises: An Added Degree of Realism

This guide provides practical information on the development of situational exercises simulations. It deals with the selection of the content of the simulation exercise, the presentation of problems to candidates and the evaluation of candidates' performance. Situational exercises or simulations place candidates in realistic situations which require them to demonstrate job-relevant qualification(s). The content of the simulation exercise may be derived directly from actual work settings or may be designed in a more general manner by presenting situations and problems which are not specific to a particular work situation. A notable characteristic of situational exercises is that candidates must demonstrate, at the time of the selection process, actual performance of the job and job tasks. Situational exercises may be used to assess simple as well as complex qualifications. For example, simple qualifications, such as the ability to operate a given piece of machinery, are readily assessed by requiring the candidates to operate the machinery. More complex qualifications, such as the ability to solve administrative problems, can be assessed by presenting candidates with a realistic administrative scenario which requires them to solve relevant problems. Situational exercises may also be used to assess more than one qualification at the same time. For example, a situational exercise designed to assess the ability to solve administrative problems could require candidates to make a presentation of their proposed solutions to a given administrative problem. This would allow for the assessment of their oral communication skills in addition to their ability to solve administrative problems.

Developing Situational Exercises: 3 Basic Steps

The information derived from situational exercises must be relevant to the qualification(s) being assessed. In following the 3 basic steps outlined below, you ensure that the situational exercise developed will provide you with reliable and valid assessment information. Step #1: Identify relevant job situations. Step #2: Developing the content of the simulation. Step #3: Assess candidates' performance. The ability to provide professional advice in a context of consultation services will be used to illustrate these 3 basic steps.

Step #1: Identify relevant job situations

Get the information you will need to develop the simulation exercise.
As a first step in the development of simulations, it is best to start by identifying job situations that are relevant to the qualification(s) that are to be assessed. From this information, the content of the simulation exercise will be developed.

From qualifications to relevant job situations

For each qualification to be assessed, ask yourself: 1. How do incumbents demonstrate that they possess some degree of this qualification? What do incumbents who are strong on this qualification do? What do incumbents who

are weak on this qualification do? Are there different aspects or components to effective performance? 2. What kind of problems elicit the required qualifications? In what job situations is the qualification demonstrated? What aspects of the situation are essential, crucial, or important? What is the difference between easy situations and difficult ones? 3. What is the impact of the different actions that could be taken in the various situations? What is the difference between effective and ineffective actions?

An example
Let's consider the ability to provide professional advice in a context of consultation services. In asking the questions above, the following information could be elicited: Consultants who are good in providing professional advice in this position a. ask relevant questions, b. listen well to their clients' concerns and c. are persuasive in presenting their ideas. An important situation is the face-to-face meeting with the clients; and that these meetings are not usually one-on-one but with groups of clients. Client satisfaction is the most important result of effective consultation and professional advice. Clients feel that their concerns were understood and taken into account in the proposed solution; they find it easy to support the proposed solution.

The above information provides relevant job situations which may be used to develop the content of a situational exercise.

Step #2: Developing the content of the simulation

At this step, the context of the simulation is identified and the items to be included are chosen or developed. As well, decisions are made about how the simulation exercise is to be presented to the candidates, and what the candidates will be required to do.

How similar to the actual job situations must the items and context of the simulation be?
The items and the context of the simulation can be taken directly from the job without any modification. Alternatively, scenarios and contents can appear quite different on the surface from those found on the job but still tap the same qualification(s). The essential requirement with respect to the simulation exercise is that it provides an opportunity to observe the required qualification(s) in action. This requirement is more important than the similarity of the assessment material to the actual job content. The following points may be useful in deciding how similar the scenarios and the content should be to the actual situations. 1. When you are interested in generic abilities, the simulation content should not be too similar to that found on the job. Consider, for example, the assessment of problemsolving ability. If the problems presented to the candidates are too similar to that found on the job, it would not be clear whether a successful approach would be attributable to problem-solving ability or familiarity with how such problems have been dealt with in the past. As a general rule, the more generic the qualification the less similar the scenario and content needs to be to that found on the job.

"Double jeopardy": A problem to avoid

Double jeopardy refers to the situation where the demonstration of a qualification you wish to assess depends on another qualification. In such a situation, it is impossible to assess one qualification independently from the other qualification. For instance, when a situational exercise relies heavily on technical knowledge, performance on other qualifications (e.g., abilities) which are not necessarily related to the technical knowledge may be adversely affected. One solution is to assess knowledge qualifications using another assessment method and to develop the simulation in such a way as to minimize the influence of technical knowledge (e.g., providing candidates with background information containing the knowledge required for handling the simulation exercise; placing candidates in a neutral scenario which does not require technical knowledge; etc.). 2. When some candidates may be familiar with duties and responsibilities of the position or with current management thinking on a problem, the content of the simulation exercise should not be too similar to that found on the job. Otherwise, this kind of "insider information" might give an unfair advantage to some candidates. Whether or not the items and the context of the simulation are similar to that found on the job, they should be of the same degree of complexity or difficulty as those found on the job.

In what types of formats can the items of the simulation be presented?

The formats which can be used for the purposes of situational exercises are virtually endless; but usually they fall into one of the three following types: Written: Letters, memoranda, reports, program proposals, calendars of events, fiscal forecasts, variance reports, briefing notes, workplans, organizational charts, etc. Apparatus: computer terminals, engines, tools, etc. Audio-visual: Videotaped discussions, films, etc. Each type of format has its own advantages. Written materials, for instance, are quite versatile, easy to use, and easily reproducible. Other types of formats are less versatile but more appropriate than written materials for certain qualifications. When the simulation involves interacting with people, actors can be used to elicit the desired skill (e.g. interviewing, negotiating, selling, etc.) to ensure all candidates are presented with equivalent challenges.

Standardize your exercise

All candidates should be provided with the same initial information, instructions and timeframes. When exercises are "interactive," as in the assessment of interviewing or negotiating skills, it is appropriate to react in a natural way to the behaviour of the candidate. In these circumstances, a "flexible script" may be of use. For example, if the candidates are expected to seek out information by asking questions, then the information given to candidates would depend on the questions asked. Candidates who ask the right kind of questions will get more information. However, for the same question, all candidates would get the same answer. The answers that would be given to the different questions constitute a "flexible script." Some additional suggestions are given below: 1. Write down your instructions. This makes it easier to verify that your instructions are clear and that nothing was forgotten. It also ensures that all candidates are given the same instructions. Ensure that candidates know all they need to know about their task: how much time they have; what they must produce, if it is a written text - how many pages; if it is a presentation - to whom and for how long; will they be interacting with the board members? with other candidates?; etc. 2. Some documentation may be given ahead of time. For more involved simulations, candidates may be given a package of information to study beforehand. For instance, such a package could be mailed to them a week before they are called in for the exercise.

3. Set appropriate time limits. Unless the speed of work is the aspect being assessed, 4.
time limits should allow enough time for most candidates to complete the task. Inform the candidates about the qualifications which will be assessed and the criteria which will be used to evaluate their performance (e.g., speed, accuracy, quality, precision, number of units, style, or a combination of the above).

The information provided should be useful to the candidate without compromising the assessment tool.

Returning to our example

To assess the ability to provide professional advice in a context of consultation services , candidates could be provided with background documentation on a policy issue. The problem need not involve specialized or technical knowledge. Candidates could be given the simulation background documentation one week ahead of time to review and prepare for the next step. The next step could involve two 1-hour meetings. In the first meeting, scheduled at 9:00 am, the candidate could meet the board to ask questions and gather further information related to the policy issue. In the second meeting, scheduled at 1:00 pm, the candidate could present to the board his or her proposed solution and attempt to get an agreement on a plan of action.

Step #3: Assess candidates' performance

Define what you're looking for ahead of time.
In looking at the job situations relevant to the qualification(s) being assessed (step 1), you have already identified the impact of the different actions and the difference between effective and ineffective actions. This information will be useful in assessing the performance of candidates. It is often the case that qualifications are broken down into assessment criteria. For example, on the basis of our analysis of the position, it may be decided to assess: a. the quality of the questions asked, b. the extent to which the candidates listen to the client's concerns, and c. their persuasiveness in presenting their ideas.

Rating scales
Rating scales are useful in assessing the candidate's performance in situational exercises. Whether a numerical or narrative rating scheme is used, it is recommended that each rating be described as clearly as possible, and in advance of the actual assessment. One useful method is to list behaviours that would be expected to earn a given rating. These behaviours are sometimes called "benchmarks.". Candidates should always be assessed against these established benchmarks rather than against each other. These benchmarks are not necessarily specific, as often there is no single best course of action. In such cases, a list of the characteristics of an effective action could be developed. For example, consider the assessment of the quality of the questions asked by a candidate. The benchmark description for a rating of "excellent," or "5 out of 5", would be "all of the important questions asked, relatively few irrelevant questions asked, questions are clear and to the point."

Multiple assessors
The assessment of candidate performance in situational exercises usually involves integrative judgements. It is precisely in these circumstances that the combined judgement of multiple assessors is best. The ratings of the assessors should be integrated by consensus. Each assessor should rate each candidate's performance independently. The ratings of the different

assessors are then compared. Discrepancies are discussed and the reasons for the divergent ratings are given. Afterwards, a second set of ratings are made, and consensus is attempted again. The process is repeated until consensus is reached. As presented in this guide, simulations are useful assessment exercises. They are valid and reliable assessment tools. The advantage of such assessment is that it is easily developed by managers and staffing officers. Although it is more time consuming than some traditional approaches to assessment, it provides the selection board with an opportunity to observe the candidate perform the qualifications required for the position being staffed in a very realistic manner. For guidance and advice on developing situational exercises, please contact the Consultation Services: Phone No.: Fax No.: e-Mail: (613) 992-9741 (613) 995-0748 Personnel Psychology Centre

Assessing Leadership and Management Talent

Selecting Leaders and Managers
Selecting high calibre leaders and managers is a critical challenge. As the pace of change accelerates, the success of your organization depends on superior leadership and sound management. Rapidly evolving technology is changing the way we do business. Employees are seeking to upgrade or change their areas of expertise, the labour market is becoming increasingly competitive, they are looking for leadership and for increased participation in decision making. In this dynamic environment, excellence in leadership and management is crucial. Identifying the talented individuals who can successfully lead and manage may first appear daunting. A proven selection strategy and varied assessment instruments are, however, available to assist you in selecting high calibre leaders and managers.

The Selection Strategy

Filling a management position involves four basic steps: Step #1: Understand the position being staffed. Step #2: Specify the qualifications required for the position. Step #3: Use appropriate methods to measure the qualifications. Step #4: Collect and integrate the information to identify the best qualified candidate.

Assessment Techniques for Identifying Leadership and Management Talent

A variety of assessment techniques can be used to measure qualifications for management positions. For example, you may use reference checks to focus on candidates' past experience and accomplishments. As well, you may use work simulations which require candidates to directly demonstrate their qualifications at the time of assessment. Regardless of which method you select, it is essential that the information provided by the assessment instrument be relevant to the qualifications you are assessing.

You may develop your own assessment instruments when staffing management positions, or you may use the ready-made assessment instruments available from the Public Service Commission or from private companies. Ready-made management assessment instruments are typically designed to evaluate generic qualifications such as planning, direction/delegation, or judgement. If you wish to assess qualifications in the context of the specific position you are staffing, such as "planning the delivery of client employment services", you will need to develop your own assessment instrument.

Developing your own management assessment instruments

When developing your own assessment instruments, you can refer to Booklet # 2 in the Assessing for Competence series which describes how to develop simulations. A list of booklets currently available in the Assessing for Competence series is found on this site under "Assessing for Competence Series". If you require further advice and assistance, consultants at the Personnel Psychology Centre (PPC) are also available to assist you in tailor-making your own assessment instruments. For example, PPC has collaborated with departments to develop simulations for such positions as National Training Coordinator and Director, Monitoring and Review.

Management Assessment Instruments Available from the Public Service Commission

Should you wish to use ready-made assessment instruments available from the Public Service Commission, you can choose from a range of instruments including simulations, the assessment centre approach and paper-and-pencil instruments.

Simulation Exercises
In an ideal world, you would assess candidates by giving them a chance to perform the management job you are staffing. However, this is simply not feasible. One excellent solution is to place candidates in a situation which duplicates the complexity, the ambiguity and the varied demands of a manager's position. This is the approach taken in management simulations. When placed in the simulation, candidates are required to deal directly with the issues and problems faced by managers. For example, management simulations require candidates to quickly assimilate a great deal of information, to make decisions and set priorities in the face of conflicting policies and incomplete information, to determine how to get the job done when resources are cut, and to deal with employee problems and client complaints. As they grapple with these issues and problems, candidates produce a wealth of information about themselves and their characteristic ways of behaving. For example, a person who is skilled in planning will demonstrate strong planning skills in the simulation. A person who has difficulty delegating will also have difficulty delegating in the simulation. Two types of management simulations are available from the Personnel Psychology Centre of the Public Service Commission: In-Basket Exercises and Interactive Simulations. Each type simulates a manager's job from a different perspective.

In-Basket Exercises
In-basket exercises assess a candidate's ability to perform a management job from an administrative perspective. In the exercise, the candidate is confronted with the issues, problems and complexity of managerial life in the form of documents such as memos from superiors, peers and subordinates; reports of various kinds; letters from stakeholders; and messages and other correspondence that have accumulated in the manager's in-basket. The candidate's task is to

take action on these varied issues and problems. These actions are then evaluated in terms of qualifications such as planning, direction/delegation and integration. Separate in-basket exercises are available for supervisory, middle management and senior management positions.

Interactive Simulations
As with in-basket exercises, interactive simulations place candidates in situations where they confront the demands of a manager's job. Rather than dealing with the varied issues and problems in writing, candidates make a presentation to the selection board describing their view of the organization and its direction; their strategies for implementing change; and their plans and decisions for solving problems and implementing improvements. The selection board can probe the candidates' reasons for their actions, challenge the candidates' decisions and present the candidates with new information or assumptions about the scenario. This dynamic face-to-face aspect allows the selection board to assess various qualifications such as self-confidence, initiative, judgement, interpersonal relations and communication. Interactive simulations are available for supervisory, middle management and senior management positions. In addition, simulations specific to Audit Manager positions and Administrative Service Manager positions are available. More detailed descriptions of the in-basket and simulation exercises are available from this Internet site. To use any of the above-mentioned tools, simply contact an assessment consultant at PPC. The consultant will answer any questions you may have, provide you with relevant documentation, and explain administrative arrangements such as scoring and the costs involved, if any.

The Assessment Centre Approach

The assessment centre approach is a unique and powerful method of identifying and developing management talent through the use of varied techniques such as individual and group simulation exercises and paper-and-pencil tests. Trained assessors observe and evaluate the candidates' performance during the various exercises. Results of the assessment centre provide an integrated picture of the candidates' strengths and weaknesses. These results can be useful in identifying those with the potential to be leaders and managers, in planning the training and development of such people and in selecting the best person for a given management position.

Benefits of Management Simulations and the Assessment Centre

Hiring managers have found simulation exercises and the assessment centre approach to be extremely useful. The reasons for this are simple. These approaches place candidates in management positions, thus allowing hiring managers to assess what candidates actually do in such positions and not simply what they say they would do. These approaches also allow hiring managers to clearly differentiate among candidates and identify the best person for the management job. As well, these assessment approaches are fair in placing all candidates in the same situation. They are also seen to be fair and are typically well received by candidates. Empirical evidence strongly supports the use of these approaches to identifying leadership and management talent.

For More Information

If you would like further information on the assessment of leadership and management talent, contact the personnel officer in your department. Assessment consultants at the PPC are also available to assist you. They can be reached by calling (613) 992-9741.