Designing and Implementing Course Evaluations Survival Training for Subject Matter Experts

` `

The importance of course evaluations Linking course evaluations to the learning and performance objectives The most appropriate level of evaluation based on time, resources and how the data will be utilized.

` Why do an evaluation?

x Will the class be taught again? x Will this class be taught by others? x Will more classes be taught on this subject? Who will see and use the information? (Who cares?) Who is going to compile the data and how will it be used?

Level 4: Results What is the impact on the organization? Level 3: Behavior Are the participants using the Information taught in the class? Level 2: Learning Can the participants remember what was taught in the training? Level 1: Reaction Did the participants like the training?

Each successive level represents a more precise measure of the effectiveness of the training program, but at the same time requires a more rigorous and timeconsuming analysis.

` ` ` ` ` `

How well was the training perceived Gives immediate feedback Easy and economical Administered at the end of the class Sometimes known as smile sheets Paper or electronic

If they dont like it they probably are not learning

A.D.D.I.E. Analysis Design Develop Implement Evaluate

Learning needs and gaps Course outline Course material Class or CBT made available to learners Feedback

Analyze

Implement

Evaluate

Design

Develop

Level 1 The Questionnaire

Evaluation Section Basic course information ` Course title, instructors name, date, location ` Instructors ability to answer questions ` Depth and complexity ` Difficulty of exercises ` Instructors knowledge ` Pace of the training ` Usefulness of visual aids ` Length of training session Comments Section: Provides descriptive data such as ` Topics to spend additional time on ` Were assessed needs accomplished? ` Are the participants more confident in their skills? ` Logistics pre, during, post ` Changes for a more effective course?

Creating test items might seem easy at first, but a lot of thought must go into test creation to make it effective. Writing effective questions takes time and practice. Whether your goal is to measure knowledge and skills, survey opinions and attitudes, or enhance a learning experience, poorly worded questions can adversely affect the quality of the results. Here are a few tips to keep in mind when you write and review questions: Goal of Test Items ` Test items should: ` Measure the objectives taught at the required skill or information level (match verbs) ` Focus on important, not trivial, subject area content ` Contain accurate information (including correct spelling) ` Be written at an appropriate level of difficulty ` Be clear and concise and as short as possible ` Avoid vague pronouns (e.g. it, they) ` Avoid information that helps answer other items ` Be bias free

Responses should: ` Be plausible ` Be grammatically consistent with the stem ` Be of approximately the same length ` Be structurally parallel ` Avoid conflicting alternatives ` Provide only one correct answer ` Avoid All of the above and use of None of the above with caution. Avoid syllogistic reasoning choices (e.g. both a and b are correct) unless absolutely necessary. ` Incorrect answers called distractors should not be too obvious ` Distractors should point out errors in thinking along the levels of Bloom's Taxonomy

Stems should: ` Provide sufficient information ` Avoid extra information ` Be grammatically correct ` Avoid clues that help give away answers ` Avoid negatives, especially double negatives ` Avoid unnecessary modifiers, especially absolutes (e.g. always, never, etc.) ` Avoid second person ` Use questions whenever possible (What, Who, When, Where, Why and How)

` ` `

A test of knowledge or skill Did they retain the information? Used to assess the amount of learning that has occurred. Often uses tests conducted before training (pretest) and after training (post test). Administered at the end of the class or several days or weeks following the class

Level 2: Learning Can the participants remember what was taught in the training?

` ` ` ` ` ` `

Paper or electronic Multiple choice, short answer, true-false Simulation, demonstration or observation Case study Role plays Table top exercises Or a game!

SurveyMonkey.com
because knowledge is everything

Measures the changes that have occurred in learners' behavior due to the training program. Evaluating at this level attempts to answer the question Are the newly acquired skills, knowledge, or attitude being used in the everyday environment of the learner? May require up-front assessment for benchmark

Level 3: Behavior Are the participants using the Information taught in the class?

` ` ` ` `

Occur several weeks or months after the training Observation Survey of managers Analysis of systems May require up-front assessment for benchmark

` ` ` `

Attempts to assess training in terms of business results. Decrease in number of complaints Decrease in queue time Increase in sales Occur several weeks or months after the training Determines if the time and $$ for the training was a good investment Observation Survey of managers Analysis of systems Will require up-front assessment for benchmark

Level 4: Results What is the impact on the organization?

Heres just a partial list of several excellent resources to continue your learning journey!
Books and Magazines ` Evaluating Training Programs: The Four Levels Donald L. Kirkpatick and James D. Kirkpatrick ` Evaluating the Impace of Training Scott B. Parry ` Handbook of Training Evaluation and Measurement Methods Jack B. Phillips ` Evaluation Basics Donald McCain ` Evaluating E- Learning William Horton ` The Accidental Trainer Nanette Miller ` The Instant Trainer C. Leslie Charles and Chris Clarke-Epstein ` The Ten Minute Trainer Sharon L. Bowman ` Training for Dummies Elaine Biech ` Training SMEs to Train ASTD InfoLine

Teaching Others What You Know Survival Training for Subject Matter Experts
Next session: Classroom Delivery Skills by invitation only to those who have participated in Modules 1- 4 and successfully completed the assessments. Friday, June 20, 2008 1:00 p.m. 5:00 p.m. Fidelity Banks Training Center, I-85 and Clairmont Road, Atlanta www.astdatlanta.org

Special thanks to ExamBuilder for providing the on-line assessment.

CREATE exams in a Web browser DEPLOY your exam with one mouse click AUTOMATE registration and exam retakes ANALYZE detailed reports and statistics www.exambuilder.com

About ASTD ASTD (American Society for Training & Development) is the worlds largest association dedicated to workplace learning and performance professionals. ASTDs members come from more than 100 countries and connect locally in nearly 140 U.S. chapters and 25 Global Networks. Members work in thousands of organizations of all sizes, in government, as independent consultants, and suppliers. The Greater Atlanta Chapter of ASTD is one of the five largest chapters within the US with nearly 550 members. We are a regional association of workplace performance professionals, providing our members with a forum for the exchange of professional development opportunities and resources as they strive to improve workplace learning and performance.

www.astdatlanta.org

Special thanks to WebEx for partnering with the ASTD Atlanta Chapter to make this event possible.
About Cisco Cisco (Nasdaq: CSCO) is the worldwide leader in networking that transforms how people connect, communicate and collaborate. Information about Cisco can be found at www.cisco.com. For ongoing news, please go to newsroom.cisco.com. With 2.3 million registered users, Cisco WebEx is the global leader in on-demand applications for collaborative business on the web. WebEx offers a full suite of web collaboration applications for specific business processes including marketing, sales, training, and support. Companies of every size gain strategic advantage by using WebEx to replace high-cost/high-touch interactions with efficient web-touch interactions. WebEx applications work across firewalls and across platforms, making them ideal for both internal and external collaboration. Cisco WebEx provides its services over the WebEx MediaTone Network, a secure global network specifically designed for on-demand applications. Please call toll free 877-509-3239 or visit www.webex.com for more information WebEX team members: Brad King Nisha.Ahluwalia Lauren Follett Jennifer Smith

SillyMonkey LLC is a boutique training company specializing in innovative games and activities. We design custom board games, card games, electronic games, video games and other interactive multimedia training components for instructor-led sessions, as well as, eLearning, mobile learning, social networks and virtual worlds. We offer high quality training services to a an impressive list of global clients that includes manufacturing, health care, retail, software, pharmaceuticals, and financial services. Our team has over 25 years of experience in traditional training and various training methodologies including ADDIE, constructivist, cognitivism and behaviorism. We use mind mapping, memory techniques, improve and storytelling. www.sillymonkeyinternational.com

Brad King, WebEx Bridget Evans, Randstad Christie Cole, RBS Lynk Darcie Drymon, GH& I Latarsha Horne, Orkin Deborah Lucas, Turner Broadcasting Deborah Thomas - SillyMonkey Garland Watkins, UPS Contractor Harvey Menden, Kimberly-Clark Jeff Collins - Arby's Laurie Gilmore - Orkin

Jon Wakeman, Atlanta Journal Constitution Katherine Brune, SunTrust Bank Kim Beckett, Xinnix Marcia Breese, CNN Mary Duncan, Deloitte and Touche Matt Fazio, Consultant Patsy Newton, Cabot Corp. Stacy Woods, UPS Contractor Stephanie Britz, Carmax Tony Foley, Childrens Healthcare of Atlanta

There is a short assessment to check your knowledge. If you took the assessment for Modules One, Two or Three go to Go to: http://testing.exambuilder.com, type in your e-mail address and password: astdatlanta First Time Users: Go to: http://testing.exambuilder.com/main/exams/index.cfm?loginaction=5 Follow the directions below to register. You will receive an e-mail from ExamBuilder within one hour with log-in information. Type in: Your First Name: Your Last Name: Your Email: Instructors name: jcollins@arbys.com Instructors creator ID: DADQ Click on Register.