HowtoDetermineAppropriate Milestones:EducationalPerspective

Erica L. Mitchell, M.D. Associate Professor Surgery

Overview
AimsoftheACGMEmilestoneproject Competenciesweneedtoassess Toolscurrentlyusedtomeasurecompetence HowwecandevelopmilestonesforVascular Surgery

ACGMEMilestoneProject

Processof education

NextAccreditation Cycle(NAS)

Educational outcomes

Whatdoweneedtoassess?
Knowledge,skills&attitudes
medicalknowledge
intellectual practical

patientcare professionalism communicationandinterpersonalskills practicebasedlearningandimprovement systemsbasedpractice

EndproductofVascularSurgicalTraining
CompetentVascularSurgeon
Workingknowledgeofvasculardiseases Coordinatesthecareofpatients Performsappropriateproceduressafely Trustworthyandempathetic Leadershipanddecisionmakingability Communicationskills Maintainsscholarship Informedonevidencebasedpractices

Howcanweassessthedomainsof competence?
Theassessmentofclinicalskills/competence/performance
expert Performanceintegrated intopractice Demonstrationof learning Interpretation/ applicationofknowledge cognition Factgathering Professionalauthenticity novice Knowledge,skills&attitude
MillerGE.AcadMed1990

behavior

Howcanweassessthedomainsof competence?
Typesofassessmentcommonlyused Writtenassessments Oralexaminations Simulation Workplacebasedassessments Multisource(360degree)feedback(MSF)
RMEpstein AssessmentinMedicalEducation NewEnglJMed2007356:387396

Assessmentofcompetence
Writtenassessments
VSITE&ABSqualifyingexam
Multichoicequestions
Singlebestanswer

Knowledge&reasoningability

Medicalknowledge Intellectual Practical

Patientcare

Professionalism

Communication& interpersonal skills

Practicebased learning& improvement X

Systemsbased practice

Limitations:cannotmeasurebehavioralandnontechnicalskills

Assessmentofcompetence
Oralexaminations
Mockorals&ABScertifyingexam
Casespecific
Unstructuredfacetofacesession

Knowledge,skillsandjudgment

Medicalknowledge Intellectual Practical

Patientcare

Professionalism

Communication& interpersonal skills X

Practicebased learning& improvement X

Systemsbased practice

Limitations:cannotreliablypredictperformanceacrossrangeofcasesand cannotassesstechnicalskills

Assessmentofcompetence
Workplacebasedassessments

Miniclinicalevaluationexercise(miniCEX) Directobservationofproceduralskills(DOPS) Procedurebasedassessment(PBA) Nontechnicalskillsforsurgeons(NOTSS Multisource(360degree)feedback(MSF)

Formativeassessments Assesswhatthetraineeactuallydoesintheworkplace Feedback Limitations:supervisedsetting
NorciniJ Workplacebasedassessmentasaneducationaltool MedTeacher200729:855871

Assessmentofcompetence
Miniclinicalevaluationexercise(miniCEX)
Directobservationofatraineeinteraction withapatientintheclinicalsetting
Casespecific

Medicalknowledge Intellectual Practical

Patientcare

Professionalism

Communication& interpersonal skills X

Practicebased learning& improvement X

Systemsbased practice

Limitations:maynotreliablypredictperformanceacrossrangeofcases

Effective2012theABSisrequiring 2clinicalassessments/resident Canbeusedinanumberof differentclinicalsettings. Potentiallyusedtoevaluate residentpatientinteractions: Dialysisaccess Claudication Criticallimbischemia Carotidstenosis Aneurysmaldisease

Howcanweassessthedomainsof operativecompetence?
Typesofassessmentcommonlyused Surrogatemeasures Simulation Workplacebasedassessments
Objectivestructuredassessmentoftechnicalskills (OSATS) Procedurebasedassessment(PBA) Nontechnicalskillsforsurgeons(NOTSS)

Assessment competence Assessmentof ofoperative technicalcompetence

Surrogatemeasures Operativespeed
Correlatespoorlywithpatientoutcome

Casenumbers
lacksvalidityandreliabilityasamethodof assessingtraineessurgicalskill

Assessmentofoperativecompetence
Simulation
Assessmentoftechnicalskillinaskillslab:
Procedurespecific Proceduralskillorsequencing

Medicalknowledge Intellectual Practical

Patientcare

Professionalism

Communication& interpersonal skills X

Practicebased learning& improvement

Systemsbased practice

Limitations:needtosamplewidelyandconsiderbothcontextandconstruct, cangamethesimulator,ceilingeffect

Assessmentofoperativecompetence
Objectivestructuredassessmentoftechnicalskills (OSATS)
Checklists
Procedurespecific Captureoccurrenceofparticular behavior Perceivedtobemoreobjective Scoresslightlymorereliable Goodforassessingtrainees

Globalratingscores
Itemscommontoanyprocedure Capturethequalityofa performance Perceivedtobemoresubjective Scoresslightlymorevalid Goodforassessingexperts

Assessmentofoperativecompetence
Directobservationofproceduralskills(DOPS)
Traineesobservedperformingaprocedure onarealpatient
Procedurespecific

Medicalknowledge Intellectual Practical

Patientcare

Professionalism

Communication& interpersonal skills X

Practicebased learning& improvement X

Systemsbased practice

Limitations:maynotreliablypredictperformanceacrossrangeofcases

Effective2012theABSisrequiring 2operativeassessments/resident Potentiallybeusedtoevaluate residentsperformingroutine coresurgicalorminorvascular procedures: I&Dabscess Radialarterialline Centralvenouscatheter Temporarydialysiscatheter

Assessmentofoperativecompetence
Procedurebasedassessment(PBA)
AssessmentoftechnicalskillintheOR:
Procedurespecific OSATS

Knowledge,skills&attitudes
Medicalknowledge Intellectual Practical Patientcare Professionalism Communication& interpersonal skills X Practicebased learning& improvement X Systemsbased practice

Limitations:needtosamplewidelyandconsiderbothcontextandconstruct

Assessmentofoperativecompetence
Nontechnicalskillsforsurgeons(NOTSS)
AssessmentofnontechnicalskillintheOR:
Situationawareness Decisionmaking Communicationandteamwork Leadership
Patientcare Professionalism Communication& interpersonal skills X Practicebased learning& improvement X Systemsbased practice

Medicalknowledge Intellectual Practical

Limitations:needtosamplewidelyandconsiderbothcontextandconstruct

Assessmentofcompetence
Multisource(360)feedback(MSF)
Peerassessment(untrainedraters)
Teammembers,physicianextenders Referringservices,anesthesiastaff Clinicstaff,wardclerks Self
Patientcare Professionalism Communication& interpersonal skills X Practicebased learning& improvement X Systemsbased practice

Medicalknowledge Intellectual Practical

Limitations:needmultipleassessments

Makingjudgmentsaboutthe performanceofpeershasformedthe basisofthereferralprocessinmedicine andtheotherprofessionsforcenturies.

NorciniJJ Peerassessmentofcompetence MedEducation200337:539543

Developingmilestonesforvascularsurgery
Systematicapproach
Definemarkersofachievementacrossthe continuumofvascularsurgicaleducation
Blueprintthesetotheentirecurriculum
Knowledge,skills&attitudes 6corecompetencies Coresurgical&specialtyspecificskills

Consensus:Delphiprocessusingspecialtyexperts

Whatweneedtodoasaspecialty
Systematicapproach
Identifyorcreatetoolsthatcanbeembracedas meaningfulmeasuresofperformance
Acrossthecontinuumofvascularsurgicaltraining Assessacrossmultiplecontexts:
Clinical,simulatedandORsetting Clinical&operativecompetencies

Assessmentcontent:taskanalysisandDelphi processesusingspecialtyexperts

Whatweneedtodoasaspecialty
Systematicapproach
Needtoensureassessmentsare:
Reliable Valid Acceptable Costeffective Educationalimpact

VanderVleutensutilityindex
Utility=Validityw xreliabilityw xacceptabilityw xeducationalimpactw xcostw
W=weighted

Evidencedocumentingtraineeslearning
Learningportfolio
Assessment Standardizedtesting
VSITE,ABSQE

Competencyassessed medicalknowledge
intellectual practical

Operativeandcaselogdata Assessmentofpatientcare
Clinicalsetting(miniCEX) ClinicorOR(DOPS) Operativesetting(PBA)

360 evaluations(MSF) Compliancewithquality performanceindicators

patientcare professionalism communicationand interpersonalskills practicebasedlearningand improvement systemsbasedpractice

EndproductofVascularSurgicalTraining
CompetentVascularSurgeon
Workingknowledgeofvasculardiseases Coordinatesthecareofpatients Performsappropriateproceduressafely Trustworthyandempathetic Leadershipanddecisionmakingability Communicationskills Maintainsscholarship Informedonevidencebasedpractices