Supplementary Appendix

This appendix has been provided by the authors to give readers additional information about their work.
Supplement to: Urbach DR, Govindarajan A, Saskin R, Wilton AS, Baxter NN. Introduction of surgical safety
checklists in Ontario, Canada. N Engl J Med 2014;370:1029-38. DOI: 10.1056/NEJMsa1308261

Introduction of surgical safety checklists in Ontario, Canada

David R Urbach MD MSc, Anand Govindarajan MD MSc, Refik Saskin MSc, Andrew S Wilton
MSc, Nancy N Baxter MD PhD

Contents:
Methods ........................................................................................................................................................ 2
Results ........................................................................................................................................................... 6
Figure S1A. Length of hospital stay (days) .................................................................................................. 8
Figure S1B. Readmission to hospital within 30 days of discharge ............................................................... 9
Figure S1C. Visit to an emergency department within 30 days of discharge ............................................. 10
Figure S2A. Length of hospital stay ........................................................................................................... 11
Figure S2B. Readmission to hospital within 30 days of discharge ............................................................. 12
Table S1. Adjusted rate ratios of surgical outcomes associated with checklist use according to the type of
checklist and date of adoption..................................................................................................................... 13
Table S2. Complete characteristics of the study subjects ........................................................................... 14
Table S3. Adjusted rate ratios of surgical outcomes ................................................................................... 16
References ................................................................................................................................................... 18
Canadian Patient Safety Institute Surgical Safety Checklist....................................................................... 19

Methods

Sources of Data
We used the Canadian Institute of Health Information (CIHI) database of ambulatory and
inpatient hospitalizations, the Ontario Health Insurance Plan (OHIP) database of physician
claims, and the Registered Persons Data Base (RPDB). These data bases include information on
almost all residents of Ontario, and are considered population-based. These data sets were held
securely in a linked, de-identified form and analyzed at the Institute for Clinical Evaluative
Sciences.

Study Periods
To allow for uncertainty regarding the precise day that a checklist was introduced, and to
account for a training period when a checklist was being adopted and deployed, we selected a
washout period of three months before and three months after the date a checklist was
introduced, representing a time window of six months. For example, if a hospital started using a
checklist on July 1, 2010, we considered all procedures performed between January 1 and March
31, 2010 to be before and all procedures between October 1 and December 31, 2010 to be
after introduction of a checklist.

Checklist Compliance
Surgical Safety Checklist Compliance is one of ten patient safety quality indicators
reported at the hospital level to the Ontario Ministry of Health and Long-term Care by provincial
mandate. The quality indicator measures (as a percentage) the number of times the three phase
surgical safety checklist was used during a surgical procedure (numerator) over the total number
2

of surgeries performed (denominator) at the institution. A designated checklist coordinator, often

the circulating nurse in the room, determines if the checklist was completed for each surgical
procedure performed.1
Compliance with surgical safety checklists is reported publicly by the Ontario Ministry of
Health and Long-term Care at the level of the individual hospital, aggregated by 6-month
intervals.2 Available data did not permit us to determine whether a checklist was used in a
particular procedure, and we were unable to measure compliance with checklists at monthly
intervals in our analysis. However, reported compliance with checklists is extraordinarily high,
and we were unable to identify subgroups of hospitals that reported compliance less than 90%.
For example, among 97 large community hospitals required to report compliance with surgical
safety checklists in JanuaryJune 2013, almost all hospitals reported compliance of 99% or
100%. The lowest reported compliance in this period was 91.6%.

Surgical Procedures
We manually reviewed all codes with a prefix 1 (Physical/Physiological Therapeutic
Interventions) to select surgical procedures. We excluded interventions such as insertion of
central venous catheter, insertion of gastrointestinal feeding tubes, obstetrical interventions, and
organ procurement for transplantation.
We included Canadian Classification of Interventions (CCI) codes in the following
categories (listed with most frequent surgical procedure): Nervous System (craniotomy with
partial excision of brain), Eye (excision of lens using phacoemulsification with insertion of
prosthesis), Ear (repair of tympanic membrane), Orocraniofacial (tonsillectomy with
adenoidectomy), Respiratory (video-assisted partial lung lobectomy), Cardiovascular (coronary

artery bypass graft), Lymphatic (excision of neck lymph node), Digestive (laparoscopic
cholecystectomy), Genitourinary (dilatation and curettage), Musculoskeletal (hernia repair), Skin
& Breast (partial mastectomy), and Other (carpal tunnel release). All appropriate CCI codes in
each category were analyzed, not just the most common procedure. Cardiovascular procedures
include peripheral vascular procedures.

Complications
Complications included one or more occurrences of acute renal failure, bleeding, cardiac
arrest, coma, deep venous thrombosis, myocardial infarction, ventilator use, pneumonia,
pulmonary embolism, stroke, major disruption of wound, infection of surgical site, sepsis, septic
shock, unplanned return to the operating room, and vascular graft failure.

Covariates
The Resource Utilization Band (RUB) simplified morbidity categories of the Adjusted
Clinical Group (ACG) system is a six-level categorization of overall morbidity burden as defined
by health resource use (0Non-users, 1 Healthy Users, 2Low Morbidity, 3Moderate, 4
High, 5Very High).
The quintile of median household income (an ecologic measure of socioeconomic status)
was measured at the level of the neighborhood, as defined by postal codes. The neighborhood of
residence is defined by the Forward Sortation Area (the first 3 characters of the 6-character
postal code).

Seasonality

We tested for the effects of seasonal variation by modelling the effect of quarter
(January-March, April-June, July-September, October-December) as a categorical variable on
the outcomes of interest. The effect of season (annual quarter) on the outcomes was not
significant. Subsequent analyses were adjusted for the month of the procedure, to account for
temporal trends.

Sensitivity analyses
We conducted sensitivity analyses to test whether the results changed during different
study periods, first comparing the same three-month post-implementation interval with a threemonth pre-implementation interval exactly one year before, and next comparing pre- and postimplementation intervals separated by only 1 month after checklist introduction.

Results

Effect of introducing a checklist in individual hospitals on various outcomes

Emergency department visits within 30 days of discharge increased statistically in 6
hospitals and declined in 5, length of hospital stay increased in 5 hospitals and declined in 4, and
readmission within 30 days of discharge increased in 3 hospitals (Figure S1AC).

Stratified analyses
There were different effects of the checklist on length of hospital stay between elective
and emergency procedures, and among some of the procedure types (Figure S2A). There were no
differences in the effect of introducing a surgical checklist among subgroups with respect to the
risk of readmission to hospital within 30 days of discharge (Figure S2B).

Multivariable models
Implementation of a checklist was not associated with statistical improvements in
operative mortality (odds ratio [OR] 0.91; 95% CI: 0.80, 1.03, P=0.13) or surgical complications
(OR 0.97; 95% CI: 0.90, 1.03, P=0.29, Table S3). There was a statistically significant
improvement in length of stay (rate ratio [RR] 0.97; 95% CI: 0.95, 0.99, P=0.007). Some
covariates included in the model to adjust for potential confounding were also associated with
the outcomes. Compared with elective surgery, emergency surgery was associated with greater
risk of mortality (OR 6.96; 95% CI: 6.07, 7.98, P<0.001), surgical complications (OR 1.69; 95%
CI: 1.46, 1.96, P<0.001), and length of stay (RR 1.82; 95% CI: 1.74, 1.90, P<0.001). Inpatient
procedures were significantly more likely to result in operative mortality (OR 5.75; 95% CI:
6

4.11, 8.03, P<0.001) and complications (OR 64.43; 95% CI: 29.46, 140.92, P<0.001) than
ambulatory procedures. Compared to those with little comorbidity, persons with the greatest
burden of comorbidity were more likely to experience death (OR 3.90; 95% CI: 2.55, 5.97,
P<0.001), complications (OR 7.11; 95% CI: 5.39, 9.68, P<0.001) and longer hospital stay (RR
2.53; 95% CI: 2.32, 2.77, P<0.001).

Sensitivity Analyses
Sensitivity analyses testing different time intervals yielded results that were substantially
similar to the primary analyses. For example, the adjusted risk of death in hospital or within 30
days was 0.74% (95% CI, 0.69 to 0.79) before and 0.65% (95% CI, 0.61 to 0.70) after
implementation of a surgical checklist when comparing three-month intervals separated by 1
year, and was 0.65% (95% CI, 0.61 to 0.70) before and 0.66% (95% CI, 0.62 to 0.71) when
comparing intervals separated by only one month.

Figure S1A. Length of hospital stay (days)

15.0

10.0

5.0

0.0

-5.0

-10.0

-15.0

Figure S1A . Within-hospital change in length of hospital stay (days) among 101 Ontario
hospitals following introduction of a surgical safety checklist. Each point represents the
difference in outcome in one hospital, adjusted for age, sex and the month of surgery. Error bars
indicate 95 percent confidence intervals. Values less than 0 indicate improvement. Hospitals are
ordered from highest (least improvement) to lowest (most improvement) values.

Figure S1B. Readmission to hospital within 30 days of discharge

15%

10%

5%

0%

-5%

-10%

-15%

Figure S1B . Within-hospital change in the percentage of patients readmitted to hospital within
30 days of discharge among 101 Ontario hospitals following introduction of a surgical safety
checklist. Each point represents the difference in outcome in one hospital, adjusted for age, sex
and the month of surgery. Error bars indicate 95 percent confidence intervals. Values less than 0
indicate improvement. Hospitals are ordered from highest (least improvement) to lowest (most
improvement) values.

Figure S1C. Visit to an emergency department within 30 days of discharge

30%

20%

10%

0%

-10%

-20%

-30%

Figure S1C . Within-hospital change in the percentage of patients with a visit to an emergency
department within 30 days of discharge among 101 Ontario hospitals following introduction of a
surgical safety checklist. Each point represents the difference in outcome in one hospital,
adjusted for age, sex and the month of surgery. Error bars indicate 95 percent confidence
intervals. Values less than 0 indicate improvement. Hospitals are ordered from highest (least
improvement) to lowest (most improvement) values.

10

Figure S2A. Length of hospital stay

Figure S2A. Rate ratios and 95 percent confidence intervals for length of hospital stay associated
with checklist use stratified by age, sex, acuity, admission category, and procedure. Adjusted
effect sizes for each strata were estimated using generalized linear models, adjusting for all other
variables shown except the stratification variable. Models estimating length of hospital stay were
limited to inpatient procedures only. P values shown are for the interaction between the
stratification variable and checklist use on the outcome.

11

Figure S2B. Readmission to hospital within 30 days of discharge

Figure S2B. Odds ratios and 95 percent confidence intervals for readmission to hospital within
30 days of discharge associated with checklist use stratified by age, sex, acuity, admission
category, and procedure. Adjusted effect sizes for each strata were estimated using generalized
linear models, adjusting for all other variables shown except the stratification variable. P values
shown are for the interaction between the stratification variable and checklist use on the
outcome.

12

Table S1. Adjusted rate ratios of surgical outcomes associated with checklist use according to the type of checklist and date of
adoption
Operative mortality
Surgical complications
Length of stay
OR
95% CI
P value
OR
95% CI
P value
RR
95% CI
P value
Type of checklist
CPSI
Own
Unknown
WHO

0.32
0.90
0.79
1.06
1.83

0.79-1.02
0.60-1.04
0.78-1.42
0.60-5.59

0.96
0.98
1.00
0.96
0.90

0.93-1.04
0.89-1.12
0.83-1.11
0.55-1.49

0.001
0.99
1.00
1.02
1.02

0.98-1.00
0.99-1.02
1.00-1.03
0.97-1.06

Date of adoption
0.81
0.91
0.01
Before April 2010
0.91
0.79-1.04
0.98
0.92-1.04
1.00
0.99-1.01
April 2010
0.85
0.69-1.04
1.00
0.91-1.10
0.99
0.98-1.00
After April 2010
0.96
0.71-1.31
0.96
0.83-1.11
0.98
0.96-0.99
OR denotes odds ratio, RR denotes rate ratio, CI denotes confidence interval, CPSI denotes Canadian Patient Safety Institute, WHO
denotes World Health Organization
Adjusted using generalized estimating equations for age, sex, procedure status, admission category, procedure type, resource
utilization band (a six-level simplified morbidity categorization of the Adjusted Clinical Groups [ACG] system as defined by health
resource use), month of surgery, rural or urban residence, and neighborhood income quintile
P value is for the interaction between the variable (type of checklist and date of adoption) and checklist use on the outcome

13

Table S2. Complete characteristics of the study subjects

Before introduction of After introduction
checklist
of checklist
Number
Percent Number
Percent
109,341
106,370
Overall
Procedure status

Elective
Emergency

97,040
12,301

88.8
11.2

93,699
12,671

88.1
11.9

Admission
category

Ambulatory
Inpatient

66,660
42,681

61.0
39.0

64,718
41,652

60.8
39.2

Procedure type*

Nervous System
Eye
Ear
Orocraniofacial
Respiratory
Cardiovascular
Lymphatic
Digestive
Genitourinary
Musculoskeletal
Skin & Breast
Other

879
21,578
1,134
9,663
1,007
4,042
549
12,867
17,785
31,381
6,212
2,244

0.8
19.7
1.0
8.8
0.9
3.7
0.5
11.8
16.3
28.7
5.7
2.1

901
21,471
1,062
9,582
1,003
3,938
498
13,206
16,340
30,554
5,807
2,008

0.9
20.2
1.0
9.0
0.9
3.7
0.5
12.4
15.4
28.7
5.5
1.9

Age

0-17
18-39
40-64
65+

7,689
18,955
43,669
39,028

7.0
17.3
39.9
35.7

7,806
18,232
42,023
38,309

7.3
17.1
39.5
36.0

Sex

Female
Male

63,591
45,750

58.2
41.8

61,672
44,698

58.0
42.0

Comorbidity
score

0-2
3
4
5

5544
51,935
32,325
19,537

5.1
47.5
29.6
17.9

5450
49,856
31,457
19,607

5.1
46.9
29.6
18.4

Neighborhood
income quintile

unknown
1 (lowest)
2
3
4

406
19,574
21,223
22,078
23,392

0.4
17.9
19.4
20.2
21.4

414
19,098
20,684
21,216
22,698

0.4
18.0
19.5
20.0
21.3

14

5 (highest)

22,668

20.7

22,260

20.9

Rurality

Urban
Rural
Unknown

94,016
1,523
95

86.0
13.9
0.1

91,246
15,015
109

85.8
14.1
0.1

Month of
procedure

January
February
March
April
May
June
July
August
September
October
November
December

7,725
4,900
4,144
5,506
2,634
4,808
4,933
6,593
18,121
18,802
21,487
9,688

7.1
4.5
3.8
5.0
2.4
4.4
4.5
6.0
16.6
17.2
19.7
8.9

4,831
3,696
5,808
2,469
5,279
6,027
7,034
15,008
19,061
20,090
11,058
6,009

4.5
3.5
5.5
2.3
5.0
5.7
6.6
14.1
17.9
18.9
10.4
5.7

Hospital type

Community
77,026
70.5
74,817
70.4
Pediatric
1,808
1.7
1,827
1.7
Small
1,713
1.6
1,690
1.6
Teaching
28,794
26.3
28,002
26.3
Percentages may not sum to 100.0 due to rounding
Each study period was three months long, extending from six months to three months prior to
introduction, and from three months to six months after
*Categorized according to Canadian Classification of Interventions (CCI) categories
Comorbidity was measured using resource utilization bands, a six-level simplified morbidity
categorization of the Adjusted Clinical Groups (ACG) system as defined by health resource use,
with 0 indicating non users of health resources and 5 indicating very high morbidity
Median household income in the neighborhood of residence

15

Table S3. Adjusted rate ratios of surgical outcomes

Operative mortality
OR
95% CI
P value
Checklist
Before
1.0

After
0.91
0.80-1.03
0.13
Age (yr)
0-17
1.0

18-39
1.20
0.66-2.18
0.56
40-64
3.57
2.00-6.36 <0.001
65+
13.86
7.76<0.001
24.76
Sex
Female
1.0

Male
1.24
1.11-1.40 <0.001
Procedure
status
Elective
1.0

Emergency
6.96
6.07-7.98 <0.001
Admission
category
Ambulatory
1.0

Inpatient
5.75
4.11-8.03 <0.001
Procedure
type
Nervous
1.0

System
Eye
0.53
0.34-0.82
0.005
Ear
0.77
0.27-2.14
0.61
Orocraniofacial
0.39
0.22-0.69
0.001
Respiratory
0.95
0.61-1.49
0.84
Cardiovascular
0.81
0.60-1.10
0.18
Lymphatic
0.94
0.48-1.83
0.85

Surgical complications
OR
95% CI
P value

RR

1.0
0.97

0.90-1.03

0.29

1.0
0.97

0.95-0.99

0.007

1.0
1.24
2.71
4.22

0.69-2.22
1.49-4.93
2.27-7.86

0.47
0.001
<0.001

1.0
1.26
1.70
2.23

1.07-1.47
1.43-2.03
1.84-2.69

0.005
<0.001
<0.001

1.0
1.14

1.08-1.21

<0.001

1.0
0.97

0.94-0.99

0.002

1.0
1.69

1.46-1.96

<0.001

1.0
1.82

1.74-1.90

<0.001

1.0
64.43

29.46-140.92

<0.001

1.0

1.0

0.04
0.27
0.51
0.83
1.69
0.93

0.002-0.53
0.08-0.85
0.35-0.75
0.61-1.12
1.36-2.10
0.63-1.37

0.40
0.35
0.55
1.11
1.25
0.87

0.28-0.57
0.22-0.56
0.44-0.69
0.92-1.35
1.09-1.44
0.72-1.05

16

0.02
0.03
<0.001
0.23
<0.001
0.72

Length of stay
95% CI
P value

<0.001
<0.001
<0.001
0.28
0.002
0.14

Digestive
Genitourinary
Musculoskeletal
Skin & Breast
Other

0.74
0.33
0.45
0.39
0.41

0.53-1.05
0.22-0.49
0.33-0.61
0.21-0.71
0.15-1.11

0-2
3
4
5

1.0
0.83
1.73
3.90

0.53-1.30
1.15-2.62
2.55-5.97

0.09
<0.001
<0.001
0.002
0.08

5.09
0.52
0.44
0.51
0.25

4.21-6.15
0.40-0.68
0.36-0.52
0.38-0.68
0.07-0.83

0.41
0.009
<0.001

1.0
2.16
3.73
7.22

1.68-2.79
2.80-4.98
5.39-9.68

<0.001
<0.001
<0.001
<0.001
0.02

0.96
0.65
0.85
0.35
0.59

0.82-1.12
0.56-0.76
0.74-0.99
0.29-0.43
0.42-0.83

0.58
<0.001
0.03
<0.001
0.002

<0.001
<0.001
<0.001

1.0
1.43
1.80
2.53

1.32-1.54
1.66-1.95
2.32-2.77

<0.001
<0.001
<0.001

Comorbidity
score*

OR denotes odds ratio, RR denotes rate ratio, CI denotes confidence interval

Adjusted using generalized estimating equations for all variables listed, as well as month of surgery, rurality, and neighborhood
income quintile
P value is for the comparison with the referent category
*Comorbidity score was measured using resource utilization bands, a six-level simplified morbidity categorization of the Adjusted
Clinical Groups (ACG) system as defined by health resource use, with 0 indicating non users of health resources and 5 indicating very
high morbidity

17

References
1.
Ontario Hospital Association. Surgical Safety Checklist. (Accessed September 29, 2013, at
http://www.oha.com/CurrentIssues/keyinitiatives/PatientSafety/Pages/SurgicalSafetyChecklist.aspx.)
2.
Health Quality Ontario. Public Reporting. Patient Safety. Percent of surgeries in which a surgical
safety checklist was performed. April 2010-June 2013. (Accessed September 29, 2013, at
http://www.hqontario.ca/public-reporting/patient-safety.)

18

Canadian Patient Safety Institute Surgical Safety Checklist

19