Вы находитесь на странице: 1из 78

U.S.

Department of Justice
Office of Justice Programs
Bureau of Justice Statistics

TECHNICAL REPORT

March 2015, NCJ 248544

Arrest-Related Deaths Program:


Data Quality Profile
Michael Planty and Andrea M. Burch, Bureau of Justice Statistics
Duren Banks, Lance Couzens, Caroline Blanton, and Devon Cribb, RTI International

Executive summary
The Arrest-Related Deaths (ARD) program is an annual
national census of persons who die either during the
process of arrest or while in the custody of state or local
law enforcement personnel. The Bureau of Justice Statistics
(BJS) implemented the ARD program in 2003 as part of the
Deaths in Custody Reporting Program (DCRP). The DCRP
was initiated to fulfill the data collection requirement of the
Deaths in Custody Reporting Act of 2000 (DICRA, P.L. 106247). It collects in-depth information on deaths during arrest
and incarceration, and it provides national-level information
on the deaths of suspects and offenders from their initial
contact with law enforcement personnel through the time
they are incarcerated in a jail or prison.
ARD data are collected to quantify and describe the
circumstances surrounding civilian deaths that take place
during an arrest or while in the custody of law enforcement.
These data describe the prevalence and incidence of arrestrelated deaths across the nation, identify the circumstances
or activities that contribute to these deaths, and reveal
trends in the causes and circumstances of these deaths in
custody at national and state levels. These data can be used
to inform specific policies that may increase the safety of
law enforcement officers and citizens, identify training
needs in law enforcement agencies, and assist in developing
prevention strategies.
The current ARD program relies on state reporting
coordinators (SRCs) in each of the 50 states and the District
of Columbia to identify and report on all eligible cases of
arrest-related deaths. BJS compiles data from the states to
produce national-level statistics on deaths that occur in the
process of arrest by, or while in the custody of, state and local
law enforcement personnel. When the DICRA reporting

requirements ended in 2006, BJS undertook efforts to


understand the variability between SRCs, and within SRCs
over time, in terms of data collection methodologies and
available resources. This variability has led to concerns about
definitions, data quality, and undercoverage error.
This report responds to issues with variability and profiles
the data quality of the ARD component of the DCRP.
The report (1) gives an overview of the ARD program,
(2) describes the data collection methods used by SRCs, and
(3) assesses the quality of the resulting data and coverage for
eligible ARD cases that involve law enforcement homicides.
The variation in methodologies resulted in a significant
underestimate of the annual number of arrest-related deaths,
including both homicides by law enforcement officers and
other types of civilian deaths. Key findings include
Nationally, the ARD program captured about 50% of the

estimated law enforcement homicides during 200309


and 2011. This assessment did not examine the coverage
of other deaths the ARD program is designed to capture,
including those due to accidents, drug overdoses, and
natural causes.

The national coverage of homicides by law enforcement

captured in the ARD program improved from 2003


through 2011. This was partially due to the use of open
source searches to identify potential cases that were
eligible for inclusion. Even with this improvement,
assessments indicate that from 31% to 41% of the
estimated homicides by law enforcement personnel were
not captured in the 2011 ARD data collection.

Significant challenges exist because of the lack of

standardized modes for data collection, definitions, scope,


participation, and the availability of resources.

1. Program overview
HIGHLIGHTS
The Arrest-Related Deaths (ARD) program collects

information on deaths occurring in the process of arrest.


An arrest-related death is defined as any death (e.g., gunshot

wound, cardiac arrest, or drowning) that occurs during an


interaction with state or local law enforcement personnel,
including those that occur
during an attempted arrest or in the process of arrest
while the person is in law enforcement custody (before

transfer to jail)
shortly after the persons freedom to leave is restricted.
Exclusions include
deaths of bystanders, hostages, and law enforcement

personnel
deaths occurring during an interaction with federal law

enforcement agents
deaths of wanted criminal suspects before police contact
deaths by vehicular pursuits without any direct

police action.

1.1 Program purpose and justification


The Deaths in Custody Reporting Act of 2000 (DICRA; P.L.
106-297), signed into law on October 13, 2000, required that
states applying for Violent Offender Incarceration/Truth-inSentencing (VOI/TIS) grants follow the U.S. Department of
Justice guidelines for a quarterly data collection of inmate
deaths from all state prisons (public and private), local jails,
juvenile detention facilities, and police lockups, as well as
deaths occurring in transit and in the process of arrest. Under
DICRA, state applicants for VOI/TIS grants were required to
provide assurance of
reporting, on a quarterly basis, information regarding
the death of any person who is in the process of arrest, is en
route to be incarcerated, or is incarcerated at a municipal or
county jail, State prison, or other local or State correctional
facility (including any juvenile facility) that, at minimum,
includes
the name, sex, race, ethnicity, and age of the deceased
the date, time, and location of death
a brief description of the circumstances surrounding

the death.

The Bureau of Justice Statistics (BJS) implemented the


data collection requirement under DICRA legislation. In
compliance with the specifications outlined in P.L. 106-297,
BJS began quarterly data collections to cover all inmate
deaths in local jails (2000), state prisons (2001), and juvenile

Arrest-Related Deaths Program: Data Quality Profile | March 2015

correctional facilities (2002). In 2003, BJS expanded the


Deaths in Custody Reporting Program (DCRP) to include
deaths that occur in the process of arrest by state and local law
enforcement agencies.
The DCRP provides data that describe the prevalence and
incidence of deaths that occurred while the decedent was in
the custody of law enforcement personnel, local jails, or state
prisons. These data collections provide policymakers, public
health officials, law enforcement officials, and correctional
administrators with a comprehensive system for monitoring
deaths within three major components of the criminal justice
system: the police, local jails, and corrections. The DCRP
collections are the only source of this type of information at
national, state, and local levels.
Prior to the DCRP, prison mortality data were collected at the
national level on a limited number of deaths (i.e., homicide,
suicide, natural causes) and at levels of aggregation that
precluded analysis of subpopulations. Jail mortality data were
collected at infrequent intervals through BJSs Census of Jails
series; however, like prison mortality data, these data were
limited to broad categories of causes and provided limited
information on the demographic characteristics of decedents.
Deaths in juvenile correctional facilities were collected
periodically by the Office of Juvenile Justice and Delinquency
Prevention (https://www.ncjrs.gov/pdffiles1/ojjdp/228128.pdf).
No national collection of arrest-related deaths, as described
by the DICRA legislation, had ever been completed.1 Arrestrelated mortality data were restricted to small local studies,
mostly at the agency level, and focused predominantly on
officer-involved homicides (Klinger, 2012; Loftin, Wiersema,
McDowall, & Dobrin, 2003).
With information collected through the DCRP, BJS can track
changes in the mortality rates of persons who have contact
with the police or are in custody in jail or prison. The DCRP
data have improved our understanding of civilian deaths while
in the criminal justice system. For example, through the DCRP,
BJS has shown that almost 40% of deaths that occur in the
process of arrest arise from causes other than officer-involved
homicides of suspects (Mumola, 2007; Burch, 2011); that in
any given year, about 80% of the roughly 3,000 jails in the
United States have no deaths and, among those jails reporting
deaths, the modal count is one death; that the leading cause of
death in prisons is cancer, followed by heart disease; and that
most inmates who died in custody had medical conditions
that predated their arrival to prison or jail, as opposed to
1Two national-level programs, the Uniform Crime Reports
Supplementary Homicide Reports (SHR) and National Vital Statistics
System (NVSS), collect information on homicides by law enforcement,
but neither the SHR nor the NVSS captures the full scope of deaths
specified by the DICRA legislation. The SHR data are limited to
justifiable homicides by law enforcement. The NVSS data identify deaths
attributed to legal intervention. Deaths that do not directly result from
law enforcement action but still occur during the process of arrest (e.g.,
vehicular deaths resulting from pursuits, suspects dying from medical
conditions, and suicides during standoff situations) are not identified in
the NVSS as legal intervention deaths.

contracting a fatal disease while in custody (Noonan & Ginder,


2013). Since its initiation in 2003, the ARD program has
released two reports on the number and characteristics of
arrest-related deaths (Burch, 2011; Mumola, 2007).
In 2006, the DICRA reporting requirements expired. BJS
continued to collect, analyze, and report on deaths in custody.
In December 2014, the Death in Custody Reporting Act of
2013 was signed into law (P.L. 113-242, see https://www.
congress.gov/bill/113th-congress/house-bill/1447/text).

1.2 Definitions
The DICRA legislation required states to submit quarterly data
on deaths that occur in the process of arrest. BJS designed
the ARD programs scope to include homicides, suicides,
accidental deaths, and deaths attributed to intoxication and
medical conditions that occur during a civilian interaction
with state or local law enforcement.
BJS operationalized the DICRA term arrest-related to
describe the range of circumstances pertaining to an arrest,
including those that occur from the process of apprehension
to detention. BJS defined a death as arrest related when the
event causing the death (e.g., gunshot wound, cardiac arrest,
or drowning) occurs during an interaction with state or local
law enforcement personnel.2 These events can be grouped
into three categories: deaths that occur (1) while a person is
detained or restricted and shortly after the persons freedom
to leave is restricted, (2) during an attempted arrest or in the
process of arrest, or (3) while the person is in custody (before
transfer to jail). Exclusions to this definition are described in
section 1.3.
Shortly after freedom to leave is restrictedAll deaths that
occur shortly after a persons freedom to leave is restricted by
state or local law enforcement personnel should be reported to
the ARD program if the circumstances causing the death occur
during the interaction with law enforcement. For example, if a
detained individual sustained an injury during an interaction
with law enforcement personnel and died later as a result of
those injuries, the death should be reported to the program.
Although the majority of deaths reported to the ARD program
involve criminal suspects, individuals not considered subjects
of arrest can be detained by law enforcement personnel,
such as during a stop-and-frisk search or during a vehicle
stop. In addition, individuals may be in the custody of law
enforcement for medical or mental health assistance. Deaths of
noncriminal persons that occur in the custody of state or local
law enforcement personnel should be reported to the ARD
2BJS

collaborated with state and local law enforcement agencies and


law enforcement professional organizations, such as the International
Association of Chiefs of Police (IACP) and the National Sheriffs
Association (NSA), to establish the specific criteria for determining which
deaths to include within the ARD programs scope. These callibrations
addressed topics regarding which deaths should be eligible for reporting,
the number and type of sources of existing data available in each state, and
expectations for how states should identify within-scope deaths.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

program when the deaths are attributed to events that occur


while the person is engaged with law enforcement.
The most common types of noncriminal deaths reportable
to the ARD program occur during law enforcements
response to service calls for assistance or transport. Often,
these service calls are civilian requests for mental health or
medical assistance. These incidents may involve individuals
who are suicidal or displaying erratic behavior or may occur
during transport to a mental health facility or hospital. Law
enforcement may have engaged with the decedent to facilitate
care during an emergency, assist with involuntary commitment
to a medical facility, or support emergency medical technicians
while transporting an individual for continued care.
During an attempted arrest or in the process of arrest
Arrest-related deaths include all deaths that occur during an
interaction with law enforcement personnel in the process
of arrest or attempted arrest, regardless of whether physical
custody was established before the death. Deaths occurring
before arrest include those that are attributed to events that
transpired either during apprehension or while the decedent
was detained for questioning or investigation.
Deaths that occur while law enforcement personnel attempt
to apprehend or arrest an individual (e.g., those that occur
during foot or vehicle pursuits of criminal suspects and during
standoffs or barricade situations) are reportable. Common
examples of deaths that occur during apprehension include
officer-involved shootings; deaths related to the use of force
or law enforcement compliance weapons3 or tactics; vehicle
accidents and collisions caused by either the decedent,
intervening law enforcement personnel, or unrelated civilians;
other types of fatal accidental injuries sustained while
attempting to elude law enforcement personnel, such as falls
from heights and drowning; and suicides committed during
standoffs and barricade situations.
Deaths that occur during interviews and interrogations,
or while a criminal suspect is detained for questioning or
investigation, are also within the scope of the ARD program
and should be reported. These arrest-related deaths may take
place at a law enforcement facility or in the field. Examples of
these types of deaths include those attributed to alcohol and
drug intoxications, sudden medical conditions (e.g., cardiac
arrest, asthma, stroke, or seizure), choking on ingested objects
or other forms of asphyxiation, and suicides.
In addition to deaths that occur before a physical arrest, deaths
that occur while law enforcement attempts to establish physical
custody of a suspect are also reportable to the ARD program.
All deaths that are attributed to weapon use by state or local
law enforcement personnel are considered arrest related
and are reported. Common examples of deaths attributed to
law enforcements use of weapons include officer-involved
shootings, complications related to the use of conducted
3Compliance

or less-than-lethal weapons are designed to induce offender


compliance while minimizing accidental, incidental, and correlative
casualties associated with weapon use.

energy devices (i.e., Tasers or stun guns), accidents caused


by the use of spike strips or other tire deflation devices, fatal
injuries due to the use of impact devices (e.g., batons and
soft projectiles) and complications due to the use of chemical
agents(e.g., pepper spray and tear gas).
All deaths caused by law enforcements use of restraint
tactics are also reported to the ARD program. Eligible
deaths attributed to restraint or use of other tactics include
fatal injuries caused by physical fighting or struggle with
law enforcement personnel; deaths caused by positional
asphyxia or restraint in prone position; fatal injuries due to
physical restraint by law enforcement personnel, such as those
attributed to the use of control holds or neck restraint; and
deaths caused by complications due to body compression.
While in custody (before transfer to jail)Deaths that occur
after law enforcement personnel have established physical
custody of an arrestee are reported to the ARD program if the
deaths occur while custody of the suspects resides with the
arresting agency. These in-custody deaths can occur at the
scene of the incident; during transport of a criminal suspect
or transport of a person in need of medical or mental health
assistance; or while a suspect is being held at a law enforcement
facility, such as a booking center or lockup facility.
For the purposes of ARD, an in-custody death refers only to
the deaths of criminal suspects under the control of state and
local police officers and sheriffs deputies during booking, but
before arraignment or transfer to a jail or prison. Once an
arrestee is arraigned or custody of the individual is transferred
to a long-term correction facility, the death is no longer within
the scope of the ARD program. The deaths of incarcerated
inmates are recorded in either the jails or prisons component
of the DCRP.
Common examples of deaths that occur during transport or
confinement include those attributed to complications related
to the use of weapons (e.g., a firearm or conducted energy
device) during the arrest incident; injuries caused by the
use of restraint or impact devices, as well as those resulting
from physical altercation; fatal intoxications attributed to the
overuse of alcohol and drugs; medical conditions; and suicides.

1.3 Exclusions
Not all deaths that occur during an interaction with state and
local law enforcement personnel should be reported to the
ARD program. Four general situations are typically excluded:
(1) deaths of bystanders, hostages, and law enforcement
personnel; (2) deaths of persons in the custody of federal law
enforcement agents; (3) deaths of wanted criminal suspects
before police contact; and (4) deaths by vehicular pursuits
without any direct police action.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

Deaths of bystanders, hostages, and law enforcement


personnelThe deaths of innocent bystanders, hostages, and
law enforcement personnel are excluded. Including persons
killed in the course of law enforcement activities against whom
no charges were intended was considered outside the scope of
the ARD program.
Data on law enforcement personnel killed during interactions
with civilians are outside the scope and are captured by the
FBI in the Law Enforcement Officers Killed and Assaulted
(LEOKA) collection.
Deaths by federal law enforcementDeaths that occurred
while the decedent was in the custody of federal law
enforcement agencies (e.g., the FBI, Drug Enforcement
Administration, or U.S. Marshals Service) are outside the
scope of the ARD program as defined in the original DICRA,
which did not apply to federal law enforcement agencies or
the Federal Bureau of Prisons. Fatal incidents involving federal
law enforcement personnel are reportable to ARD only if
the interaction causing death also included state or local law
enforcement personnel.4
Deaths of wanted criminal suspects before police contact
The death of a criminal suspect wanted by law enforcement
personnel are not reported to the program if the death
occurred before the decedent came into contact with law
enforcement. The death of a person with an active arrest
warrant are not reported to the ARD collection unless law
enforcement personnel were present when the event causing
the death occurred. If law enforcement personnel were actively
attempting to apprehend a wanted suspect at the time the event
causing the death occurred, the death is within the scope of
the collection.
Deaths by vehicular pursuits without any direct police
actionAlthough some vehicular deaths can be reported to
the ARD program, those in which law enforcement did not
take direct action against the subject or his or her vehicle
should be excluded from the data collection. Cases in which
a vehicular pursuit of an arrest subject involved direct action
taken by law enforcement personnel against the subject, such
as shooting at the subjects vehicle, ramming it, or otherwise
forcing the vehicle to stop or leave the road (e.g., roadblocks
or spike strips), are within the scope of the collection. Deaths
resulting from vehicular pursuitsregardless of speedare
not reportable to the ARD program if the pursuit did not
involve any direct law enforcement action taken against the
subject and instead merely entailed following the subject.
4Current

Death in Custody Reporting Act of 2013 (P.L. 113-242) includes


federal law enforcement within scope.

2. Methodology
HIGHLIGHTS
Centralized state reporting coordinators (SRCs) operate the

state-level ARD data collection.


The ARD program has three main tasks:
identify arrest-related deaths
acquire information about arrest-related deaths
submit the CJ-11 form, Quarterly Summary of Arrest-

Related Deaths (or summary of incidents) and the CJ-11A


form, Arrest-Related Death Report (or incident report).
SRCs use one or more of the following sources to identify

eligible ARD cases:


law enforcement
medical examiner or coroner
prosecutors office
Uniform Crime Reports (UCRs)
National Violent Deaths Reporting System (NVDRS)
open-source media search.

To implement the ARD collection, BJS developed a program


methodology that relied on establishing centralized SRCs and
using them to operate state-level ARD data collections (see
appendix B for a description of how the current ARD program
methodology was developed).5
Since 2003, SRCs have been responsible for identifying all
ARD reportable deaths in their states and for obtaining
information about these deaths. BJS provided guidance in how
to collect ARD data but did not require SRCs to use a specified,
standard methodology. BJS recommends that states use one
or more of the following procedures to identify arrest-related
deaths: implementing legislation; contacting state medical
examiner and county coroner officers; involving state and local
prosecutors offices; contacting the state police; expanding the
role of the state UCR reporters; or conducting open-source
searches.
Once a reportable death is identified, SRCs are directed to
work with the law enforcement agency involved with the
death to obtain the required data elements about the incident.
When the SRC is unable to obtain full reporting from the law
enforcement agency involved in the death, the SRC is asked
to compile details of the event from official source documents
5When

the DICRA legislation was enacted, only California and Texas had
statutes requiring that information on arrest-related deaths be collected
and reported at the state level. For the remaining 48 states and the District
of Columbia, the ARD program was the first attempt to collect statewide
counts of all deaths that occur in the process of arrest. In developing the
program, BJS contacted multiple offices in each state to determine an
appropriate reporting agent and data provider.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

and to complete a CJ-11A Incident Report (appendix A). As


a result, SRCs have developed strategies for identifying and
collecting data that vary by the resources available to them.
Consequently, the current ARD program design reflects many
state-specific methodologies that are integrated into a single,
national-level program.

2.1 Data providers


Information about data providers participating in the ARD
program is available for calendar years 200311.6 During 2011,
47 reporting offices submitted ARD data (table 2-1). Most
States have maintained participating SRCs over time, with
36 states submitting ARD data every year of data collection
since 2003 (states are not shown in table 2-1). Although a state
may not have participated in the ARD program in a given year,
arrest-related deaths may have been identified and reported by
BJS staff in these states.
During 2011, a state criminal justice agency was the most
common data reporting contact (16 states), followed by a
department of public safety or other state law enforcement
agency (14 states) (table 2-2). In three states, the department of
corrections took a lead role in compiling ARD records.
Participation in the ARD program is voluntary without direct
payment or compensation. Only SRCs employed by state
justice statistics analysis centers (SACs) have received federal
funds to support ARD-related work. SACs are units or agencies
at the state government level that use information from all
components of the criminal justice system to analyze statewide
policy issues (see www.jrsa.org). SACs are maintained in
various entities, such as state planning or coordinating
agencies, governors offices, criminal justice agencies,
and universities.
For 32 states in 2011 (68%), the SRC office reporting ARD data
also served as the SAC.

2.2 Data collection process


The ARD program data collection process includes three
main tasks:
(1) identify arrest-related deaths
(2) acquire information about arrest-related deaths
(3) submit CJ-11summary of incidents and CJ-11A
incident reports.
SRCs are responsible for identifying all arrest-related deaths
within their states. Arrest-related deaths may be identified
directly by the SRC or indirectly through assistance from
another entity involved with the death (e.g., law enforcement
agency) or death investigation (e.g., medical examiners or
coroners office, prosecutors office). SRCs use a variety of
6Although ARD data were accepted after 2011, program staff did not
actively collect data for calendar years 2012 or 2013.

TABLE 2-1
States or jurisdictions that did not report to the Arrest-Related Deaths Program, 200311
State or jurisdiction
Total reporting
Arkansas
District of Columbia
Georgia
Louisiana
Maryland
Mississippi
Montana
Nebraska
Nevada
New Mexico
North Carolina
Oklahoma
Tennessee
Wisconsin
Wyoming

2003
44
/

/
/
/
/
/

2004
46

2005
46

2006
44

/
/

2007
47

2008
43
/
/
/

/
/

2009
42
/
/
/

/
/

/
/

2010
47
/

/
/

2011
47
/

/
/

Note: States not listed reported to the ARD program in each year from 2003 through 2011.
Reported.
/ Not reported.
Source: Bureau of Justice Statistics, Arrest-Related Deaths Program, 2011.

TABLE 2-2.
Number of state reporting coordinators, by state, Statistical Analysis Center (SAC) status, and type of reporting office,
2011
Located in SACs
Type of reporting office
Number of all reporting offices
State criminal justice agency
Department of public safety/law enforcement
University
Attorney generals office
Governors office
State department of correction
Office of the chief medical examiner
Office of public health

Total
47
16

Number
32
14

14

4
4
3
3
2
1

4
2
3
0
0
0

State
AL, AZ, CT, DE, IL, IA, KS, KY,
MT, NE, NY, RI, VA, WV
CO, FL, MO, NJ, OH,
OK, SC, SD, TN
AK, MS, NV, NM
CA, NH
MD, MA, UT

Not located in SACs


Number
State
15
2
PA, VT
5
0
2
0
3
2
1

DC, HI, MI, MN, WA


ND, TX
ID, IN, LA
ME, NC
OR

Note: Four states did not report data in 2011.


Source: Bureau of Justice Statistics, Arrest-Related Deaths Program, 2011.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

techniques to identify deaths that are consistent with ARD


program definitions. BJS encourages SRCs to take an active
approach in identifying deaths, rather than relying solely
on a system of voluntary reporting by law enforcement or
nongovernmental entities. Active strategies are those in
which the SRC proactively searches for arrest-related deaths,
such as methods that include surveying law enforcement
agencies, mining existing data collections, and conducting
regular open-source searches, in addition to possibly receiving
voluntary submissions from law enforcement agencies and
medical offices.
SRCs are responsible for obtaining complete incident reports
for each arrest-related death in the state and for transmitting
instruments to ARD program staff for data entry, analysis,
and dissemination at the national level. Although all three of
these components are incorporated into the national program
design, the processes by which data are collected vary by SRC.
2.2.1 Active and passive approaches to case identification
In 2011, most SRCs employed active strategies for identifying
deaths. Of the 45 SRCs reporting information about their
data collection procedures, 41 reported using a proactive
strategy to identify arrest-related deaths. The most common
active identification strategy involved conducting open-source
searches for reportable deaths. In total, 32 SRCs reported
conducting web-based searches as a means for identifying
reportable deaths: 16 conducted solely open-source searches,
and 16 conducted open-source searches in addition to another
strategy (table 2-3).
SRCs using passive methodologies rely on other agencies to
identify arrest-related deaths. Passive strategies are those in
which data are collected through voluntary submissions of
completed CJ11A incident reports from other entities, such as
law enforcement agencies and medical examiners or coroners
offices. SRCs employing passive methodologies do not conduct
independent searches for arrest-related deaths. SRCs in four
states said that deaths were identified solely through voluntary
reporting by law enforcement agencies.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

TABLE 2-3
Number of states or jurisdictions using active case
identification strategies, by data source, 2011
Source
All states or jurisdictions
Active case identification strategies
Single-method identification
Open-source search
Law enforcement survey/referral
Medical examiner data request
Othera
Multiple-method identification
Open source and law enforcement survey/referral
Open source and database searchb
Open source and medical examiner data request
Medical examiner and prosecutor survey
Passive case identification strategies
Unknown case identification strategies
Nonparticipating states

Number
51
41
24
16
3
3
2
17
7
5
4
1
4
2
4

aIncludes sole use of the National Violent Deaths Reporting System (one state) or
reports to the office of the attorney general (one state).
bIncludes National Incident-Based Reporting System (NIBRS), National Violent
Deaths Reporting System (NVDRS), and Uniform Crime Reports (UCR).
Source: Bureau of Justice Statistics, Arrest-Related Deaths Program, 2011.

2.2.2 Number and type of case identification strategies


Of the 41 SRCs with known active case identification strategies,
17 mentioned using multiple strategies for identifying arrestrelated deaths, while 24 (59%) said that they used a single
method for identifying cases. Among those 24 SRCs, opensource searches were the most common (16 SRCs), followed
by direct reporting by law enforcement agencies (3), a query of
medical examiners or coroners databases (3), a query of the
NVDRS (1), and officer-involved death reports submitted to
the office of the attorney general (1).
The use of open-source searches as a method of identifying
arrest-related deaths was also common among SRCs using
multiple case identification strategies. Among the SRCs
implementing multiple strategies for identifying deaths,
16 in 17 used open-source searches combined with other
methods, such as surveys of law enforcement (7 SRCs) and
queries of medical examiners or coroners databases (4 SRCs),
NVDRS (3 SRCs), and the National Incident-Based Reporting
System (NIBRS) or UCR (2 SRCs). One other SRC searched
medical examiners databases and conducted statewide surveys
of prosecutors reports.

2.2.3 Primary identification methodology


Of the 45 participating states with known case identification
methodologies in 2011, 26 SRCs used the media and opensource searches as their primary manner of identifying
arrest-related deaths (table 2-4).7 Eight SRCs identified
deaths primarily through direct reporting by law enforcement
agencies, five through medical examiners or coroners offices,
three through the NVDRS programs, two through the state
attorney generals or prosecutors offices, and one through the
UCR program.
Although SRCs used similar strategies for identifying deaths,
differences were found in how methodologies are applied.
SRCs reported using media accounts as their primary case
identification methodology more frequently than any other
method; however, this strategy varied the most across states.
For example, many SRCs stated that their methodology was
to watch the local television news, read the newspaper, or
both. Although some arrest-related deaths will be identified by
monitoring local news, this method is not likely to be sufficient
for identifying the full universe of reportable deaths. One
problem relates to whether local outlets are capable of covering
7SRCs

using multiple strategies for identifying arrest-related deaths were


able to indicate which method was their primary method for collecting
data and which strategy served as a secondary method for collecting data.

TABLE 2-4.
Primary case identification methodologies, 2011
Source
All participating states or jurisdictions
Open-source media search
Law enforcement agency report
Medical examiner database
National Violent Deaths Reporting System
State attorney generals or prosecutors office
Uniform Crime Reports

Number
45
26
8
5
3
2
1

Percent
100%
58
18
11
7
4
2

Note: The primary source for identifying arrest-related deaths was unknown in two
states. Four states did not report data in 2011.
Source: Bureau of Justice Statistics, Arrest-Related Deaths Program, 2011.

the ARD scope for the entire state, while another problem
relates to whether this type of monitoring is conducted
systematically. It was not clear from discussions with SRCs that
routine procedures were employed when monitoring news
outlets (e.g., daily monitoring or monitoring the same outlets).
It was also unclear how SRCs determined which outlets to
monitor and whether any investigation was conducted to
assess the appropriateness or accuracy of the source content.
Besides traditional media, many SRCs also used open-source
Internet searches to identify deaths. Although some specifically
mentioned monitoring the web page of the state police for
incidents of arrest-related deaths, others had a more difficult
time specifying their procedures. Some SRCs also reported that
they conducted searches for terms relating to arrest-related
deaths, but it was unclear whether searches were conducted
ad hoc or routinely. It was also unclear if the same terms were
searched consistently or if SRCs varied their terms when
searching for arrest-related deaths. Without information about
specific search terms used by SRCs, the variation of search
terms are unknown across SRCs. Overall, although 26 states
reported using media searches as their primary mode for
identifying arrest-related deaths, no two SRCs used the exact
same procedures.
2.2.4 Types of state reporting coordinators and primary
methods for identifying arrest-related deaths
SRCs were categorized by the location of their offices in an
attempt to better understand whether there is a relationship
between the type of reporting office and the primary method
for identifying arrest-related deaths. Among the SRCs located
in state criminal justice agencies, 10 in 14 relied on opensource searches as their primary method for identifying arrestrelated deaths. Half (7) of the SRCs located in departments of
public safety or another state law enforcement agency also used
open-source searches as their primary means of identifying
deaths. Among the 14 SRCs located in departments of public
safety or another state law enforcement agency, three identified
arrest-related deaths primarily through direct reporting by law
enforcement agencies (table 2-5).

TABLE 2-5.
Primary information source for identifying arrest-related deaths, by location of state reporting coordinator office,
2011
Location of office
Total
State criminal justice agency
Dept. of public safety/law enforcement
University
Attorney general's office
Governor's office
Department of correction
Medical examiner's office
Office of public health

All offices
45
14
14
4
4
3
3
2
1

Open source
search
26
10
7
3
1
1
3
1
0

Data request to National Violent


Direct reporting by medical examiner Death Reporting
law enforce-ment
or coroner
System
8
5
3
2
0
1
3
3
0
1
0
0
2
0
0
0
1
1
0
0
0
0
1
0
0
0
1

Investigative
reports
2
0
1
0
1
0
0
0
0

Uniform Crime
Reports
1
1
0
0
0
0
0
0
0

Note: The primary source for identifying arrest-related deaths was unknown in two states. Four states did not report data in 2011.
Source: Bureau of Justice Statistics, Arrest-Related Deaths Program, 2011.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

2.2.5 Source of case information


Once an arrest-related death is identified, the SRC is
responsible for either obtaining information about the death
directly or coordinating efforts by other entities to obtain caselevel information. In states in which SRCs identify reportable
deaths, the SRC may take responsibility for acquiring
information about the incident or may request that another
agency involved with the death (e.g., law enforcement agency,
medical examiners office, or prosecutors office) provide
information about the circumstances surrounding the death.
In states in which deaths are identified by other agencies,
the agency that identified the death takes responsibility for
gathering information.
SRCs responsible for directly obtaining information about
the death acquire case information from a variety of sources,
including police reports or press releases, death records or
autopsy reports, legal accounts, and the media. Non-SRC
entities responsible for gathering case information typically use
the same data sources as SRCs because these agencies are often
those responsible for initially collecting the information. For
example, law enforcement agencies completing the CJ-11A will
likely use information from the police reports they produced.
2.2.6 Overlap in sources used to identify deaths and to
obtain case information
SRCs vary in the number and types of sources they use to
obtain information about arrest-related death incidents. Some
SRCs rely on a single source of information to both identify
deaths and gather information about incidents. Other SRCs
use a multistep data collection process in which information
used to complete the CJ-11A comes from a source other than
the one used to identify the death. For instance, a singlesource data collection process could involve using a media
account both to identify the arrest-related death and to obtain
information about it. A multisource process involves using one
source (e.g., media report) to identify a reportable death and
one or more additional sources of information (e.g., a police
report or autopsy report) to obtain case information.
Eleven SRCs (24%) used a single source of information for
identifying arrest-related deaths and completing CJ-11A
incident reports (figure 2-1). Of those, the most common
source of information was law enforcement reports (six SRCs),
followed by media reports (two SRCs), medical examiner
records (two SRCs), and UCR data (one SRC).
Most SRCs (72%) relied on multiple sources of information
to identify deaths and to obtain incident data during 2011.
Thirty-six percent of SRCs relied on both the source of
information used to identify arrest-related deaths and at

Arrest-Related Deaths Program: Data Quality Profile | March 2015

FIGURE 2-1
Overlap in sources used to identify deaths and to
complete CJ-11A incident reports, 2011
Same source used to
identify deaths

11

Different sources used


to identify deaths

17

Identification source and


additional sources used

17
2

Missing
0

10

15

20

Note: Four states did not report data in 2011.


Source: Bureau of Justice Statistics, Arrest-Related Deaths Program, 2011.

least one additional source to obtain the incident data for


completing the CJ-11A. Thirty-six percent of SRCs used a
multistep data collection process in which one source was used
to identify deaths and a different source was used to obtain
information about the death. The most common example of
a multisource process involved using open-source searches to
identify an arrest-related death and a police report to complete
the CJ-11A.
During 2011, about two-thirds of SRCs used only official
sources of information for obtaining data regarding incidents
of arrest-related deaths (table 2-6). Law enforcement reports
were the most common source of incident-level information
about arrest-related deaths. A small number of SRCs (4%)
relied on unofficial sources of information, such as journalistic
accounts of events, to obtain details about reportable incidents.
TABLE 2-6.
Types of data sources used to obtain arrest-related death
case information, 2011
Source
Total state reporting coordinators
Official sources data
Law enforcement report only
Medical examiner report only
Multiple official sources
Combination of official and unofficial sources
Unofficial sources of information

Number
45
30
19
4
7
13
2

Percent
100%
67%
42
9
16
29%
4%

Note: The primary source for identifying arrest-related deaths was unknown in two
states. Four states did not report data in 2011.
Source: Bureau of Justice Statistics, Arrest-Related Deaths Program, 2011.

2.3 Completion of CJ-11 and CJ-11A instruments


The ARD program uses two data collection instruments, the
CJ-11 Quarterly Summary of Arrest-Related Deaths (summary
of incidents) and the CJ-11A Arrest-Related Death Report
(incident report). The CJ-11 form is completed by SRCs and is
a state-level count of arrest-related deaths. SRCs are directed
to submit a CJ-11 form for each reporting period and to show
a count of zero if no arrest-related deaths are identified within
the reporting period. BJS requests that SRCs submit zero
counts to distinguish between the measured absence of an
identified death and missing data.
The CJ-11A form is a detailed incident report that describes
the circumstances surrounding each reportable death. SRCs
are asked to submit a complete CJ-11A incident report for
each death recorded on the state-level CJ-11 summary form.
Depending on the method used to identify the death and
the agency responsible for acquiring information about the
incident, the SRC may take responsibility for completing the
CJ-11A forms or may request that another agency complete
the forms.
Information about the entity responsible for completing
CJ-11A forms during 2011 was available in 45 states. The law
enforcement agency involved with the death was primarily
responsible for completing CJ-11A incident reports in 20 states
(44%) (figure 2-2). SRCs obtained case information and
completed CJ-11A reports in 16 states (36%), followed by staff
in medical examiners offices (2 states) and prosecutors offices
(1 state). Multiple entities took responsibility for completing
sections of the CJ11A incident report in six states, with joint
efforts by the SRC and law enforcement agency in three states,
the SRC and medical examiners or coroners office in two
states, and the law enforcement agency and medical examiners
office in one state.
FIGURE 2-2.
Entity responsible for completing CJ-11A incident
reports, 2011
Number of states

The ARD data collection is conducted on a calendar-year cycle.


The program operates continuously and accepts submissions
throughout the year. The ARD program also accepts reports of
arrest-related deaths that occurred before the current calendar
year and updates to previously reported data.
In January, BJS notifies SRCs that it is beginning a new year
of ARD data collection and provides them with key program
materials, including (1) the CJ-11 summary of incidents form,
(2) the CJ-11A incident report, and (3) a question-by-question
guide to aid in completing the CJ-11A (see appendix A for
survey instruments and revisions to the CJ-11A incident
report, 200313). In addition, BJS provides the SRCs with
general information about the program, such as key dates,
methods for submitting data, and contact information for ARD
program staff.
When the DICRA legislation was in effect, SRCs were required
to submit ARD data quarterly. They were asked to submit
a CJ-11 Summary of incidents and corresponding CJ11A
incident reports within 60 days of the end of the quarter. After
DICRA expired in 2006, BJS relaxed the quarterly reporting
requirement and gave SRCs the option of submitting their data
either quarterly, annually, or on a schedule of their choosing.
Of SRCs reporting ARD data in 2011, about half (49%)
submitted data annually, while 28% continued to collect and
report data quarterly (figure 2-3). Data can be submitted any
time during the year, but SRCs must submit all annual CJ-11
and CJ-11A reports no later than March 1 of the following
calendar year.
2.3.2 Data submission
After the SRCs complete their annual data collections, they
transmit this state-level data to BJS for analysis. Historically,
the ARD data collection was paper based. Data providers
FIGURE 2-3.
Arrest-related deaths data collection schedule, 2011
Percent
50

25

49%

40

20

30

15

28%
20

10
LEA and ME

5
0

2.3.1 Data collection cycle

10

SRC and ME
LEA and SRC

Law enforcement
State
Medical
agency (LEA) reporting
examiners'
coordinator (SRC) office (ME)

Prosecutors'
office

0
Multiple
entities

Note: The primary source for identifying arrest-related deaths was unknown in two
states. Four states did not report data in 2011.
Source: Bureau of Justice Statistics, Arrest-Related Deaths Program, 2011.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

Annual

Quarterly

9%

9%

Semiannual

Ad hoc

6%
Monthly

Note: Detail may not sum to total due to rounding. Four states did not report data
in 2011.
Source: Bureau of Justice Statistics, Arrest-Related Deaths Program, 2011.

10

would complete paper versions of the CJ-11 and CJ-11A


forms and mail or fax them to BJS. As the program developed,
BJS began accepting the forms electronically through email.
Electronic versions of the forms were submitted either as
fillable PDFs or scanned documents. BJS staff would then enter
information from the CJ-11A forms into a national database.
To improve efficiency, BJS established an Internet-based
data collection tool for the ARD (see appendix B for a
full description). This tool allows SRCs to enter CJ-11A
information directly on a web-based reporting platform rather
than transmitting the completed forms to BJS and its ARD
program staff. Currently, ARD data can be submitted at any
time and are accepted via postal mail, fax, or electronically
through email, an uploaded PDF, or direct entry through the
web-based reporting tool.
As noted previously, annual state-level ARD data are due by
March 1 of the next calendar year. ARD staff review these
data and assess them for completeness and consistency before
entering them into the national database for analysis (see
appendix C for a description of the BJS verification process).
In addition, records are reviewed to determine whether each
reported death falls within the ARD scope.

3. Program assessment
HIGHLIGHTS
A national- and state-level comparison of ARD and SHR

data collected 200305 showed that, despite convergence


at the national level, there were considerable differences in
ARD program coverage at the state level. Similar differences
within states at the agency level were also found in an
analysis of data reported to ARD and SHR in 200609.
Nationally, the ARD program estimates that it has captured

about 50% of all law enforcement homicides during 200309


and 2011.
The national coverage of homicides by law enforcement that

the ARD program captures improved from 2003 through


2011. Even with this improvement, assessments indicate that
between 31% and 41% of homicides by law enforcement
personnel were not captured in the 2011 ARD data.

The ARD program relies primarily on state-level respondents


to gather and report information on deaths associated with the
arrest process for all state and local law enforcement agencies.
As described in the Methodology section (chapter 2), the
manner in which these incidents are identified and subsequent
information collected varies between and within states and
over time, leading to nonsampling errors from differences in
use and interpretation of definitions, scope, and data collection
modes. Although an empirical assessment of the relative level
of error associated with each data collection approach is not
known and would be very difficult to design, many of these
errors have been realized from internal audits and anecdotal

Arrest-Related Deaths Program: Data Quality Profile | March 2015

accounts from data providers and SRCs. Subsequently, this


nonstandardized data collection process invites concerns about
the coverage, bias, and reliability of reports.
This chapter describes critical aspects of the ARD data
collection methodology, noting important limitations
and challenges to producing valid and reliable national
estimates. These challenges include variations in state-level
methodologies related to (1) whether SRCs actively identify
within-scope cases and the level of effort they apply to their
search for reportable cases, (2) what SRCs use as their primary
case identification strategies, (3) whether SRCs use one or
multiple methods for identifying deaths, (4) what source of
information they use to complete CJ-11A forms and whether
they use multiple sources of information, and (5) which agency
is responsible for completing the form. To highlight these
challenges, we describe issues related to scope and definitions,
to data collection strategies, and modes. We follow this
description with an assessment of the coverage of the ARD
program, focusing on law enforcement homicides.

3.1 Defining eligible incidents


The ARD program has definitions and a scope for eligible
incidents that are difficult to consolidate into a comprehensive,
precise set of terms. The program relies on a set of examples
to help identify eligible cases and guide data collection
agencies. An arrest-related death can occur at any time and in
any location where law enforcement personnel and civilians
interact, including while fleeing or being pursued, at scenes
of arrest, at locations of medical or mental health assistance,
and while the person is being transported to law enforcement
or medical facilities or is detained at lockups or booking
centers. The breadth of the ARD scope and the variety of
circumstances and locations in which eligible deaths can occur
make it difficult for state-level reporting agents to identify all
reportable deaths.
3.1.1 Arrest-related
The term arrest-related in the DICRA legislation
encompasses a variety of circumstances from the process of
apprehension to detention. Confusion about the program
scope arose from the expansive nature of the terms in the
process of arrest and arrest-related. For example, in the
law enforcement community, the process of arrest means
Mirandizing and handcuffing suspects and is not used to
describe pursuits, officer-involved shootings, or other uses of
force. However, for ARD purposes, arrest-related deaths can
occur before, during, or after an arrest, as long as the event
that causes the death occurs during an interaction with state
or local law enforcement personnel. An arrest-related death
describes deaths caused by law enforcement personnel, and
those attributed to suicide, intoxication, accidental injury, and
medical emergencies or health complications. The resulting
characterization of arrest-related deaths is more inclusive
than a collection of officer-involved shootings or justifiable

11

homicides by law enforcement, which may be more easily


understood as eligible incidents. The breadth of the programs
definition of arrest-related creates problems in identifying
eligible cases, likely contributing to a significant undercount
of incidents.
3.1.2 Custody
The term custody includes physically touching or putting
hands on an arrestee. For the ARD program, it also includes
any act that indicates an intention to gain control of an arrest
subject under the authority of the law or any situation in
which a persons freedom to leave is restricted by state or local
law enforcement personnel. This includes deaths occurring
while law enforcement officers actively pursue or attempt to
apprehend persons of interest or criminal suspects, regardless
of whether physical custody is established. For example, many
ARD cases are officer-involved shootings, and virtually none
of these incidents involved subjects who were already in the
physical custody of law enforcement personnel.
Custody includes individuals in physical custody of law
enforcement without being in the process of arrest. To gain
control of a situation, law enforcement personnel sometimes
handcuff or detain persons against whom they do not plan to
file charges (e.g., mentally ill or intoxicated persons, or others
who pose a danger to their own safety).
3.1.3 Reporting deaths in jails
The goal of the DCRP is to identify and record all deaths of
civilians in contact with law enforcement and correctional
personnel. BJS implemented three separate data collection
systems to capture deaths occurring in the process of arrest and
those that occur while the person is detained in jail or prison.
Deaths reportable to the ARD program are limited to those
that occurred while the decedents were in the custody of law
enforcement. Therefore, deaths that occur after an individual
has been processed and custody has been transferred to a
holding facility are outside the ARD scope. Given that the
functions of some law enforcement agencies include detention,
some SRCs reported difficulty in determining whether the
decedent died in the custody of law enforcement or the
custody of a correctional facility.

3.2 Assessment of data collection modes


3.2.1 Standardization
The ARD program has implemented variable methodologies
since its first collection in 2003. As a result, data providers have
designed and developed state-level data collections according
to their preferences, their capabilities, and the resources
available to them. The flexibility of the ARD data collection
allows state-level data providers to use a variety of data sources
and to rely on those that are most reliable in their particular
states, but it presents significant challenges for collecting

Arrest-Related Deaths Program: Data Quality Profile | March 2015

statistically sound, national-level data because of the variability


in data quality both within and across states. BJS has developed
standardized data collection instruments (CJ-11A form),
training materials, working definitions, and a web tool to
assist SRCs in gathering and submitting the ARD information
(see Appendix B). However, significant challenges within and
between states exist in terms of mode, source, and quality.
3.2.2 Variation in program methodologies across states
SRCs developed their data collection methodologies in
relation to their preferences and resources. As a result, the
ARD program operations varied among participating states
(see chapter 2: Methodology). Deaths may be identified from
one or more resources from among media, open sources, law
enforcement agencies, and medical examiners or coroners
offices. The information sources consulted for completing each
CJ-11A form also differed, and included media reports, police
reports, medical examiner or coroner reports, case reports
from the division of criminal information, or any combination
of these sources. Data collection methodologies also differed
by the agency responsible for completing the CJ-11A form,
which can include the SRC, a law enforcement agency, a
medical examiners or coroners office, or a combination of
those offices.
BJS assessed the SRCs primary method for identifying deaths,
the sources of information used to complete the CJ-11A
incident report, and the entity completing the CJ-11A. BJS
determined that SRCs used a wide range of methods for
collecting ARD data in 2011. The most common method
(nine SRCs) for collecting ARD data was for the SRC to
identify cases through open-source searches.8 Then, the SRC
asked the law enforcement agency involved with the death to
complete a CJ-11A incident report using the agencys official
data about the case. The second most common method (six
SRCs) for collecting ARD data was direct reporting by the
law enforcement agency involved with the death. In these
states, the law enforcement agency involved with the arrestrelated death took responsibility for identifying it, obtaining
and completing a CJ-11A incident report using the agencys
official data, and submitting the form to the SRC. In three
states, SRCs identified deaths through media searches and
completed CJ-11A instruments by obtaining the manner and
cause of death from a medical examiners or coroners office.
The rest of the information requested on the CJ-11A form was
obtained through the same open-source record used to identify
the death, namely, media accounts of events. Two SRCs
identified arrest-related deaths through open-source searches
and completed the corresponding CJ-11A instruments with
information obtained in the media account used to identify the
death. The remaining 24 states reporting information about
their 2011 ARD data identification and collection strategies
used unique methodologies with varying combinations of the
source used to identify the arrest-related death, the agency
8The

open-source search methodology is a general description for review


of publicly available data. This term does not indicate a specific or single
procedure or strategy for identifying and collecting data.

12

responsible for completing the CJ-11A form, and the source of


information for completing the CJ-11A form.
Most states (26) relied solely on open-source searches to
identify ARD-eligible cases. Although about half of the
participating states said that they used open-source searches
as their primary strategy for identifying deaths, no standard
methodology exists for this type of identification procedure.
Therefore, each of these 26 states was likely to use different
mechanisms, searches, and monitoring plans for identifying
arrest-related deaths.
3.2.3 Variation in program methodologies within states
Some SRCs noted that they did not have a standard operating
procedure for identifying or collecting information on arrestrelated deaths. They could specify the primary strategies
they used to identify cases (e.g., media or law enforcement)
and whether or not they also used alternative methodologies
for identifying deaths. However, the SRCs suggested that
it was more difficult to describe their exact data collection
procedures because they often varied from one case to another.
For example, SRCs relying on media or open-source searches
described identifying cases by a variety of means. These
included watching television, reading local newspapers, using
Google Alerts to conduct daily web-based searches of key
words or phrases, monitoring the state police web page, and
searching other websites. In discussions with SRCs, it appeared
to be more common for deaths to be identified ad hoc rather
than through specified, standard procedures conducted
routinely. Of the SRCs reporting more standardized procedures
for identifying cases, some said that their methods of obtaining
the data needed to complete the CJ-11A varied from case to
case. For instance, some states that relied on law enforcement
agencies to complete the CJ11A forms had contingency plans
for collecting data if the law enforcement agency involved with
the death refused to supply the information. One SRC reported
that only about 50% of law enforcement agencies return the
completed CJ-11A form. When the law enforcement agency
failed to provide the completed form, the SRC attempted
to obtain a report from the medical examiners office for
the information they needed to complete the form. If the
information required to complete the CJ-11A was unavailable
from the medical examiners office, the SRC used media
reports to complete the report.
Data collection methodolgies for SRCs participating in
the ARD program differed over time. A survey of SRCs
conducted in 2007 revealed that 42 of 47 that reported data
used more than one source of data to identify deaths and
complete CJ-11A forms. Commonly referenced sources
included local law enforcement agencies (43 SRCs), media
(30), medical examiners or coroners offices (23), state
police (19), state UCR/SHR reporters (9), and state attorneys
general or prosecutors (6). A follow-up discussion conducted
in 2012 suggested that 28 of the 46 states reporting data
used a single source of information for identifying cases. Of
these, commonly referenced single sources of information

Arrest-Related Deaths Program: Data Quality Profile | March 2015

included media (16 SRCs), law enforcement agencies (7),


medical examiners or coroners offices (3), the state attorney
general or prosecutor (1), and the state NVDRS reporter (1).
Twenty of the 46 SRCs reporting data indicated that they used
multiple sources of information to complete CJ-11A forms
for identified deaths (16 used two sources, and 4 used three or
more sources). As a result of the variability both within and
between states, records were often identified through a variety
of means and comprised both official and unofficial accounts
of the events.

3.3 Assessment of data quality and coverage


Given the concern about the lack of standardization both
across and within SRCs in terms of data collection modes,
definitions, scope, and availability of the necessary resources,
BJS assessed the quality of the programs coveragethat is,
how well the program performs in capturing the full universe
of eligible arrest-related deaths.
3.3.1 200305 national and state assessment
Initial efforts began in 2007 with a focus on assessing the
coverage of the 200305 ARD program by comparing
homicides reported to ARD with those reported in the
Supplementary Homicide Report (SHR) (Mumola, 2007). The
SHR is a component of the FBIs Uniform Crime Reporting
(UCR) program, which compiles information on crimes
known to state and local law enforcement agencies in the
United States. While the UCR provides aggregate annual
counts of the number of homicides occurring in the United
States, the SHR data provide additional information about
each homicide incident, including the jurisdiction, month,
year, and circumstances associated with the incident. Like the
ARD, the SHR data have limitations. Most notably, the SHR
program is voluntary, so a number of law enforcement agencies
either do not consistently send their data to the SHR or do not
submit it at all.9 The SHR, however, represents the only other
national database that can provide sufficient information on
the circumstances surrounding law enforcement homicides to
allow comparison with the ARD program.10
The ARD-SHR comparisons were limited to law enforcement
homicides because, unlike the ARD program, the SHR does
not record law enforcement-involved deaths that are attributed
to suicide, intoxications, accidental injury, or natural causes.
Also unlike the ARD program, the SHR includes only those
homicides by law enforcement in which the use of force was
ruled justifiable. Deaths due to unjustified use of lethal force
by law enforcement personnel are counted with other murders
9See

The Nations Two Measures of Homicide. NCJ 246832, BJS web,


September 2014.
10The NVSS is a national database that captures homicides occurring in
the United States, but it does not link homicides that are arrest-related
with specific law enforcement agencies as the SHR program does. This
link to a law enforcement agency is critical for capturing-recapturing the
analytic approach used to assess the ARD coverage.

13

and cannot be disaggregated for comparison to ARD. The ARD


counts of homicide by law enforcement include all deaths that
result from the use of force.
Results of this assessment indicate that, nationally, a similar
number of officer-involved homicides were recorded in the
ARD and SHR programs. In the aggregate, the two programs
collected very similar counts from 2003 through 2005, with
1,095 homicides recorded in ARD and 1,082 justifiable
homicides recorded in the SHR. However, the aggregated,
national-level comparison of the ARD and SHR data masked
differences in reporting at the state level.
Similarly, the aggregate analysis at the state level likely also
masked differences within states at the agency level. In most
states, the ARD and SHR counts of law enforcement homicides
showed small differences in the number of deaths reported
to each program, but because arrest-related deaths are rare
events, even small differences in state-level counts could
result in a substantial change in the proportion of reported
deaths. In 33 states, the two measures differed by fewer than 10
deaths from 2003 through 2005. Nine states reported counts
that differed by at least 20 deaths, with the difference ranging
from 22 to 194 homicides. Californiaone of 2 states with a
statute requiring that all arrest-related deaths be reported to
the state attorney generals officehad the largest reporting
variation, with 354 justifiable homicides by law enforcement
reported to the SHR from 2003 through 2005 but less than half
as many (160) reported to the ARD program. Over the same
period, Florida reported 98 law enforcement homicides to the
ARD program but did not report any data to the SHR. Taking
the higher count reported by each state for each year yields a
total of 1,489 reported law enforcement homicides from 2003
through 2005more than a third as many as those reported to
either program.
3.3.2 200609 agency assessment
BJS conducted an agency-level analysis of homicides by law
enforcement reported to the ARD program and to the SHR
for 200609 to determine the number of cases recorded
in the SHR that were not recorded in ARD and vice versa.
Homicides were disaggregated by agency for this comparison,
using the agencys originating identification (ORI) number.
Law enforcement agencies were matched across data sources,
and the numbers of recorded homicides were compared to
better understand how the coverage issues identified in the
state-level analysis were affected by variation in reporting by
law enforcement agencies. BJS was interested in determining
whether the differences in reporting were due to (1) different
agencies reporting to each system or (2) the same agencies
reporting different counts to both systems.
BJS conducted this analysis by creating overage variables to
represent the number of additional cases reported to the ARD
program and to the SHR by each law enforcement agency. For
example, if the Springfield Police Department reported one
homicide by law enforcement to ARD and three homicides by
law enforcement to the SHR, the ARD overage variable would

Arrest-Related Deaths Program: Data Quality Profile | March 2015

equal 0 and the SHR overage variable would equal 2. Although


this analysis was limited in that it did not determine whether
the one case reported to ARD was also one of the three cases
captured in the SHR, it was able to provide evidence of at
least two homicides by the Springfield Police Department
that were not captured in ARD. The SHR overage homicides
were summed as an indication of the minimum number of
homicides by law enforcement missing from ARD (identified
as ARD missing). Similarly, the ARD overage cases were
transformed into SHR missing. For the Springfield Police
Department, the ARD missing equaled 2.
This procedure was conducted for every law enforcement
agency that reported a homicide by law enforcement to either
ARD or the SHR from 2006 through 2009. The analysis
resulted in a minimum of 701 cases reported to the SHR that
were not recorded in ARD, and a minimum of 935 cases that
were recorded in ARD, that were not captured in the SHR.
Although this analysis was limited by several factorsnamely,
that the data were not disaggregated by year of death or
characteristics of the decedentit demonstrated coverage
error at the agency level.
3.3.3 200311 individual case assessment
To better understand the source of discrepancies in reporting
across the ARD program and the SHR, BJS compared
individual-level homicide records in each collection. For this
assessment, BJS examined law enforcement homicides in
ARD and compared them with justifiable homicides in SHR
using the data available from the entire ARD program and
a capture-recapture analysis.11 This analysis addresses a key
obstacle to understanding the coverage of the ARD program.
Because the ARD program includes only state and local law
enforcement agencies that have been involved in one or more
identified arrest-related deaths and does not include agencies
that did not have an identified arrest-related death, it is difficult
to make inferences about coverage using ARD data alone.
Because the SHR data also provide incident-level information
on homicides, they can be used to help assess the coverage of
the ARD data collection. The assessment consisted primarily
of matching records from the two programs and determining
the amount of overlap between collections to estimate the total
universe of reporting deaths. Similar to the other assessments
using SHR data, this assessment is limited to justifiable
homicides by law enforcement (see Arrest-Related Deaths:
Technical Report, NCJ 248544, BJS web, March 2015).
Under the current ARD collection method, SRCs identify
deaths, then usually contact the local agencies in which the
deaths occurred to get the information they need to complete a
CJ-11A for each death. By this method, BJS can affirm that an
identified death occurred and met the criteria to be included
in ARD, but it cannot confirm that there were in fact no other
11ARD data are available for program years 200309 and 2011. However,
the ARD program is designed to capture all arrest-related deaths,
including those due to accidents, drug overdoses, and natural causes.
These incidents are not assessed in this analysis.

14

arrest-related deaths. This is because BJS does not survey each


of the approximately 18,000 state and local law enforcement
agencies in the United States to determine whether arrestrelated deaths occurred.
By linking records submitted to the ARD and SHR collections,
it is possible to make inferences about the total size of the
officer-involved homicide population and, therefore, about
the population coverage achieved in the ARD program. This
analytical technique, known as capture-recapture, is typified
by studies of wild animal populations, although it is widely
used in the study of human populations as well, particularly
in coverage evaluations for censuses. The ARD coverage
assessment is based on traditional capture-recapture methods,
but unique characteristics of ARD data necessitate special
techniques, including probabilistic record linkage between data
sources and adjustments for unobserved agencies.
3.3.4 Probabilistic case-level linking
Capture-recapture analysis depends on the ability to link
records across data sources. In wildlife applications, this
requirement is easily met as captured animals are identified
through tagging. In the context of a coverage assessment using
an auxiliary data source not collected as part of the study,
this requirement becomes more difficult. Because decedents
are not uniquely identified across data sources (e.g., by
name or Social Security number), record linkage must be
based on comparable decedent demographic and incident
characteristics. As linkages made without unique identifiers
are inherently uncertain, decedents are matched between ARD
and SHR sources probabilistically on the basis of measured
similarity between records, within a monte carlo replication
framework. This means that although a given ARD decedent
will be linked to either no SHR decedents or a single SHR
decedent in a particular replicate, the nature of the link for that
ARD decedent (i.e., whether it is linked and, if so, to which
SHR decedent) is allowed to vary across replicates according
to the probabilities associated with particular record pairings,
on the basis of measured similarity between cases. Probabilistic
linking is random in nature and incorporates additional
uncertainty into the process of estimating population coverage.
Monte carlo replication allows for this additional uncertainty
to be captured in variance estimates.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

3.3.5 Capture-recapture analytic approach


Once decedent records have been linked across the ARD and
SHR, capture-recapture methods can be used to estimate the
total law enforcement homicide population and the proportion
of that population covered by the ARD program. This analysis
makes use of the commonly applied Lincoln-Petersen (LP)
estimator, with modifications for bias amelioration necessitated
by the characteristics of the ARD program (Seber 1982).
The simple LP method assumes that law enforcement agencies
that were not recorded in the ARD data did not have any
officer-involved homicides during the observation period.
This approach provides the highest possible ARD coverage
estimate and the lowest estimate of the total population of law
enforcement homicides because it assumes that the data are
representative of agencies with true zero counts rather than
missing responses. Essentially, the simple LP approach presents
a best case scenario for estimating the ARD coverage rate.
The adjusted LP estimate assumes that a portion of the law
enforcement agencies that did not report to the ARD program
did have homicides and that they had, on average, the same
number of officer-involved homicides as law enforcement
agencies that reported data to the program. The adjusted
approach assumes that some law enforcement agencies with
homicides are unrecorded in the ARD data and that these
agencies represent a random sample of the overall set of
agencies with homicides.
3.3.6 Results
Following the simple LP approach, the estimated ARD
coverage of all law enforcement homicides is 49%, with a
95% confidence interval ranging from 47% to 50%. In other
words, the ARD program captured only about half of all
law enforcement homicides in the United States during the
observation period. The adjusted LP approach, the ARD
program is estimated to capture 36% of all law enforcement
homicides, with a 95% confidence interval ranging from 35%
to 38%. Therefore, the coverage of law enforcement homicides
captured in ARD could be as low as 35% or as high as 50%.
These results indicate that the ARD program may provide a
slightly higher coverage of homicides by law enforcement than
the SHR, which is estimated to cover 46% of officer-involved
homicides at best. Further analysis of the data suggests that
about 28% of homicides by U.S. law enforcement agencies are
not captured by either the ARD or SHR programs. Overall,
these findings indicate serious deficiencies in the ability of the
ARD program, as well as the SHR, to capture a universe of
reportable homicides by law enforcement.

15

Despite these deficiencies, coverage of these deaths has


improved since the inception of the program. Differences
in state participation (see table 2-1) and changes in statelevel methodologies contribute to national variation in
annual coverage rates. Despite dips in coverage in 2005 and
2008, results from both the simple LP and the adjusted LP
approaches indicate that ARD coverage of law enforcement
homicides has increased over time, with the most pronounced
improvement occurring in the most recent years of data
collection (figure 3-1).
FIGURE 3-1.
Proportion of law enforcement homicide universe
covered by ARD, by estimation method, 200309 and
2011
Percent

The assessment is not without limitations. Conducting capturerecapture approaches at the state level was determined to be
not statistically sound, given the relatively low number of cases
reported to the ARD and SHR programs during 2011. For
example, four states (Delaware, Rhode Island, Vermont, and
Wyoming) had zero homicides by law enforcement recorded in
either ARD or the SHR. As such, it was not possible to estimate
the coverage rate of homicides by law enforcement using
simple and adjusted LP approaches at the state level.

4. Conclusion
HIGHLIGHTS
Significant challenges exist with the lack of a standardized

mode for data collection, definitions, scope, agency


participation, and the availability of resources.

100

The current findings are consistent with prior studies that

80

No agency adjustment

have shown considerable variation from state to state and


over time in reporting to one national system or another.

60

BJS may provide more prescriptive guidance or requirements

40

regarding identifying cases and collecting data, but should


also incorporate the differences between states in the
availability of reliable sources for information on arrestrelated deaths.

Agency adjustment

20
0

When choosing solutions for a complete and accurate data

2003

2004

2005

2006

2007

2008

2009 2010*

2011

*Data are not shown for 2010 due to an incomplete data collection.
Source: Bureau of Justice Statistics, Arrest-Related Deaths Program, 2011.

During 2003, the first year of data collection, the estimated


coverage of homicides by law enforcement was from about
27% (adjusted LP) to 42% (simple LP). Except for data
collected in 2008, the coverage of law enforcement homicides
captured in ARD improved from 2006 through 2011.12 Using
the simple LP approach, the ARD coverage of officer-involved
homicides grew from 51% in 2007 to 55% in 2009 and 69% in
2011. The adjusted LP estimate (worst case coverage scenario)
for 2011 (59%) indicated better coverage than the simple LP
estimate (best case coverage scenario) for 2009 (55%).
These data indicate that the data collection methods used in
more recent years are associated with better coverage of law
enforcement homicides. This conclusion is supported by the
fact that the adjusted LP approach also demonstrates improved
coverage over time, growing from 40% in 2007 and 2009 to
59% in 2011. The gap between the adjusted and unadjusted
coverage estimates narrowed by 2011, a further indication
of improved methodologies. Even with this improvement,
between 31% and 41% of actual homicides by law enforcement
personnel were not captured in the 2011 ARD data.
12The

2010 ARD data were not included in the capture-recapture work.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

collection, BJS should consider ways to standardize the data


collection process, implement mandatory reporting by law
enforcement agencies, dedicate resources and funding to
support data collection efforts at the state and local levels,
and focus on a more restricted definition and scope of
eligible cases.

The ARD program is designed to be a national census of


civilian deaths when the event causing the death occurs
during an interaction with state or local law enforcement.
Understanding when law enforcement officers are involved
in a situation in which a community member dies during
an arrest is critical to a democratic society (Fyfe 1981, 2002;
McEwen, 1996; Klinger, 2008). BJS designed the ARD data
collection methodology to rely on reporting from independent
state reporting coordinators (SRCs). The manner in which
ARD data are collected varies from state to state and often
depends on the resources available to SRCs. This variability
in approaches has led to questions about whether these data
collection methodologies are capable of capturing the universe
of arrest-related deaths and of law enforcement homicides
in particular.
Previous research has explored the utility of the Supplementary
Homicide Report (SHR), the National Vital Statistics System
(NVSS), law enforcement agency records, media reports,
and the ARD program for providing reliable statistics on the
number of law enforcement homicides in the United States.
This body of literature has generally concluded that there is
no single, reliable system either for reporting or accessing

16

data about arrest-related deaths; no one system consistently


outperforms others. Furthermore, comparison of various data
sources at the national level masks significant variations in
reporting to one source or another at the state and jurisdiction
levels (Loftin et al., 2003; Mumola, 2007; Sherman &
Langworthy, 1979).
The current ARD methodology summarized in the
Methodology section (chapter 2) and the results of the capturerecapture analysis described in the Program Assessment section
(chapter 3) echo the conclusions of this body of research and
highlight the continuing obstacles to collecting valid and
reliable information about arrest-related deaths. At best, the
ARD program captured approximately half of the estimated
law enforcement homicides in the United States during 2003
09 and 2011. Consistent with other research on the utility of
existing sources of data regarding arrest-related deaths, we
found that the current data collection process results in a
significant underestimation and potentially a biased picture of
arrest-related deaths in the United States (Borrego, 2011; Ho et
al., 2009). Some evidence indicates that the systems coverage
has improved over time. Considering the 2011 data alone, the
ARD program was estimated to capture as many as 69% of
all law enforcement homicides in the United States, up from
capturing 42% in 2003.
Although no current national-level system sufficiently
measures all arrest-related deaths in the United States, previous
research has shown that one system captures more cases
than another in some states but not in others (e.g., Loftin et
al., 2003; Mumola, 2007). Some evidence also indicates that
using data from a combination of systems will result in better
coverage. The current analysis found that the ARD program
covered approximately half of the estimated law enforcement
homicides in the United States during 200309 and 2011, but
combining data from the SHR and the ARD increased coverage
to 72%. To improve ARD coverage of arrest-related deaths,
the ARD program might therefore build on two aspects of
the current design to (1) allow some variation in the methods
used to identify cases from state to state and (2) require that
multiple data sources be used to compile information about
arrest-related deaths within each state.
First, variation in the methods used to identify arrest-related
deaths and collect information about them might be warranted
from state to state, depending on the resources available within
that state. The source that captures more arrest-related deaths
varies from state to state, which should not be ignored when
developing a more robust ARD data collection system.
Second, most SRCs employed a methodology that combines
information from multiple sources. Given the limitations
of existing national-, state-, and local-level data systems,
combining data from all available systems should provide
a more comprehensive picture of arrest-related deaths.
Identifying cases from multiple systems will require significant
resources to ensure that reports are not duplicated. Further,

Arrest-Related Deaths Program: Data Quality Profile | March 2015

timeliness and availability are an issue. Many datasets are not


available in the current year and may lag multiple years behind
other sources. Relying on multiple systems would also require
an SRC who has access to and who systematically reviews all
the relevant systems.
A survey that collects information about arrest-related deaths
directly from local law enforcement agencies (rather than
through the SRCs) provides an additional option for improving
ARD program coverage. However, this method would require
significant resources.
To achieve any improvements in coverage and consistency in
reporting, the ARD program needs more prescriptive guidance
or requirements on how arrest-related deaths are identified
and report them to BJS. Currently, BJS provides definitions,
examples, and general guidance on how SRCs might go about
identifying arrest-related deaths and reporting information
about them, but it does not require that SRCs follow a specific
methodology. Furthermore, case identification and data
collection methodology in each state is largely determined and
driven by the individual SRCs own experience and available
resources. Applying consistent definitions and systematic
means of identifying deaths and collecting information about
them from all appropriate sources in the jurisdiction is critical
to improving ARD program coverage.
The scope of the current ARD program may also contribute
to underreporting. The analyses reported in the Program
Assessment (chapter 3) and cases found in the SHR and NVSS
are limited to law enforcement homicides. The ARD program
is designed to capture all arrest-related deaths, including
those due to accidents, drug overdoses, and natural causes.
It is the only system that attempts to collect these data, and it
could provide critical information on cases that are missing
or underreported in other sources, such as deaths following
the use of tactics commonly referred to as less than lethal,
including Tasers and restraint procedures. However, arrestrelated deaths other than law enforcement homicides can
be difficult for SRCs to define and identify from the sources
available to them. Furthermore, the more clearly defined law
enforcement homicide type of arrest-related death still suffers
from significant undercoverage, as shown in the capturerecapture analyses presented in chapter 3. Improvements to the
ARD program might be realized by first focusing solely on law
enforcement homicides. After the data collection methodology
has been defined, made routine, and shown to result in
sufficient coverage for law enforcement homicides, a more
expansive definition of arrest-related deaths might be explored.
BJS should consider ways to standardize the data collection
methodologies to improve the reliability and validity of
capturing eligible cases and overall data quality.

17

References
Borrego, A. (2011). Arrest-related deaths in the United States:
An assessment of the current measurement. Masters degree
thesis, Arizona State University. http://repository.asu.edu/
attachments/56537/content/Borrego_asu_0010N_10572.pdf
Burch, A. (2011). Arrest-related deaths, 20032009statistical
tables. (NCJ 235385). Washington, DC: Bureau of Justice
Statistics. Retrieved from http://www.bjs.gov/content/pub/pdf/
ard0309st.pdf
Fyfe, J. (1981). Observations on police deadly force. Crime &
Delinquency, 27, 376389.
Fyfe, J. (2002). Too many missing cases: Holes in our
knowledge about police use of force. Justice Research and Policy,
4, 87102.
Ho, J.D., Heegaard, W.G., Dawes, D.M., Natarajan, S., Reardon,
R.F., & Miner, J.R. (2009). Unexpected arrest-related deaths
in America: 12 months of open source surveillance. Western
Journal of Emergency Medicine, 10(2), 6873.
Klinger, D.A. (2008). On the importance of sound measures
of forceful police actions. Criminology and Public Policy, 7,
605617.

Loftin, C., Wiersema, B., McDowall, D., & Dobrin, A. (2003).


Underreporting of justifiable homicides committed by police
officers in the United States, 19761998. American Journal of
Public Health, 93(7), 11171121.
McEwen, T. (1996). National data collection on police use
of force. (NCJ 160113). Washington, DC: Bureau of Justice
Statistics. Retrieved from http://www.bjs.gov/content/pub/pdf/
ndcopuof.pdf
Mumola, C.J. (2007). Arrest-related deaths in the United
States, 20032005. (NCJ 219534). Washington, DC: Bureau of
Justice Statistics. Retrieved from http://bjs.ojp.usdoj.gov/index.
cfm?ty=pbdetail&iid=379
Noonan. M.E., & Ginder, S. (2013). Mortality in local jails
and state prisons, 20002011statistical tables. (NCJ 242186).
Washington, DC: Bureau of Justice Statistics. Retrieved from
http://www.bjs.gov/content/pub/pdf/mljsp0011.pdf
Seber, G.A.F. 1982. The estimation of animal abundance and
related parameters. Macmillan Publishing, New York, New
York, USA.
Sherman, L.W., & Langworthy, R.H. (1979). Measuring
homicide by police officers. Journal of Criminal Law and
Criminology, 70, 546560.

Klinger, D.A. (2012). On the problems and promise of research


on lethal police violence: A research note. Homicide Studies, 16,
7896.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

18

Appendix A
CJ-11 and CJ-11A data collection instruments and
revisions, 200313
Tracking deaths in the process of arrest by state and local
law enforcement presented a wide array of circumstances that
are difficult to capture. To establish a standardized reporting
form and process, in 2002 the Bureau of Justice Statistics (BJS)
began working with states and law enforcement professional
organizations to develop preliminary questionnaires. The
process yielded two data collection forms, the CJ-11 Quarterly
Summary of Deaths in Law Enforcement Custody (commonly
called the summary of incidents form) and the CJ-11A
Addendum, Law Enforcement Custodial Death Report
(commonly called the incident report) (http://www.bjs.gov/
index.cfm?ty=dcdetail&iid=428#Questionnaires).
The CJ-11 summary of incidents form was to serve as a
quarterly summary of arrest-related deaths occurring in each
state disaggregated by sex. In addition, the CJ-11 provided
instructions for submitting data, examples of reportable deaths,
and descriptions of the types of deaths that should be excluded
from the data collection. State reporting coordinators (SRCs)
were asked to submit a completed CJ-11 form indicating
their statewide count of reportable deaths for each quarter of
the calendar year. SRCs were instructed to complete a CJ-11
form with a count of zero if no arrest-related deaths were
identified in the state. BJS used these forms to determine state
participation in the Arrest-Related Deaths (ARD) program.
SRCs were instructed to report zero counts to distinguish
between the measured absence of an identified death and
missing data.
On the CJ-11A incident report, SRCs were asked to describe
the circumstances surrounding each arrest-related death they
listed on the summary of incidents form, detailing one death
per CJ-11A form. The 2003 CJ-11A form comprised 13 items
to be completed for each death and an additional set of 5 items
for those deaths that occurred before booking (A1A5) or
a set of 4 items for deaths that occurred after booking (B1
B4). It included items about the decedent, such as his or her
name, date of birth, sex, and race/Hispanic origin. It featured
questions regarding the manner and cause of death and asked
whether the cause of death had been determined by a medical
examiner or coroner. The CJ-11A also captured information
about the decedents interaction with law enforcement (i.e., the
name and originating identification [ORI] number of the law
enforcement agency involved, location of the event, whether
charges had been filed against the decedent, and the most
serious offense with which the decedent was being charged at
the time of death) and whether the death occurred before or
after booking.
For arrest-related deaths that occur before booking, SRCs were
instructed to complete Section A of the CJ-11A form. Section
A included additional items about the incident between law
enforcement and the decedent, including whether the decedent

Arrest-Related Deaths Program: Data Quality Profile | March 2015

sustained fatal injuries during the event, and if so, how the
injuries were sustained; whether the decedent was restrained in
the time leading up to the death, and if so, the type of restraints
devices were used by law enforcement; and the decedents
behavior during the interaction with law enforcement (i.e.,
appeared intoxicated, threatened officers, resisted being
handcuffed or arrested, tried to escape or flee, physically fight
with the officers involved, or used a weapon to threaten or
assault officers). Section A also included items regarding the
type of weapon that caused the death and the type of location
where the decedent expired (e.g., crime or arrest scene,
medical facility, or en route to booking center).
SRCs were instructed to complete Section B of the CJ-11A
form for instances in which the death occurred after booking
while the decedent was being held in a temporary holding
facility (but before arraignment transfer to a long-term
correctional facility). Section B included items on when the
decedent was booked into the law enforcement facility (i.e.,
date and time of entry) and the decedents condition at the
time of booking (e.g., appeared intoxicated, exhibited mental
health problems, or exhibited medical problems). Section B
also included information about who caused the death (i.e.,
deceased, other detainees, or law enforcement or correctional
staff) for deaths attributed to homicide and accidental injury
and the means of death (e.g., firearm, blunt instrument,
cutting instrument, strangulation, or intoxication) for
nonnatural deaths.

Revisions to the instruments


BJS has continued to evaluate the CJ-11 and CJ-11A since
the instruments were implemented in 2003. The content of
the CJ-11 summary form has remained relatively stable. A
national listing of SRCs was added to the CJ-11 in 2005 and
remained until 2013, when it was transferred to the CJ-11A
incident report. The CJ-11 summary report was updated in
2013 to include revised reporting instructions and the option
to complete the form as an annual summary (in addition to the
continued use of reporting quarterly information). In addition,
SRCs were no longer required to disaggregate state-level
summary counts by sex.
The CJ-11A incident report has undergone several revisions
since the original version. The first revision to the instrument
was in 2007, which were minor and included (a)removing stun
gun or Taser as a response to What type of weapon(s) caused
the death? and (b) adding conducted energy device (e.g.,
Taser or stun-gun) as a response to the type of restraint device
used in the time leading up to the death or the events causing
the death. In 2008, a response of No medical/mental health
assistance call was added as a response to Had charges been
filed against the deceased at the time of death? In addition, a
response of Pepper spray, mace was added as a response to
the type of restraint device used in the time leading up to the
death or the events causing the death. Finally, conducted energy
device was added back to the responses categories for What
type of weapon(s) caused the death?

19

More significant chances to the CJ-11A instrument occurred in


2009, when the title of the instrument changed from Deaths
in Custody Law Enforcement Custodial Death Report to the
Arrest-Related Death Report. This change was made to help
clear up confusion about the scope of the program. Even among
law enforcement agencies providing data to the ARD program,
the deaths in custody terminology led some to assume
officer-involved shootings are not reportable because in the vast
majority of cases, the decedent is not in the physical custody
of law enforcement. The shift to the term arrest-related was
intended to describe the wider scope of deaths reportable to
the program.
In addition to the title change, the 2009 CJ-11A was
structurally changed to remove the designation between
sub-sections A and B. Before 2009, agencies completed either
Section A or Section B depending on whether the death
occurred before or after booking. Beginning with the 2009
version of the CJ-11A, respondents were asked to complete
the items formerly in Section A for all reportable deaths. If
the death occurred after to booking, respondents were asked
to also complete all items formerly in Section A in addition
to the items formerly found in Section B. While this change
did not affect reporting for deaths occurring before booking,
information on events that took place during the arrest were
now required for deaths occurring post-booking.
Other changes to the 2009 CJ-11A included adding responses
to preexisting items and adding an item on the decedents use
of weapons during the event. A notable addition to the 2009
instrument was the inclusion of exhibit any mental health
problem as a response to indicate the decedents behavior
during the incident. Other additions to the form included a
response category for injuries inflicted by law enforcement
officers during transit or booking and widening the scope
of the decedents behavior during the incident to include
certain attempted actions. For example, previous versions of
the instrument asked respondents to indicate whether the
decedent grabbed, hit, or fought with law enforcement officers,
where the 2009 version sought to also include attempted
physical altercations with law enforcement.
While the majority of the content remained the same,
modifications were made to the 2010 CJ-11A. Some changes
included a shift in focus from obtaining knowledge about
whether or not a medical examiner or coroner conducted
an evaluation to determine the official cause of death, to
whether or not the information about the decedents manner
and cause of death supplied on the CJ-11A were determined
from documents created by a medical examiner or coroner. In
addition, alcohol/drug intoxication was added as a response
to whether the deceased died from a medical condition or
from injuries sustained during the arrest process. Other
changes included adding firearm discharge as a device used
by law enforcement during the arrest process and adding dont
know as a response option for questions.
In 2010, BJS held a meeting of experts to review of the ARD
collection and solicit feedback on the data CJ-11A form.
Participants in this meeting included law enforcement
Arrest-Related Deaths Program: Data Quality Profile | March 2015

personnel, medical examiners and coroners, SRCs, and


researchers in the law enforcement and public health fields.
The ARD CJ-11A incident report was revised in 2012 in light
of comments received from the meeting. The revised 2013
CJ-11A instrument was pilot tested in June 2012 with nine of
the programs current SRCs.
In June 2012, BJS asked nine of the programs current SRCs to
assess the revised instrument and related instructions in terms
of substance, breadth, and clarity. Specifically, BJS asked the
SRCs to evaluate whether the questionnaire items enhanced
the quality, utility, and precision of the information to be
collected and to note any items that were unclear, concerns
with respect to the availability of the information requested,
and response burden. Lastly, SRCs were asked to complete the
revised version of the CJ-11A form for a death record that was
previously submitted to the ARD program.
BJS removed items that required respondents to speculate about
outcomes or that relied on respondents opinions or perceptions.
For example, BJS removed items related to alleged criminal
involvement in the events leading up to the death, removed
an item that had asked respondents to indicate whether the
decedent appeared intoxicated or exhibited mental health
problems. BJS removed other items that were repetitive or could
be determined from responses to other items. For example,
Item 13 in the old instrument (Did the deceased die from a
medical condition, injuries sustained during the arrest process,
or alcohol/drug intoxication?) was omitted because this
information could be determined from information provided in
response to tem 9 (What was the cause of death?). In addition,
four questions about deaths occurring at a booking center or
police lockup were omitted because the items either asked for
speculative information or were redundant.
Before revision, the CJ-11A did not have reporting instructions
or a national list of SRCs, both of which were included only
on the CJ-11 summary of incidents form. The revised CJ-11A
instrument includes a description of what deaths should be
reported, contact information for assistance in completing the
form, instructions about how to submit the completed form,
and contact information for each SRC. The revised CJ-11A also
includes a data supplied by section to assist SRCs in tracking
where the information about each death is coming from.
BJS changed the order of items on the revised CJ-11A to
improve the flow of the instrument. Items were reorganized
into groups that reflect characteristics of the incident,
characteristics of the decedent and actions the decedent took
during the interaction, actions of law enforcement personnel
during the interaction, and characteristics of the death. The
structure of questions changed from a Mark all that apply
format to a yes/no format to improve the quality of the data.
Forcing those filling out the form to reply to each response
category lessens the risk that people will skip over responses
that should have been marked. In addition, after the pilot test,
BJS removed an item regarding whether the decedent had a
history of mental illness out of concern that the information
obtained from this question would not be reliable.

20

Appendix B
Description of the internet-based data collection tool
A fully integrated web portal was to enhance the ArrestRelated Deaths (ARD) program data collection process by
centralizing efforts to communicate the purpose and goals
of the program. The web portal (www.bjsard.org) consists
of both a public website for communicating information
about the ARD program and a private online data entry and
case management system that is available to state reporting
coordinators (SRCs). The ARD portal provides users with
a centralized location for accessing program information,
such as background information, a flier for distribution
to law enforcement agencies, an explanation of programrelated terms, responses to frequently asked questions,
program policies and announcements, arrest-related deaths
publications, CJ-11 and CJ-11A data collection instruments,
and on-demand training webinars.
The main benefit of the web portal is that it reduces the time
it takes to enter ARD data and gives the ARD project team
greater control over the reporting process. The ARD portal
contains a private and secure section where SRCs can submit
their CJ-11 and CJ-11A data. It allows SRCs to directly enter
CJ-11A information on a web-based reporting tool (WRT)
rather than having to obtain completed forms and transmit
them to the ARD program staff. Before the launch of the WRT,
ARD program staff would receive data from SRCs and enter it
into a national database.
The WRT ensures that SRCs are using the correct CJ-11A form
based on the year of the incident, which is a level of control
that cannot be obtained when using hardcopy forms. For
example, respondents in some states routinely report data on
older, outdated CJ-11A forms. Requiring SRCs to use the WRT
minimizes their use of outdated forms because the online
forms are updated automatically.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

When using the WRT, SRCs can monitor, review, edit, and
approve their submissions before the data are deemed final in
the system, which is useful when SRCs receive updated case
information. SRCs may have to track an arrest-related death
for a long time before they have all the relevant information
to officially submit the case. Once a case is entered, it remains
in the users list of cases and the SRC can add to the record as
information becomes available. The WRT facilitates cleaner
data entry because it increases efficiency in data collection by
reducing the need for post-entry editing and review.
The WRT contains a back-of-the-house validation code to
ensure that the initial submission is as accurate as possible.
Specifically, as the SRC enters a case the WRT will notify the
SRC of inconsistencies in the data and require that the SRC
address them before final submission. For example, if the SRC
indicates that a weapon was used but did not specify what type
of weapon, the WRT will display a message telling the user to
specify the type of weapon before the WRT will move to the
next item. The WRT also allows help text to be embedded next
to each question to guide the SRCs when they are filling out
the form.
The WRTs audit function tracks when changes are made to
submitted data and who made the changes. It can also track
information such as the time lapse time between initial entry
of the case and completed verification. This audit history
enables the Bureau of Justice Statistics to electronically
compare an original data submission to the final case record.
The WRT also improves data security. Submitting the ARD
data via the WRT increases the security of that data by
encrypting the web session using Secure Socket Layers (SSL).
SSL technology is an improvement over the former method of
sending files as email attachments that may or may not have
been encrypted. Logistical problems, such as mailbox size
limitations, are also reduced. SRCs also can upload data files to
the ARD web portal in a secure fashion.

21

Appendix C
Description of BJSs data verification process
State reporting coordinators (SRCs) submit their states
incident-level data (CJ-11A forms) to the Bureau of Justice
Statistics (BJS) for national-level analysis. BJS staff review them
for completeness and internal consistency before entering
them into a national database.
If a CJ-11A report has missing information and responses that
appear to be inconsistent or inaccurate, BJS staff take steps to
complete or reconcile the information. Records are considered
inconsistent or inaccurate if a response to one CJ-11A item
contradicts a response to another item on the form. For
example, a CJ-11A form might have a cause of death listed as
gunfire but indicate that no weapons were used during the
incident.
In other cases, discrepancies in reporting may be identified
across an SRCs records, suggesting that coding procedures
were not uniform. For example, deaths in which the manner
was coded as accidental alcohol/drug intoxication and the
cause of death coded as overdose could have a variety of
responses to Did the deceased die from a medical condition
or from injuries sustained during the arrest process?
Those responses might include (1) medical condition only,
(2) injuries only, (3) both medical condition and injuries, and
(4) dont know.
If missing information and apparent inconsistences are
discovered, BJS staff try to obtain additional case information
to supplement the data, both by filling in missing data and
verifying the validity of responses. Staff use the decedents
name and the name of the law enforcement agency involved
in the death to conduct open-source searches. These searches
yield information from media reports, law enforcement
press releases, autopsy reports or death evaluations, and legal
proceedings. Staff use these sources to supplement missing
data and to correct inconsistent or contradictory information
on CJ-11A forms. All modifications to submitted CJ-11A
forms are noted in a Microsoft Word document, along
with explanations for changes, and then sent to SRCs for
verification.
During this verification process, BJS prepares a status report.
The status report (1) serves as a receipt listing every name
recorded in the collection for each SRC, (2) indicates whether
each case submitted is complete or incomplete, (3) identifies
cases that are out-of-scope, and (4) identifies cases that are
within-scope that the SRC did not submit.
Status reports contain the names of all decedents recorded
in each annual collection to confirm that BJS received all the
data the SRCs submitted and ensure that BJS staff and the
SRCs agree on who is included in the data. In addition, this
list is an avenue through which BJS educates SRCs about any
data submitted that were out-of-scope and data that were not
identified through the SRCs methodology.

Arrest-Related Deaths Program: Data Quality Profile | March 2015

These reports also indicate whether BJS considers each case to


be complete or incomplete. BJS records a case as complete if
every item on the CJ-11A has a response and if the responses
are internally consistent. If all responses are entered on the
CJ-11A, but one or more responses are not consistent (or
do not make logical sense) when considered with other
item responses, the record is classified as incomplete. If BJS
determines a record is complete, the status report will indicate
(1) the name of decedent, (2) the date of death, and (3) that the
record is complete and data will be recorded in the national
database as originally submitted.
BJS considers records as incomplete if there are missing data or
if there are inconsistences across the submitted responses. In
addition to recording the decedents name and date of death,
the status report will also indicate the specific items that have
missing or inconsistent data. In some instances, BJS will mark
a record as incomplete to note a change made to the data. BJS
makes an effort to explain why changes were made to data to
improve the standardization and quality of future data.
Another example of missing data occurs when CJ-11A forms
lack the manner and cause of death. This information is often
left out because the autopsy report or death certificate was
not available when the record was submitted. In these cases,
BJS follows up with the SRCs at the end of the data collection
cycle to determine if updated information is available.
Cases are recorded as incomplete if responses seem logically
inconsistent.
The status reports also indicate records that were submitted
by the SRC and excluded from the collection by BJS. The most
common reason cases are excluded is because the record has
already been recorded in the Deaths in Custody Reporting
Program (DCRP) Jails data collection. BJS checks all ArrestRelated Deaths (ARD) program records against other DCRP
collections to identify duplication across programs. If a death is
reported to both ARD and one of the correctional components
of the DCRP, the case remains in the DCRP Jails or Prisons
collections and is excluded from ARD. BJS uses a hierarchy
which specifies that duplicate cases submitted to the DCRP
should be retained as correctional (i.e., DCRP Jails or Prisons)
cases because presence in one of the correctional components
indicates that custody of the decedent was transferred from law
enforcement at some point.
Lastly, the status report notifies SRCs of reportable deaths
identified by ARD program staff that were not included in
the SRCs submission. In the process of obtaining additional
information on the cases submitted, BJS staff may identify
arrest-related deaths referenced in the open-source materials.
For example, a journalistic account of the death of one
arrestee might note that it was the law enforcement agencys
third officer-involved death of the year and recount the
circumstances surrounding the two previous deaths. BJS staff
would then check the records submitted for the two additional
deaths. If an identified death had been previously submitted
by the SRC, BJS staff would not take additional steps beyond
verifying that the information provided on the CJ-11A

22

was correct. If an identified death has not been previously


submitted, staff will note the name of the decedent, the law
enforcement agency involved with the death, and the date
or month of death on the status report. When possible, BJS
provides SRCs with additional information about the cases
the SRC missed, such as a copy of the law enforcement press
release or legal proceedings. SRCs are asked to follow up on
newly identified cases to confirm that they are eligible for
inclusion in the program, and if eligible, complete a CJ-11A
incident report.
BJS follows up with the SRC to improve the quality of the
data submitted and the standardization of future program
data. Concentrated follow-up with SRCs regarding the quality
of the data submitted occurs during the summer. SRCs are
reminded to submit their responses to issues identified on the
status report and to submit any additional cases they may have
identified after their initial submission of data. All followup contact with the SRCs typically occurs via email. SRCs
may respond to status reports in whatever format is easiest
for them, and ARD program staff will modify the previously
submitted CJ-11A forms as needed. Follow-up contact for
SRCs who did not submit their states ARD data begins 60 days
after the end of the year. These respondents receive emails
asking them to submit their data from the prior calendar year
as soon as possible.
The ARD data verification process was automated in 2012.
ARD staff developed a codebook for standardizing state-level
data submissions, which was tested for inter-rater reliability
before being implemented. A SAS-based programming code
was written to conduct missing, error, range, and consistency
checks on data submitted by SRCs. Beginning with the 2011
collection, data were reviewed and a machine-edit process
conducted a random review of records to check for data entry
errors. This process is based on a series of checks, guided by
the ARD codebook, that review and flag potentially erroneous
data. Flagged cases are then exported to an Excel file for review
by BJS staff.

Paradata
In preparing the status reports, BJS identifies common sources
of reporting errors both across and within SRCs reports. BJS
began documenting reporting errors in 2010 while processing
the 200709 data to better understand data quality issues.
BJS used the 200709 data to create state-level narratives that
indicated the number of (1) records submitted by the SRC,
(2) submitted out-of-scope records excluded during BJSs
verification procedures, and (3) within-scope records identified
by BJS independent of what was submitted by SRCs.
In addition, BJS categorized all identified arrest-related
deaths by the manner and cause of death (e.g., homicide
by law enforcementgunshot, or suicideasphyxiation),
and whether specific records had information missing or
inconsistent responses (and the CJ-11A item number in
which problems were identified). These SRC assessments also

Arrest-Related Deaths Program: Data Quality Profile | March 2015

indicated the manner and cause of deaths identified solely by


BJS through open-source searches to determine whether there
was a pattern in the types of cases BJS was able to identify
through open-source searches. In addition, BJS staff collected
information on whether the case could be found online using
the data the SRC submitted, through searches of the decedents
name and the name of the law enforcement agency involved
with the death.
The results of these initial assessments indicated that SRCs
were coding data inconsistently both within and across
SRC agencies. Inconsistencies in responses were related to
more complex manners and causes of death. Data contained
discrepancies when the manner and cause of death appeared
to contradict each other. For example, a medical examiner or
coroner might rule the manner of death as a homicide and
list the cause of death as cardiac arresta cause that might
intuitively be associated with illness or natural causes. In
these instances, SRCs indicated problems with completing
CJ-11A items that pertained to whether the deceased died
from a medical condition or from injuries sustained during
the event, how the injuries were sustained (e.g., inflicted by law
enforcement, self-inflictedaccidental, or not applicable), and
whether a weapon caused the death. These types of deaths were
often associated with the use of a stun gun or Taser during
the interaction with law enforcement personnel or events that
involved foot pursuits and physical struggles or altercations.
Other CJ-11A discrepancies were related to items indicating
whether or not charges had been filed against the deceased
at the time of death and, if so, which offenses were the most
serious offenses being charged. Variation in criminal codes
across states contributed to inconsistencies in reporting
information. However, more significant issues were caused by
inconsistent interpretations of the items on the CJ-11A form
related to potential charges. Some CJ-11A records contained
information about the reason for the interaction with law
enforcement, such as a citizen-initiated call for service or
an unlawful action observed by law enforcement. Other
records contained only information about illegal activities
that occurred during the interaction and provided no insight
as to why or how the decedent came into contact with law
enforcement personnel.
Although these records may have indicated the initial reason
for the interaction (which may have not been criminal), many
failed to capture criminal behaviors that transpired during
the event. The most common source of error was the failure
to include assaults against law enforcement personnel during
the interaction with the decedent. During BJSs verification
process, staff identified a number of records that did not
include actions taken by the decedent against law enforcement
personnel such as physical altercations, assaults with weapons,
attempted murders, and homicides of law enforcement.
SRCs were inconsistent both within and across states when
completing information related to expected charges. This
finding was especially pronounced when the initial reason for
the law enforcementdecedent interaction was attributed to a

23

request for service regarding mental health assistance.


In looking at the 200709 data, it became clear that SRCs were
not applying consistent program definitions to cases when
determining whether they were eligible for reporting. SRCs
were reporting deaths outside of the ARD program scope,
as well as failing to identify eligible deaths. In addition, the
analysis of the paradata indicated that specific CJ-11A items
were more problematic than others, and that certain types of
arrest-related deaths also contributed higher levels of reporting
error.
BJS took additional steps to quantify reporting error in 2010.
ARD staff began collecting information about the records
submitted by SRCs that included the date each CJ-11A was
received by BJS, the date each form was entered, and the entity
that identified the death (i.e., solely identified by the SRC,
independently identified by both the SRC and BJS, or solely
identified by BJS). In 2011, ARD staff expanded the paradata
to include the date and reason for contacting the SRC, the date
when a change was made to the data submitted, the name of
the person who modified the data submitted, and the reason
the data were changed.

BJS identification methods


Independent of the SRCs efforts to identify ARD-reportable
deaths, BJS began to monitor open sources for arrest-related
deaths in January 2010. BJS staff identify within-scope deaths
by using Google Alerts and by monitoring websites. Google
Alerts are email updates of the latest relevant Google results
based on queries established by the user to monitor the web
for specific terms. BJS established a Google Gmail account for
the ARD program and specified the words that Google should
look for. The Google Alert system then searches media outlets
nationwide and compiles relevant articles into an email for
each specified term. The ARD email account receives daily
emails that alert BJS when the specified search terms are used
in the title of an article or news story.
Staff look through the alert emails to determine if an arrestrelated death was captured in an article identified by Google.
Each alert email may contain a series of articles, some
containing accounts of within scope deaths. For example, an
alert for the term standoff can yield 30 articles; of those,
three articles may contain accounts of an arrest-related death.
If an arrest-related death is identified through the Google
Alerts, BJS will save the media accounts of the incident and
forward the information to the SRC in the state where the
death occurred. At that point, the SRC takes responsibility
for verifying that the death is reportable and for obtaining a
complete CJ-11A incident report. Depending on the timing

Arrest-Related Deaths Program: Data Quality Profile | March 2015

of the arrest-related death incident and on the SRCs data


submission schedule, CJ-11As for deaths identified by BJS may
be received at the same time as the other arrest-related deaths
in the state or the information may be received independently
at a later point in time.
BJS also monitors websites to identify deaths. These websites
contain information supplied by the public about deaths that
occur during interactions with law enforcement personnel.
BJS monitors TNT Truth Not Tasers (http://truthnottasers.
blogspot.com/), a website that lists the names of individuals
that died after an alleged shock from a conducted energy
device. One of its pages, A list of the dead, contains the
decedents names, age, date of death, and city of death. BJS
compares this list to data submitted by SRCs to determine
whether there are names on the TNT list that were not
reported to the ARD program. If a previously unreported
death is identified using the TNT list, BJS will seek additional
information about the incident to better assess whether the
death is reportable to program. Similar to the Google Alerts
procedures, staff will forward information about the event to
SRCs and the SRC will take responsibility for verifying the
inclusion of the death in the program and obtaining a complete
CJ-11A incident report.
Other websites used by BJS to identify arrest-related deaths
include the Wikipedia page List of killings by law enforcement
officers in the United States (http://en.wikipedia.org/wiki/
List_of_killings_by_law_enforcement_officers_in_the_
United_States,_August_2012). This website contains lists of
people killed by nonmilitary law enforcement officers, whether
in the line of duty or not, and regardless of reason or method.
The Wikipedia list contains the decedents name, age, date of
death, city and state of death, and a brief description of the
event. The website also links a media account of the event as
the reference for the information provided. BJS also monitors
Cops Shooting People (http://copsshootingpeople.wordpress.
com/) and Civilians Down (http://civiliansdown.com/site/
#sthash.Er0FGjOZ.dpbs).
BJS also routinely monitors the Officer Down Memorial Page
(ODMP), a web-based list of law enforcement officers who
have died. Although the deaths of law enforcement officers
is outside of the scope of the ARD program, the ODMP
website is monitored to identify instances in which civilians
and officers both die during an interaction. For example, the
ODMP website indicates that 12 officers died as a result of
vehicular assault during 2011. The perpetrators of two of the
assaults also died during the interaction and were within scope
of the ARD program.

24

The Bureau of Justice Statistics of the U.S. Department of Justice is the


principal federal agency responsible for measuring crime, criminal
victimization, criminal offenders, victims of crime, correlates of crime,
and the operation of criminal and civil justice systems at the federal, state,
tribal, and local levels. BJS collects, analyzes, and disseminates reliable and
valid statistics on crime and justice systems in the United States, supports
improvements to state and local criminal justice information systems, and
participates with national and international organizations to develop and
recommend national standards for justice statistics. William J. Sabol is
acting director.
This report was written by Michael Planty and Andrea M. Burch, Bureau of
Justice Statistics, and Duren Banks, Lance Couzens, Caroline Blanton, and
Devon Cribb, RTI International. Lynn Langton provided statistical review and
verification of the report.
Lynne McConnell and Jill Thomas edited the report. Barbara Quinn produced
the report.
March 2015, NCJ 248544

Office of Justice Programs


Innovation Partnerships Safer Neighborhoods
www.ojp.usdoj.gov

Arrest-Related Deaths
Program Assessment
Technical Report
Duren Banks
Lance Couzens
Caroline Blanton
Devon Cribb
RTI International

NCJ 248543
March 2015

Contents
Executive summary .....................................................................................................................1
1 Background ..............................................................................................................................3
2 Methodology ............................................................................................................................4
2.1 Probabilistic case-level linking ......................................................................................4
2.1.1 Calculation of similarity scores .........................................................................5
2.1.2 Thresholds for matching ....................................................................................6
2.1.3 Linking algorithm ..............................................................................................6
2.1.4 Monte carlo replication ......................................................................................8
2.2 Capture-recapture analytic approach .............................................................................8
2.2.1 Simple poststratified Lincoln-Petersen approach ..............................................9
2.2.1.1 Assumptions........................................................................................9
2.2.1.2 Strengths and limitations...................................................................10
2.2.2 Poststratified Lincoln-Petersen approach with constant ratio
adjustment for unobserved agencies .............................................................10
2.2.2.1 Assumptions......................................................................................11
2.2.2.2 Strengths and limitations...................................................................11
2.2.3 Data assumptions and coverage estimate implications ....................................12
3 Findings ..................................................................................................................................13
3.1 Arrest-related death coverage ......................................................................................13
3.2 Variability in ARD methodology across states ............................................................23
3.2.1 ARD law enforcement homicide reporting methodology................................27
3.2.2 Location of SRC ..............................................................................................29
3.3 Number of years at least one death was reported to ARD and SHR ...........................31
4 Conclusions and recommendations ........................................................................................32
4.1 Relative strengths and weaknesses of the ARD program methodology ......................32
4.1.1 Types of cases that were covered well by the ARD program ..........................32
4.1.2 Identification approaches that are associated with better coverage .................32
4.2 Recommendations ........................................................................................................33
4.3 Study limitations ..........................................................................................................34
References .................................................................................................................................37
Appendix A: Similarity scoring for ARD/SHR matching.........................................................38
Introduction ........................................................................................................................38
iii

Continuous variables ..........................................................................................................38


Categorical variables ..........................................................................................................39
Calculating the overall score ..............................................................................................40
Appendix B: Linking algorithm ................................................................................................41
Estimate the agency population size and agency undercoverage adjustment factor .........41
Classify cases as matchable or unmatchable .....................................................................41
Probabilistic linking between ARD and SHR ....................................................................42
Recombine cases ................................................................................................................43
Stratify by individual-level characteristics ........................................................................43
Estimate population size and variance by stratum .............................................................43
Estimate population size ....................................................................................................44
Adjust population size for agency undercoverage .............................................................44
Estimate ARD coverage .....................................................................................................44
Replication .........................................................................................................................44
Appendix C: Race/Hispanic origin coding ................................................................................46
Hispanic origin in the SHR ................................................................................................46

List of tables
Table 1: ARD and SHR law enforcement homicides, by match outcome, 200309
and 2011 ................................................................................................................ 16
Table 2: ARD law enforcement homicides, by match outcome, 200309 and 2011 .............. 16
Table 3: ARD and SHR law enforcement homicides, by matching variable and match
outcome, 200309 and 2011 ................................................................................. 19
Table 4: ARD law enforcement homicides, by matching variable and match outcome,
200309 and 2011 ................................................................................................. 20
Table 5: ARD law enforcement homicide decedent characteristics, by source, 2003
09 and 2011 ........................................................................................................... 22
Table 6: ARD coverage estimates and confidence intervals, by estimation approach,
200309 and 2011 ................................................................................................. 23
Table 7: ARD law enforcement homicides, by region, state, and source, 2011 ..................... 25
Table 7: Primary methodology for identifying arrest-related deaths, by state, 2011 .............. 27
Table 8: SRC agency location, 2011 ....................................................................................... 29
Table 9: Number of years at least one death was reported to ARD and SHR......................... 31

iv

List of figures
Figure 1: ARD and SHR coverage and overlap of the universe of law enforcement
homicides in the United States, with no agency adjustment, 200309 and
2011....................................................................................................................... 14
Figure 2: Estimated proportion of law enforcement homicide universe covered by
each source, with no agency adjustment, 200309 and 2011 ............................... 15
Figure 3: Percent of observed ARD law enforcement homicides, by match score,
200309 and 2011 ................................................................................................. 17
Figure 4: Percent of law enforcement homicide universe covered by ARD, by
estimation method, 200309 and 2011 ................................................................. 24
Figure 5: Law enforcement homicides, by source and primary reporting method,
2011....................................................................................................................... 28
Figure 6: Law enforcement homicides, by source and SRC location, 2011 ........................... 30
Figure 7: ACS 5-year Hispanic origin, by county ................................................................... 48

Executive summary
The Bureau of Justice Statistics (BJS) designed the Arrest-Related Deaths (ARD) program to be
a census of all deaths that occur during the process of arrest in the United States. The manner in
which these data were collected varied from state to state, and often depended on the data
systems available to the state reporting coordinators (SRCs) responsible for data collection
throughout the state, the involvement of local law enforcement agencies or medical
examiners/coroners offices, and other support that the SRC may have had to conduct the data
collection. This variability in approach has led to questions about whether these data collection
methods were capable of capturing the universe of arrest-related deaths and law enforcement
homicides in particular. BJS requested RTI International to conduct an assessment of the ARD
program to evaluate (1) the coverage of the program in comparison to Supplementary Homicide
Reports (SHRs) maintained by the Federal Bureau of Investigation (FBI) and (2) various aspects
of the current program methodology. The coverage assessment matched law enforcement
homicides captured by the ARD program to those found in the SHR justifiable homicide file,
followed by a capture-recapture analysis to provide information on the scope and characteristics
of cases eligible for inclusion in the ARD program that are captured in one or both of these data
systems. The ARD law enforcement homicides and SHR justifiable homicide files are similar;
however, some law enforcement homicides that were not classified as justifiable are not
identified in the SHR.
RTI calculated the size of the law enforcement homicide population in the United States and the
ARD program coverage using two methods to estimate the lower and upper bounds of ARD
coverage. We found that over the study period from 2003 through 2009 and 2011, the ARD
program captured, at best, 49% of all law enforcement homicides in the United States. The lower
bound of ARD program coverage was estimated to be 36%. These findings indicate that the
current ARD program methodology does not allow a census of all law enforcement homicides in
the United States.
The ARD program captured approximately 49% of law enforcement homicides, while the SHR
captured 46%. An estimated 28% of the law enforcement homicides in the United States are not
captured by either system. However, the methodology for identifying ARD cases has changed
over the observation period. In 2011, the ARD program was estimated to cover between 59%
and 69% of all law enforcement homicides in the United States, depending on the estimation
method used. While this coverage estimate still does not result in a census, it does suggest
improvements over time in the overall approach to identifying law enforcement homicides and
reporting them to the ARD program.
We found considerable variability between states in the proportion of law enforcement
homicides that are reported to the ARD program only, the SHR only, or to both sources in 2011.
Twelve states reported only to the ARD program in 2011, while no states reported cases only to
the SHR. Additional analyses to explore the effect of case identification methodology and SRC
affiliation failed to identify a specific ARD methodology that was associated with better program
coverage in 2011.

Page 1

The current analyses only compared ARD program coverage to the SHR. Other sources may also
provide additional information about the extent of law enforcement homicides in the United
States or in selected jurisdictions, and coverage of the various data collection systems. These
sources include the Fatal Injury Reports that are part of the National Vital Statistical System
maintained by the Centers for Disease Control and Prevention and direct reports from local law
enforcement agencies. In addition, the ARD program assessment examined only arrested-related
deaths that are the result of law enforcement homicides. Arrest-related deaths due to illness,
overdose, accidents, and other manners of death are likely even more difficult to identify and, if
included, could have a significant downward impact on our coverage estimates. However, no
other national data collection exists that examines arrest-related deaths due to a manner other
than law enforcement homicide. If BJS pursues a collection to measure law enforcement
homicides or all manners of arrest-related deaths in the United States, changes must be made to
the data collection methodology to support more complete coverage.

Page 2

1 Background
By definition, the Arrest-Related Deaths (ARD) program was to be a national census of law
enforcement homicides that occurred during the process of arrest or during an attempt to obtain
custody by a state or local law enforcement agency in the United States. The program was
implemented by the Bureau of Justice Statistics (BJS) in 2003 as part of its Deaths in Custody
Reporting Program (DCRP) in response to the Deaths in Custody Reporting Act of 2000 (DICRA).
In addition to the ARD program, the DCRP includes collections that measure deaths occurring in
jails and state prisons.
The ARD program methodology relied on state reporting coordinators (SRCs) in each of the 50
states and the District of Columbia. SRCs were responsible for understanding the scope and
definition of the ARD program, identifying eligible cases in their state, and working within
available resources to collect and report information about those cases. The manner in which these
data were collected varied from state to state, and often depended on the data systems available to
the SRC, the involvement of local law enforcement agencies or medical examiners/coroners
offices, and other support that the SRC may have had to conduct the data collection. This variability
in approach led to questions about whether the existing data collection methods were capable of
capturing the universe of arrest-related deaths and law enforcement homicides in particular.
The Supplementary Homicide Reports (SHRs) represent another national-level data source that
provides information on law enforcement justifiable homicides in the United States. 1 The SHR is
part of the Uniform Crime Reporting (UCR) Program conducted by the Federal Bureau of
Investigation (FBI). While the UCR summary program provides aggregate annual counts of the
number of homicides occurring in the United States, the SHR data provide additional details about
each homicide incident, including the jurisdiction, month, year, victim and offender demographic
characteristics, weapon, the circumstances surrounding the incident (e.g., argument, robbery, gangrelated), and the relationship between the victim and offender, if known.
To determine whether the ARD program met the criteria for a valid and reliable statistical
collection, BJS tasked RTI to undertake a program assessment to better understand the utility of the
ARD data collection approach and to effectively assess alternative approaches. RTI conducted the
ARD program assessment to evaluate (1) the homicide coverage of the program in comparison to
the Supplementary Homicide Reports (SHRs) and (2) the current program methodology. The
coverage assessment matched ARD cases to the SHR. A capture-recapture analysis provided
information on the scope and characteristics of cases eligible for inclusion in the ARD program
that are captured in one or more of these data systems. The program methodology assessment
further compared ARD cases identified by various approaches currently employed by the state
reporting coordinators (SRCs) and the ARD staff, including media searches, access to state
reporting systems, and direct reporting from law enforcement agencies. The program assessment
process and findings are documented below.

1
The SHR homicide also includes a small number of law enforcement-related homicides that are not classified as
justifiable, but those cases cannot be identified.

Page 3

2 Methodology
The Arrest-Related Deaths (ARD) program relied on a census collection from agencies that
experienced one or more deaths to estimate the number of arrest-related deaths in a particular
calendar year. Because the census was (1) obtained through self-reporting by agencies, as
opposed to targeted collection of a probability sample derived from a robust agency frame, and
(2) included no agencies that did not experience a death, it is difficult to make inferences
regarding any undercoverage using information from the ARD program alone. In light of these
constraints, a coverage assessment that utilizes comparable data from an auxiliary source is
appropriate.
The U.S. Department of Justice, through the Federal Bureau of Investigation (FBI), releases
annual law enforcement-reported homicide data in the form of Supplementary Homicide Reports
(SHRs). Because the SHR data provide incident-level information on homicides, they can be
used to help assess the coverage of the ARD data collection. Whereas the ARD collection
documents all manners of death that occur during the arrest process or in the presence of law
enforcement, the SHR is limited to homicides and therefore does not include suicide, accidents,
illness or natural causes, drug intoxications, and other manners of death not considered to be
homicides. Therefore, to make comparisons between these data collections, each must be
restricted to a similar population of cases. For the analyses presented below, we restricted the
SHR to only the subset of homicides in which an offender is killed by the police (justifiable
homicides) and subset the ARD to those incidents defined as homicides committed by law
enforcement. Both of these subsets are limited to incidents involving state or local law
enforcement agencies. The SHR and ARD data sets include all law enforcement homicides
reported in the years that the ARD program produced annual datasets: 2003 through 2009 and
2011. Some states did not report any SHR data in some years. With the exception of Florida,
which did not report any SHR data for the entire period, no additional information was added to
the FBIs SHR file for other states. Homicide cases for Florida were collected independently
from the FBI collection and added to the SHR file for this analysis.
By linking ARD decedent records to those captured in the SHR, it is possible to make inferences
about the total size of the law-enforcement homicide population and, thereby, about the
population coverage achieved in the ARD program. This analytic technique, known as capturerecapture, is typified by studies of wild animal populations, though it is widely used in the study
of human populations as well, particularly in coverage evaluation for censuses (Wolter, 1986).
Though the ARD coverage assessment is based on traditional capture-recapture methods, unique
characteristics of ARD data necessitate special techniques, including probabilistic record linkage
between data sources and adjustments for unobserved agencies.

2.1 Probabilistic case-level linking


Capture-recapture analysis depends on the ability to link records across data sources. In wildlife
applications, this requirement is easily met as captured animals are identified through tagging. In
the context of a coverage assessment utilizing an auxiliary data source not collected as part of the
coverage study, this requirement becomes more difficult. Because decedents are not uniquely
identified across data sources (by name or Social Security number, for example), record linkage
must be based on comparable decedent demographic and incident characteristics in an attempt to
Page 4

create a unique link. However, linkages made without unique identifiers are inherently uncertain,
decedents are matched between ARD and SHR sources probabilistically based on measured
similarity between records, within a monte carlo replication framework. This means that
although a given ARD decedent will be linked to either no SHR decedents or a single SHR
decedent in a particular replicate, the nature of the link for that ARD decedent (whether it is
linked and, if so, to which SHR decedent) is allowed to vary across replicates according to the
probabilities associated with particular record pairings, based on measured similarity between
cases. Probabilistic linking is random in nature and incorporates additional uncertainty into the
process of estimating population coverage; monte carlo replication allows for this additional
uncertainty to be captured in variance estimates.

2.1.1 Calculation of similarity scores


Records cannot be uniquely identified across sources so linking decedent records is done
probabilistically, based on the measured similarity between two records. The first step in
determining link probabilities for a given ARD decedent and the decedents potential matches in
the SHR is to calculate a similarity score. Similarity is based on the following variables that are
common to both ARD and SHR:

Originating Agency Identifier (ORI)

Sex

Year

Date reported/date of death

Age

Race/Hispanic origin

Decedent records are considered 0% similar if they do not match on at least ORI, sex, and year
and are not eligible for linking. For records that have common values for these three variables
across sources, scores are calculated based on the values of the date, age, and race/ Hispanic
origin variables. These measures include a mix of continuous and categorical variables;
therefore, the method for calculating the similarity score must account for both types.
The portion of the similarity score determined by the continuous variables (date and age) is
calculated as a function of the standard Euclidean distance for two variables, such that two
records having identical values across sources have a similarity of 1. The portion of the similarity
score determined by the continuous variables approaches 0 as dates and ages become more
dissimilar.
The portion of the similarity score determined by the race/Hispanic origin values is based on a
penalized scoring approach described in Boriah et al. (2008). This approach assigns a similarity
of 0 if the variables disagree across sources and a value of less than 1 when the variables agree.
The amount that matching records similarity score is discounted is determined by the
probability that the race/Hispanic origin value would be in agreement by random chance alone.
This approach is based on the premise that matches that have a high probability of occurring by
random chance should not count as much as those that would be observed very rarely by chance

Page 5

alone. The final similarity score is calculated as a weighted mean of the components based on
continuous variables (weight = 2/3) and race/Hispanic origin (weight = 1/3).
Once similarity scores are calculated for each potential pairing of a given ARD decedent and
SHR decedents from the same ORI, sex, and year, these values are used to create an additional
score. This score represents the combined dissimilarity between the ARD case and all of its
potential matches. Following calculation of this dissimilarity score, all scores are scaled such that
their sum is 1. These scaled scores are treated as relative probabilities and determine how cases
are linked. A detailed description of similarity score calculation, including all formulas, may be
found in appendix A.

2.1.2 Thresholds for matching


Potential record pairings fall into one of two categories: those that are within a predetermined
threshold of similarity and those that are not. If two records each have the same value of ORI,
sex, year, and race/Hispanic origin, there is a difference in the dates of report and death less than
or equal to 30 days, and there is a difference in recorded age of less than or equal to 5 years 2, the
pairing is thought to be within an acceptable threshold to consider the records as representing the
same decedent. In these instances, the scaled similarity scores (match probabilities) for all
pairings for the same ARD decedent that are not within threshold are set to 0. This includes the
scaled dissimilarity score, which represents the joint probability across potential pairings that no
SHR record represents the same decedent. This means that when an ARD decedent has one or
more pairings that are within threshold, the probability that the record will not be linked at all is
0. In some cases, a single ARD case may have more than one pairing that is within threshold. In
these instances, a link will take place according to probabilities resulting from rescaling the
similarity scores among only the within-threshold pairings such that they sum to 1. ARD
decedent records that have a single potential pairing that is within threshold undergo the same
rescaling process, resulting in a single potential pairing with a link probability of 1; these
pairings are thereby referred to as certainty pairings. For ARD decedent records with no potential
within-threshold pairings, linking occurs according to the probabilities associated with all
potential pairings and the calculated probability that no pairing is correct. Thus, these decedent
records may remain unlinked, and in the event that all potential pairings are very dissimilar, this
will be the most probable outcome.

2.1.3 Linking algorithm


The first step in the linking process is separating ARD and SHR decedent records into two
groups: those that have ORI, sex, and year values common across data sources and those that do
not. The latter group is considered unlinkable; those records with common values of ORI, sex,
and year are linked probabilistically according to the following steps (a more technical
description is provided in appendix B):

Age values in the SHR are considered imprecise and have been observed to clump around 5-year marks.
Additionally, the SHR date is the month and year the homicide was reported. These factors necessitate a certain
level of leeway in the criteria for linking and are what drive the linking threshold definitions.

Page 6

1. Randomly sort the ARD cases within ORI, sex, and year. The first step in linking ARD
decedent records to the SHR is a random sort. Because the ARD decedents are processed
sequentially and a linked SHR decedent is unavailable for linking to other ARD records
processed after the link is made, it is important to ensure that that the order in which the
ARD cases are processed is not fixed. This is necessary for monte carlo replication as
described in 2.1.4.
2. For each ARD case, calculate a similarity score corresponding to each SHR case with
the same ORI, sex, and year. In this step, similarity scores are calculated as described in
2.1.1, above. Note that for any ARD decedent record that is not the first to be processed in its
ORI, sex, and year classification, there may be SHR decedents ineligible for linking, as they
are linked to other ARD records processed previously. The similarity scores for such
ineligible pairs are set to 0.
3. Treating the scores from (2) above as probabilities, calculate the dissimilarity score (the
probability that the ARD case is not represented by any SHR decedent in the same
ORI, sex, and year). Given the small number of variables that overlap between the ARD and
SHR and the inability to perfectly match across sources, it is important to assign a nonzero
probability to the possibility that the ARD decedent is not represented by any of the SHR
decedent records in the same ORI, sex, and year.
4. Determine the number of cases that are within threshold. Each potential pairing for a
given ARD decedent is checked against the threshold criteria described in 2.1.2. If one or
more pairings are within threshold, all other scores (including the one described in (3),
above) are set to 0.
5. Scale scores. Though the scores described above may be individually interpreted as
probabilities, their sum may be greater than 1, complicating both their joint interpretation and
the process of linking. In this step, each of the scores is divided by their sum, maintaining
their relative proportions. Note that as a result of (2) and (4) above, several outcomes are
possible:
i.

No potential SHR decedent records have been previously linked to other ARD
records, and there are no within-threshold potential pairingsin this case there
will be a nonzero probability that no linkage will be made.

ii.

No potential SHR decedent records have been previously linked to other ARD
records and there is one potential within-threshold pairingin this case the
within-threshold pairing is a certainty.

iii.

No potential SHR decedent records have been previously linked to other ARD
records and there are multiple potential within-threshold pairingsin this case, a
linkage will be made only according to the probabilities among within-threshold
pairings, as all other probabilities have been set to 0 prior to scaling.

iv.

One or more potential SHR decedent records have been previously linked to other
ARD records and there are one or more potential SHR records available for
linking these cases are then treated identically to i. through iii., above.

v.

One or more potential SHR decedent records have been previously linked to other
ARD records and there are no potential SHR records remaining for linkingin

Page 7

this case, the probability that the ARD decedent record is not represented in the
SHR is 1 and the record is not linked.

2.1.4 Monte carlo replication


The inability to precisely match decedents between the ARD and the SHR necessitates a
probabilistic linking approach, but this technique is inherently random in nature, achieving the
most probable outcomes only in expectation. By repeating the linking algorithm many times, the
variability resulting from that randomness can be incorporated into final estimates.
The monte carlo framework leverages two points at which randomness enters the linking
process: the use of random numbers for sorting ARD decedent records and the use of random
numbers to select from potential pairings. This means that uncertainty in record linking resulting
from the lack of overlapping variables, and/or errors in variables that do overlap, will be
incorporated in final estimates.
Generation of random number streams depends on selecting seed valuesnumbers that are fed
into the algorithm as a starting point for generating the stream. By setting random number seeds
in a predictable way (for repeating the results) that is a function of both the replicate number and
the ORI, sex, and year grouping, variation inherent to the probabilistic linking can be correctly
accounted for. Given the resource-intensive nature of probabilistic linking in general, a replicate
count of 1,000 was selected to balance resource usage and precision.

2.2 Capture-recapture analytic approach


Once decedent records have been linked across the ARD and SHR, capture-recapture methods
can be used to estimate the total population of law enforcement homicide and, thereby, the
proportion of that population covered by the ARD program. This analysis makes use of the
commonly applied Lincoln-Petersen (LP) estimator, with modifications for bias amelioration
necessitated by the characteristics of the ARD program.
Use of the basic LP estimator implies acceptance of several assumptions, including the
following:
a. The population of interest is closed geographically and in time and is of fixed size;
b. Correct matches may be made between sources;
c. Each data source is free of spurious cases, such as those that are out of scope or that
are duplicates;
d. The probability of inclusion is independent across lists;
e. The probability of inclusion is constant within lists; and
f. The probability of inclusion is constant across lists.
For the purposes of evaluating ARD coverage, some of the above assumptions may be accepted
as true, while others must be accepted with qualification and, in some cases, in conjunction with
specific analytic techniques designed to minimize the biases incurred as a result of their likely
violation. Given the goal of the analysisto determine if ARD coverage is high enough to
Page 8

justify continuing the program in its current formviolation of some assumptions is not as
problematic as would be the case in other applications because the direction of incurred bias is
known. This allows for reasonable assertions of upper limits for ARD coverage, which provides
sufficient information given the circumstances.
In addition to the assumptions presented as a.f. listed above, this analysis must account for an
additional, and seemingly unique, challenge: estimating the impact that unobserved law
enforcement agencies with law enforcement homicides have on decedent population coverage.
Because agencies with law enforcement homicides report inconsistently to the ARD and SHR,
proper evaluation of coverage must account for whether or not agencies that are not observed in
either source experienced law enforcement homicides, and, if so, how many. Sections 2.2.1 and
2.2.2, below, address two methods of conducting the capture-recapture analysis, including
assumptions made and the benefits of each technique. A detailed, technical description of the
techniques employed is provided in appendix B.

2.2.1 Simple poststratified Lincoln-Petersen approach


This approach is the simpler of the two described here, as it does not incorporate any correction
for unobserved agencies. In other words, under this approach there is an implied assumption that
any agency not observed in either the ARD or the SHR has not experienced a law enforcement
homicide, or, put another way, this method assumes that every agency with homicides is
observed in at least one source. Though this assumption is strong (and very likely violated), the
results of this method are still quite useful given the context in which theyre being used.
Specifically, assuming that all unobserved agencies are so-called true zeroes (they have not
experienced any cases of law enforcement homicide), results in a coverage estimate that
represents the best-case scenario. Thus, the simple, poststratified LP estimate may be interpreted
as an upper bound for ARD coverage, with the true coverage rate falling somewhere below.
From a decision-making perspective, knowing the best-case estimate may be sufficient,
especially if the estimated coverage is considered unacceptable.

2.2.1.1 Assumptions
Assumptions a. and c. from 2.2 are accepted as valid in this approach. With both sources limited
to cases from 200309 and 2011, and all arising from U.S. agencies, assumption a. is reasonable.
Because each source has been subset to the common subpopulation of justifiable law
enforcement homicides, and BJS has verified that no duplicates are included in the files
provided, assumption c. is also reasonable. 3
As stated previously, given the inability to uniquely identify decedents across data sources,
assumption b. cannot be reasonably made. Conducting the analyses within a monte carlo

As part of the data quality checks, the ARD files were sorted several ways (i.e., state, year, name, date of death,
and date of birth) to identify duplicate records. This method was successful in identifying duplicate entries even
when the decedent had different names listed (e.g., first and last name inverted or middle name used as last name).
In addition, every ARD case was cross-referenced with the DCRP Jails file to identify duplicates across DCRP
collections. BJS used a similar method with the SHR file and removed any duplicates identified.

Page 9

replication framework allows for incorporation into final estimates of error incurred through
violation of this assumption, and for this reason, no single replicate can be evaluated for linking
correctness. Individual linkages may be made in a single replicate with low probability, though
across replicates they will occur rarely. Taken as a whole, across all replicates, average linking
outcomes represent the best matches given the available agency and individual information.
Assumption d., commonly referenced in the context of behavioral response in capture-recapture
literature, cannot be validated and potential biases incurred through violation cannot be mitigated
or quantified given the information available. It is worth noting, however, that the nature of this
bias may not be particularly important for this analysis. In the context of this study, a behavioral
response would mean that those responsible for collecting and reporting data to the ARD and
SHR programs would be more or less likely to report to one program if they had already reported
to the other. Because (1) it is reasonable to assume that such an effect could work in both
directions (i.e., a record may be more or less likely to be captured in one source if it has been
previously captured in the other) and (2) there is no reason to believe that one source consistently
captures decedent records before the other, any bias resulting from behavioral responses is
assumed to be diffused and to contribute little, if any, directional bias to final coverage estimates.
Homogeneity of capture probabilities implied by assumptions e. and f. cannot be guaranteed
within or across data sources, so the LP estimator of population size may contain downward bias,
leading to an overestimation of coverage (Wolter, 1986). The common method for dealing with
this biasand the one employed in this analysisis poststratification. Once decedent records
are linked, they are grouped into strata corresponding to the cross-classification of available
demographic characteristics. The simple LP estimator is then used within strata and the results
are aggregated over the complete population. This approach is based on the idea that, once
grouped, capture probabilities become homogeneous within strata. Poststratification on these
variables may not account for all bias because the number of reliable characteristics with enough
samples per category common to both sources is low (race/Hispanic origin and age only). Again,
in the context of this analysis, remaining bias may have little to no effect on the analytic utility of
the results, as the direction of the bias (if not its magnitude) is known. Heterogeneity in capture
probabilities yield population size estimates that are deflated and coverage rates that are
artificially high. Thus, final coverage estimates can be interpreted as the upper bounds on the
actual coverage rate.

2.2.1.2 Strengths and limitations


This simple LP estimator method has the notable benefit of being the simpler of the two methods
implemented, as it does not require estimation of the number of unobserved agencies with law
enforcement homicides. Additionally, this method yields an estimate that represents the best-case
scenario for ARD coverage. From the perspective of a user wanting to know the most accurate
coverage estimate, rather than its upper boundary, this method is not ideal.

2.2.2 Poststratified Lincoln-Petersen approach with constant ratio adjustment


for unobserved agencies
Unlike the approach described in 2.2.1, the poststratified LP method rejects the assumption that
any agency not observed in either the ARD or the SHR has not experienced a law enforcement
Page 10

homicide. It therefore must account for two additional unknown factors: the number of
unobserved agencies with law enforcement homicides and the number of homicides experienced
among those agencies.
Under this approach, the simple LP estimator is used to estimate the total number of agencies
that experienced law enforcement homicides in the reference period. By treating the ratio of this
estimate over the number of agencies actually observed across both sources as an agency
undercoverage adjustment factor, decedent population estimates corresponding to each
poststratum (as described in 2.2.1) may be extrapolated to represent the decedent population
among unobserved agencies. Because the decedent population is inflated by a constant factor,
this approach implies a strong assumption that missing agencies with homicides are missing at
random (i.e., the distribution of missing agencies in terms of decedent counts is identical to that
of the observed agencies). 4 Notably, agencies that are captured in both the ARD and SHR have,
on average, more homicides than those observed in only one source (5.2 vs. 1.5), and it may be
reasonable to infer that unobserved agencies are more like the latter group. In other words,
assuming that missing agencies are just like those that have been observed likely results in an
inflationary bias in the decedent population size estimate and, therefore, a deflationary bias in the
corresponding ARD coverage estimate. This implies that the coverage estimate derived using a
constant ratio adjustment for unobserved agencies represents a worst-case scenario, or lower
bound, for ARD coverage. Presenting the coverage estimates derived using this method along
with the one described in 2.2.1 yields a bounded range representing the most extreme
assumptions regarding the nature of unobserved agencies. Given the requirements of the
analysis, this is likely sufficient and benefits from making as few assumptions as possible.

2.2.2.1 Assumptions
Beyond the assumptions inherent to the agency undercoverage adjustment factor noted above,
the assumptions for this model are identical to those described in 2.2.1.1 and are not repeated
here.

2.2.2.2 Strengths and limitations


The postratified LP methodology provides additional benefit over that described in 2.2.1 in that it
rejects the (likely false) assumption that all unobserved agencies are so-called true zeroes. The
notable downsides of this approach are the necessity of making additional assumptions regarding
the nature of unobserved agencieseach additional assumption introduces new sources of
potential error. Additionally, as this method treats unobserved agencies as being like those that
are observed, in terms of the volume of law enforcement homicides, it potentially yields a
deflated coverage estimate.

Use of a simple, unstratified LP estimator likely results in deflationary bias in the law enforcement homicide
agency population estimate; its use is necessitated by resource constraints that precluded collection and inclusion of
agency-level characteristics for poststratification.

Page 11

2.2.3 Data assumptions and coverage estimate implications


The sections discussed above deal with theoretical assumptions underpinning the LP estimator.
As noted, these assumptions have been addressed where possible and, in all cases, their potential
effects on coverage estimates have been discussed. Given the context in which these estimates
are being used and the careful statements regarding their interpretation, the analysis is robust to
violations of these assumptions. Fundamental assumptions about the data, however, are largely
unverifiable.
For certain data items, specifically date and race/Hispanic origin variables that are used in
linking and that are measured differently across sources, comparability is maximized through
recoding (as noted in appendixes A and C) and the impact of remaining inconsistencies can be
inferred based on the amount of variability in final estimates across replicates, as the linking
procedure results in cases being linked probabilistically even when dates and race/Hispanic
origin codings are not in agreement across data sources. Because the range of final ARD
coverage estimates across replicates is less than 2 percentage points, it is reasonable to infer that
inconsistencies in these variables across sources do not have a large impact. Furthermore, the
small range of replicate estimates indicates that final estimates are not being driven by the
probabilistic linking.
Across replicates, 68% of decedent records remain unmatched after linking. Of these unmatched
records, 53% come from ARD and an overwhelming 82% of those are not even eligible to enter
the probabilistic linking procedure because they have no match in the SHR on ORI, year, and
sex. Given the relative rarity of female law enforcement homicides and the low estimated
probability of decedent records appearing in the incorrect year, ORI can be viewed as the linking
variable with by far the largest impact on final estimates. Data quality issues relating to ORI
codes may therefore affect estimated coverage estimates, though, unfortunately, errors resulting
from problems with ORI codes are of an unknown magnitude and direction.

Page 12

3 Findings
3.1 Arrest-related death coverage
As requested by BJS, RTI estimated the size of the law enforcement homicide population in the
United States and the Arrest-Related Death (ARD) program coverage using two methods. The
simple, poststratified Lincoln-Petersen method assumes that all nonreporting law enforcement
agencies did not have any law enforcement homicides during the observation period. This
approach provides the highest possible ARD coverage estimate and the lowest estimate of the
total population size (also referred to as the right bookend approach). Following this approach,
the ARD coverage of all law enforcement homicides is estimated to be 48.73%, with a 95%
confidence interval ranging from 47.28% to 50.18%. In other words, the ARD program captured
about half of all law enforcement homicides in the United States during the observation period.
The second coverage estimate assumes that all nonreporting agencies have the same number of
law enforcement homicides as agencies that did report to either the Supplementary Homicide
Reports (SHRs) or the ARD program. The poststratified Lincoln-Petersen approach with a
constant ratio adjustment for unobserved agencies provides the left bookend or lower estimate
of ARD coverage. Under this approach, the ARD program is estimated to capture 36.43% of all
law enforcement homicides in the United States, with a 95% confidence interval ranging from
34.50% to 38.35%.
Figure 1 is a Venn diagram of the ARD and SHR coverage following the simple, poststratified
Lincoln-Petersen method (the first coverage estimate described above), which assumes that all
nonreporting law enforcement agencies did not have any law enforcement homicides during the
observation period. Under these assumptions, the universe of law enforcement homicides in the
United States across all years examined (200309 and 2011) is 7,427, or an average of 928 law
enforcement homicides per year. The ARD program captures approximately 49% of these
homicides, while the SHR captures 46%. About 28% of the law enforcement homicides in the
United States are not captured by either system (see figure 2).

Page 13

Figure 1: ARD and SHR coverage and overlap of the universe of law enforcement
homicides in the United States, with no agency adjustment, 200309 and 2011

Page 14

Figure 2: Estimated proportion of law enforcement homicide universe covered by each


source, with no agency adjustment, 200309 and 2011

Unobserved
28%

SHR only
23%

ARD only
26%

ARD and SHR


23%

RTI implemented a monte carlo replication approach to match SHR and ARD cases (see table 1).
There were an estimated 5,324 law enforcement homicides reported to either system in the
observed years. More than two-thirds of those homicides were not able to be matched across data
sources. The level of unmatched cases was greatest in 2005 (75.6%) and lowest in 2011 (60.4%).
Certainty matches across sources are created when cases match on the originating agency
identifier (ORI) number, year, and sex, within 1 month of date of death (or date of report for
SHR) and within 5 years for age. Across all observed years, approximately 21% of all law
enforcement homicides reported to either source were certainty matches, ranging from 15% in
2008 to 26% in 2009 and 2011. These certainty matches represent 94.5% of ARD cases with one
or more SHR pairings satisfying the certainty criteria noted above (not shown). The remaining
5.5% have more than one match that satisfies the certainty criteria (4.9% have two matches, and
the remaining 0.6% have between three and six matches).

Page 15

Table 1: ARD and SHR law enforcement homicides, by match outcome, 200309 and 2011
Probability match score
No match
ARD program data
collection year
All LE homicides

AVG N

0.01 0.25
%

AVG N

0.26 0.50

AVG N

0.51 0.75
AVG N

0.76 0.99
AVG N

Certainty match

Total

AVG N

AVG N

3,643.9

68.4

221.0

4.2

190.1

3.6

150.8

2.8

6.8

0.1

1,111.9

20.9

5,324.5

100

2003

424.6

71.3

23.6

4.0

19.3

3.2

15.2

2.5

0.1

0.0

112.6

18.9

595.3

100

2004

416.7

70.6

18.9

3.2

19.4

3.3

11.8

2.0

0.6

0.1

123.1

20.8

590.4

100

2005

456.7

75.6

19.8

3.3

16.8

2.8

14.8

2.5

1.8

0.3

94.4

15.6

604.3

100

2006

469.8

70.9

22.7

3.4

27.0

4.1

24.4

3.7

0.1

0.0

118.4

17.9

662.4

100

2007

450.1

66.2

25.2

3.7

23.2

3.4

21.1

3.1

1.5

0.2

158.9

23.4

680.0

100

2008

482.6

73.4

37.1

5.6

21.1

3.2

18.5

2.8

0.0

0.0

98.0

14.9

657.3

100

2009

443.4

62.7

30.1

4.3

27.7

3.9

17.2

2.4

1.2

0.2

187.1

26.5

706.7

100

2011

500.1

60.4

43.6

5.3

35.7

4.3

27.8

3.4

1.5

0.2

219.4

26.5

828.0

100

Note: The 'AVG N' represents the mean count across all replicates.

Table 2: ARD law enforcement homicides, by match outcome, 200309 and 2011
Probability match score
ARD program
data collection
year

0.01 0.25

No match

0.26 0.50

AVG
N

1,939.5

53.6

221.0

6.1

190.1

5.3

150.8

4.2

6.8

0.2

2003

206.3

54.7

23.6

6.3

19.3

5.1

15.2

4.0

0.1

2004

201.4

53.7

18.9

5.0

19.4

5.2

11.8

3.1

2005

229.3

60.8

19.8

5.3

16.8

4.5

14.8

3.9

2006

254.4

56.9

22.7

5.1

27.0

6.0

24.4

2007

225.0

49.5

25.2

5.5

23.2

5.1

All LE homicides

AVG
N

0.76 0.99

AVG N

AVG
N

0.51 0.75
%

AVG
N

Certainty match
AVG N

Total

AVG N

1,111.9

30.7

3,620.0

100

0.0

112.6

29.9

377.0

100

0.6

0.2

123.1

32.8

375.0

100

1.8

0.5

94.4

25.0

377.0

100

5.5

0.1

0.0

118.4

26.5

447.0

100

21.1

4.6

1.5

0.3

158.9

34.9

455.0

100

2008

229.3

56.8

37.1

9.2

21.1

5.2

18.5

4.6

0.0

0.0

98.0

24.3

404.0

100

2009

232.7

46.9

30.1

6.1

27.7

5.6

17.2

3.5

1.2

0.2

187.1

37.7

496.0

100

2011

361.0

52.4

43.6

6.3

35.7

5.2

27.8

4.0

1.5

0.2

219.4

31.8

689.0

100

Note: The 'AVG N' represents the mean count across all replicates.

Page 16

There were 3,620 cases reported to ARD in the years observed (see table 2). Across all years,
54% of those were not able to be matched to an SHR case, and 31% were a certainty match.
Probability matches included 11.4% of ARD cases with a low probable match (probability score
of 0.50 or less), and 4.4% with a probability match score between 0.51 and 0.99. Among all
matched ARD cases, 47.6% were linked to a single SHR case across replicates (not shown), and
74.7% were matched to five or fewer distinct SHR cases. The maximum number of distinct SHR
pairings for a single ARD case across replicates was 24. Across years, the lowest overlap with
the SHR occurred in 2005 when 61% of all ARD cases were not able to be matched to an SHR
case. In 2009, however, 38% of ARD cases were a certainty match to SHR, 15% were a probable
match, and 47% were a no match. Figure 3 further shows the distribution of match scores for
cases reported to ARD across all years observed.

Figure 3: Percent of observed ARD law enforcement homicides, by match score, 200309
and 2011
No match

0.01 - 0.25

0.26 - 0.50

0.51 - 0.75

0.76 - 0.99

Certainty match

100.0
90.0
80.0
70.0

30.7

29.9

25.0

24.3

26.5

32.8

34.9

37.7

31.8

60.0
50.0
40.0
30.0
20.0
10.0
0.0

53.6

54.7

53.7

60.8

56.9

49.5

56.8

46.9

52.4

All years

2003

2004

2005

2006

2007

2008

2009

2011

Table 3 describes characteristics of law enforcement homicides reported to either the SHR or the
ARD program by match score. As expected, law enforcement homicide victims with an
unknown age are more likely to have no match. Law enforcement homicide victims that are
identified as white, non-Hispanic (75%) are most likely to have a no match compared to other
race/Hispanic origin groups, while victims of an other race are most likely to have a certainty
match (26%) compared to unknown or other unspecified race/Hispanic origin groups.

Page 17

The characteristics of cases reported to the ARD program by probability match score are
described in table 4. Cases with victims of any unknown age are more likely to have no match
compared to any known age group. Similarly, cases with victims of an unknown race/Hispanic
origin are more likely to have a low probability match score compared to any known
race/Hispanic origin groups. ARD cases with a victim identified as Hispanic (28%) are least
likely to result in a certainty match compared to other known race/Hispanic origin groups.
As described in appendix C, the race/Hispanic origin variable is measured differently by each
data source, and there is a high amount of missing data regarding Hispanic origin in the SHR.
This measurement issue may account for some of the differences in probability match scores by
race/Hispanic origin group. A subgroup analysis on states with larger proportions of Hispanic
populations was conducted to provide further insight into whether the difference in probability
match scores across race/Hispanic origin groups is meaningful. RTI calculated coverage
estimates among five states that report Hispanic origin to SHR greater than 75% (Arizona,
California, North Carolina, Pennsylvania, and Texas). Following the simple, poststratified
Lincoln-Peterson approach (upper bound), the ARD coverage in these five states is 71.91% (95%
confidence interval range from 69.47% to 74.34%). With agency adjustment (providing the
lower bound), the estimated ARD coverage is 54.90% in these five states (95% confidence
interval range from 50.52% to 59.29%). Given the wide range of these results, it is likely that
they are confounded with other, unmeasured agency characteristics, such as agency size,
reporting requirements, and the prevalence of law enforcement homicides. That is, the estimates
with these five states were intended to act as sensitivity analyses for measuring the Hispanic
origin of victims of arrest-related homicides; however, they suggest that other variables may be
driving ARD program coverage estimates.

Page 18

Table 3: ARD and SHR law enforcement homicides, by matching variable and match outcome, 200309 and 2011
Probability match score
No match

0.01 0.25

AVG N

AVG N

0.76 0.99
AVG N

Certainty match
AVG N

Total

3,643.9

68.4

221.0

4.2

190.1

3.6

150.8

2.8

6.8

0.1

1,111.9

20.9

5,324.5

100

Male

3,500.4

68.3

220.4

4.3

189.7

3.7

143.9

2.8

6.0

0.1

1,063.4

20.8

5,123.7

100

Female

140.6

71.1

0.6

0.3

0.4

0.2

6.9

3.5

0.8

0.4

48.5

24.5

197.8

100

17 or younger

115.4

70.0

10.7

6.5

5.4

3.3

2.8

1.7

0.0

0.0

30.5

18.5

164.9

100

1824

850.9

67.8

52.4

4.2

52.4

4.2

39.9

3.2

1.1

0.1

258.5

20.6

1,255.2

100

2534

1,087.1

67.4

69.4

4.3

64.7

4.0

46.0

2.9

0.9

0.1

344.1

21.3

1,612.0

100

3544

786.3

68.0

41.6

3.6

35.5

3.1

37.3

3.2

2.2

0.2

252.7

21.9

1,155.6

100

4554

509.5

70.5

30.0

4.2

21.1

2.9

14.6

2.0

0.8

0.1

147.1

20.3

723.3

100

55 or older

246.2

69.9

12.3

3.5

8.0

2.3

6.7

1.9

0.0

0.0

79.0

22.4

352.2

100

Unknown

48.5

79.1

4.5

7.4

3.0

5.0

3.5

5.7

1.7

2.8

0.0

0.0

61.3

100

White, nonHispanic

1,764.9

75.0

27.9

1.2

47.4

2.0

36.5

1.5

3.2

0.1

472.5

20.1

2,352.3

100

Black, nonHispanic

1,093.6

64.9

77.0

4.6

70.5

4.2

36.2

2.1

2.2

0.1

404.2

24.0

1,683.7

100

Hispanic, any
race

581.7

59.2

89.1

9.1

56.1

5.7

54.9

5.6

1.4

0.1

199.7

20.3

982.7

100

All LE homicides

0.51 0.75

AVG N

Matching variable

AVG N

0.26 0.50

AVG N

Sex

Age

Race/ Hispanic
origin

Other

86.0

63.2

4.8

3.5

4.9

3.6

4.9

3.6

0.0

0.0

35.5

26.1

136.2

100

Unknown

117.8

69.5

22.2

13.1

11.2

6.6

18.3

10.8

0.0

0.0

0.0

0.0

169.6

100

Note: The 'AVG N' represents the mean count across all replicates.

Page 19

Table 4: ARD law enforcement homicides, by matching variable and match outcome, 200309 and 2011
Probability match
No match
Matching variable

0.01 0.25
%

AVG N

0.51 0.75
AVG N

0.76 0.99
AVG N

Certainty match
AVG N

Total

AVG N

1,939.5

53.6

221.0

6.1

190.1

5.3

150.8

4.2

6.8

0.2

Male

1,865.7

53.5

220.4

6.3

189.7

5.4

143.9

4.1

6.0

Female

73.8

56.3

0.6

0.5

0.4

0.3

6.9

5.3

0.8

17 or younger

57.5

53.8

10.7

10.0

5.4

5.0

2.8

2.7

0.0

0.0

30.5

28.5

107.0

100

1824

427.7

51.4

52.4

6.3

52.4

6.3

39.9

4.8

1.1

0.1

258.5

31.1

832.0

100

2534

555.0

51.4

69.4

6.4

64.7

6.0

46.0

4.3

0.9

0.1

344.1

31.9

1,080.0

100

3544

426.7

53.6

41.6

5.2

35.5

4.5

37.3

4.7

2.2

0.3

252.7

31.7

796.0

100

4554

297.3

58.2

30.0

5.9

21.1

4.1

14.6

2.9

0.8

0.2

147.1

28.8

511.0

100

55 or older

143.1

57.5

12.3

4.9

8.0

3.2

6.7

2.7

0.0

0.0

79.0

31.7

249.0

100

Unknown

32.2

71.5

4.5

10.1

3.0

6.7

3.5

7.8

1.7

3.8

0.0

0.0

45.0

100

White, nonHispanic

926.6

61.2

27.9

1.8

47.4

3.1

36.5

2.4

3.2

0.2

472.5

31.2

1,514.0

100

Black, nonHispanic

550.8

48.3

77.0

6.8

70.5

6.2

36.2

3.2

2.2

0.2

404.2

35.4

1,141.0

100

Hispanic, any
race

322.9

44.6

89.1

12.3

56.1

7.7

54.9

7.6

1.4

0.2

199.7

27.6

724.0

100

All LE homicides

AVG N

0.26 0.50

AVG N

1,111.9

30.7

3,620.0

100

0.2

1,063.4

30.5

3,489.0

100

0.6

48.5

37.1

131.0

100

Sex

Age

Race/ Hispanic
origin

Other

45.8

47.8

4.8

5.0

4.9

5.1

4.9

5.2

0.0

0.0

35.5

37.0

96.0

100

Unknown

93.2

64.3

22.2

15.3

11.2

7.7

18.3

12.6

0.0

0.0

0.0

0.0

145.0

100

Note: The 'AVG N' represents the mean count across all replicates.

Page 20

Table 5 describes the characteristics of law enforcement homicides that were reported to the
ARD program, the SHR, or both. No differences emerged in cases reported to either system or to
both in terms of age and race/ Hispanic origin of the victim. The vast majority of all law
enforcement homicides resulted from a firearm, with no difference in the proportion of
homicides resulting from other means (e.g., other weapon) across data sources. Across ARD
program years, there does not appear to be a difference in the characteristics of cases captured by
the ARD Program, the SHR, or both.

Page 21

Table 5: ARD law enforcement homicide decedent characteristics, by source, 200309 and 2011
Percent of all law enforcement homicide victims
200306

All years
Decedent characteristics

ARD
only

SHR only

ARD and
SHR

3.4 %

2.9 %

ARD
only

200709

SHR only

ARD and
SHR

3.2 %

3.0 %

ARD
only

SHR
only

2011
ARD and
SHR

ARD
only

SHR
only

ARD and
SHR

Age
17 or younger

3.0 %

3.3 %

3.3 %

4.1 %

3.5 %

1.5 %

1.1 %

1.7 %

1824

22.1

24.8

24.1

22.2

24.9

25.1

22.0

25.3

23.8

21.9

22.2

22.5

2534

28.6

31.2

31.2

29.3

31.4

31.4

29.0

30.6

31.5

26.3

33.6

30.2

3544

22.0

21.1

22.0

23.4

23.3

23.6

21.1

18.9

20.8

20.3

18.0

21.0

4554

15.3

12.5

12.7

14.2

11.2

10.5

14.5

13.3

13.4

19.9

16.2

16.0

55 or older

7.4

6.1

6.3

5.7

5.3

5.3

8.2

6.6

6.2

10.0

8.4

8.5

Unknown

1.7

1.0

0.8

2.0

0.7

1.2

2.0

1.4

0.7

0.2

0.5

0.1

Race/ Hispanic origin


White, non-Hispanic

47.8 %

49.2 %

35.0 %

49.8 %

49.9 %

35.2 %

43.4 %

47.0 %

35.3 %

51.0 %

55.2 %

33.8 %

Black, non-Hispanic

28.4

31.8

35.1

26.5

30.1

33.2

33.3

34.4

35.7

23.9

29.8

38.0

Hispanic, any race

16.7

15.2

23.9

17.1

15.9

25.5

16.2

14.6

22.5

16.3

13.3

23.2

Other

2.4

2.4

3.0

3.1

2.9

4.4

1.7

2.0

2.4

1.9

0.5

1.2

Unknown

4.8

1.4

3.1

3.5

1.1

1.8

5.4

1.9

4.1

6.9

1.2

3.7

Manner of death
Firearm

96.3 %

96.0 %

97.0 %

99.4 %

97.0 %

99.9 %

90.6 %

94.9 %

92.5 %

99.4 %

94.8 %

100 %

Other

3.7

4.0

3.0

0.6

3.0

0.1

9.4

5.1

7.5

0.6

5.2

0.0

Missing

0.0

0.0

0.0

0.1

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

Page 22

3.2 Variability in ARD methodology across states


As described in Section 3.1, RTI calculated the size of the law enforcement homicide population
in the United States and the ARD program coverage using two methods. The first, a simple,
poststratified Lincoln-Petersen method, assumes that all nonreporting law enforcement agencies
did not have any law enforcement homicides during the observation period and therefore has no
agency adjustment. The second coverage estimate assumes that all nonreporting agencies have
the same number of law enforcement homicides on average as agencies that did report to either
the SHR or the ARD program. This poststratified Lincoln-Petersen assumes that the population
of interest is closed geographically and in time and is of fixed size, with a constant ratio
adjustment for unobserved agencies. Table 6 and figure 4 show coverage estimates using each
method by the year observed. The highest coverage rate with no agency adjustment was found in
the most recent collection year of 2011 (69%).

Table 6: ARD coverage estimates and confidence intervals, by estimation approach, 2003
09 and 2011
No agency adjustment

Year

95%
Lower bound

Coverage
estimate

Agency adjustment

95% Upper
bound

95% Lower
bound

Coverage
estimate

95%
Upper
bound

All study years

47.28 %

48.70 %

50.18 %

34.50 %

36.43 %

38.35 %

2003

36.87

41.84

46.82

21.96

26.61

31.27

2004

37.30

42.89

48.49

24.41

29.46

34.52

2005

33.67

38.55

43.43

19.90

24.25

28.59

2006

42.50

46.99

51.48

28.73

33.37

38.02

2007

46.45

50.51

54.58

35.18

39.61

44.04

2008

36.50

40.81

45.12

22.57

26.82

31.07

2009

51.37

54.98

58.59

35.00

39.64

44.29

2011

65.12

69.01

72.90

53.85

58.58

63.30

Page 23

Figure 4: Percent of law enforcement homicide universe covered by ARD, by estimation


method, 200309 and 2011

Percent of estimated law enforcement homicides

100%

80%

69%

No agency adjustment
60%
42%

43%

40%

55%

51%

47%

59%
41%

39%

40%

40%
20%

27%

29%

33%
27%

24%

Agency adjustment
0%
2003

2004

2005

2006

2007

2008

2009

2011

Year

The methodology for identifying ARD cases has changed over the observation period, and
figure 4 suggests that the methods used in more recent years are associated with better coverage.
To better understand the methodologies used to identify arrest-related deaths in 2011, RTI
conducted interviews with state reporting coordinators (SRCs). These interviews explored the
primary method each SRC used for identifying cases, whether direct reporting from law
enforcement agencies was involved, and whether the SRCs conducted any sort of follow-up or
verification of cases reported to them. In addition, RTI tracked information about reporting
methods and SRC location throughout the 2011 data collection period. This information was
used to further examine the utility of various methodologies for identifying arrest-related deaths,
as implemented by the SRCs during the 2011 data collection.
The remaining analyses examine differences in case matching by state and reporting method.
RTI grouped states by the primary data collection methodology and the location of the SRCs.
Then, RTI conducted analyses to compare the number of cases reported only to ARD, only to
SHR, or to both programs. Because the methodology for reporting was determined largely
through the SRC interviews and information gained during the 2011 data collection period, these
analyses are limited to 2011 data only. Table 7 describes state-level variation in reporting. In
2011, 83% of law enforcement homicides reported to either system were estimated to be
captured in the ARD program. Approximately 139 of the estimated 828 law enforcement
homicides captured by either system were reported only to the SHR.

Page 24

There is considerable variability among states in the proportion of law enforcement homicides
that are reported only to the ARD program, only to SHR, or to both sources. Three states
reported to both systems but had no estimated overlap between the two (Iowa, Minnesota, and
Arkansas). In other words, no law enforcement homicides reported to ARD were matched to
homicides found in the SHR, and the SHR deaths were not matched to any of the ARD deaths in
these three states. Twelve states reported all law enforcement homicides captured by either
system to ARD: Connecticut, Massachusetts, New Hampshire, New York, Kansas, North
Dakota, Alabama, Mississippi, West Virginia, Alaska, Idaho, and Montana. No states reported
law enforcement homicides only to the SHR. Reporting to ARD is estimated to capture at least
75% of cases reported to either system for 38 states, 11 of which reported at least 20 law
enforcement homicides to either system in 2011. Missouri, Maryland, Hawaii, and Nevada were
each estimated to report more cases to the SHR than the ARD program. However, ARD still
captured 82% of the estimated cases in Maryland and 84% of the cases in Nevada, and Hawaii
had very few arrest-related deaths reported to either system. In Missouri, half of the cases
reported to either system were found in the ARD, but 87% of the cases reported to either system
were found in the SHR. Rhode Island, Vermont, Delaware, and Wyoming did not report any law
enforcement homicides to either system in 2011.
Table 7: ARD law enforcement homicides, by region, state, and source, 2011
ARD only
Region and state

SHR only

ARD and SHR

AVG N

Row %

Col %

AVG N

Row %

Col %

AVG N

Row %

Col %

361.3

43.6

100

139.3

16.8

100

327.7

39.6

100

50.0

55.6

13.8

9.0

10.0

6.5

31.0

34.4

9.5

Connecticut

2.0

100

0.6

0.0

0.0

0.0

0.0

0.0

0.0

Maine

2.6

30.6

0.7

2.6

30.6

1.9

3.4

38.9

1.0

Massachusetts

5.0

100

1.4

0.0

0.0

0.0

0.0

0.0

0.0

New Hampshire

4.0

80.0

1.1

0.0

0.0

0.0

1.0

20.0

0.3

New Jersey

4.0

25.0

1.1

4.0

25.0

2.9

8.0

50.0

2.4

26.0

100

7.2

0.0

0.0

0.0

0.0

0.0

0.0

Pennsylvania

6.4

23.3

1.8

2.4

8.7

1.7

18.6

68.0

5.7

Rhode Island

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

Vermont

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

78.7

50.6

21.8

31.7

20.4

22.8

45.3

29.1

13.8

Illinois

9.1

28.4

2.5

1.1

3.4

0.8

21.9

68.2

6.7

Indiana

8.7

46.6

2.4

5.7

30.6

4.1

4.3

22.8

1.3

Iowa

4.0

80.0

1.1

1.0

20.0

0.7

0.0

0.0

0.0

Kansas

6.0

85.7

1.7

0.0

0.0

0.0

1.0

14.3

0.3

Michigan

8.0

44.4

2.2

5.0

27.8

3.6

5.0

27.8

1.5

Minnesota

2.0

50.0

0.6

2.0

50.0

1.4

0.0

0.0

0.0

Missouri

2.4

13.2

0.7

9.4

51.2

6.8

6.6

35.6

2.0

Nebraska

1.0

33.3

0.3

1.0

33.3

0.7

1.0

33.3

0.3

North Dakota

1.0

100

0.3

0.0

0.0

0.0

0.0

0.0

0.0

Total
Northeast

New York

Midwest

Page 25

ARD only
Region and state

SHR only

ARD and SHR

AVG N

Row %

Col %

AVG N

Row %

Col %

AVG N

Row %

Col %

29.5

88.0

8.2

0.5

1.4

0.3

3.5

10.6

1.1

South Dakota

3.0

42.9

0.8

3.0

42.9

2.2

1.0

14.3

0.3

Wisconsin

4.0

50.0

1.1

3.0

37.5

2.2

1.0

12.5

0.3

125.8

41.0

34.8

52.8

17.2

37.9

128.2

41.8

39.1

Alabama

7.0

100

1.9

0.0

0.0

0.0

0.0

0.0

0.0

Arkansas

3.0

75.0

0.8

1.0

25.0

0.7

0.0

0.0

0.0

Delaware

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

District of
Columbia

0.4

6.6

0.1

0.4

6.6

0.3

4.6

86.8

1.4

Florida

20.8

24.2

5.8

19.8

23.1

14.2

45.2

52.7

13.8

Georgia

12.9

43.1

3.6

3.9

13.0

2.8

13.1

43.8

4.0

Kentucky

8.0

61.5

2.2

3.0

23.1

2.2

2.0

15.4

0.6

Louisiana

3.7

42.8

1.0

1.7

20.0

1.3

3.3

37.2

1.0

Maryland

2.8

13.4

0.8

3.8

18.2

2.7

14.2

68.3

4.3

Mississippi

7.0

87.5

1.9

0.0

0.0

0.0

1.0

12.5

0.3

12.0

80.0

3.3

1.0

6.7

0.7

2.0

13.3

0.6

Oklahoma

5.1

31.5

1.4

3.1

19.1

2.2

7.9

49.4

2.4

South Carolina

5.4

52.1

1.5

3.4

32.9

2.5

1.6

15.0

0.5

Tennessee

2.1

15.8

0.6

2.1

15.8

1.5

8.9

68.5

2.7

Texas

23.1

47.1

6.4

7.1

14.5

5.1

18.9

38.4

5.8

Virginia

10.5

56.8

2.9

2.5

13.6

1.8

5.5

29.6

1.7

2.0

100

0.6

0.0

0.0

0.0

0.0

0.0

0.0

106.7

38.7

29.5

45.7

16.6

32.8

123.3

44.7

37.6

Alaska

1.0

50.0

0.3

0.0

0.0

0.0

1.0

50.0

0.3

Arizona

5.0

21.7

1.4

3.0

13.0

2.1

15.0

65.3

4.6

California

52.7

34.7

14.6

27.7

18.3

19.9

71.3

47.0

21.8

Colorado

10.1

59.1

2.8

5.1

29.8

3.7

1.9

11.1

0.6

Hawaii

0.1

4.4

0.0

2.1

68.1

1.5

0.9

27.4

0.3

Idaho

3.0

100

0.8

0.0

0.0

0.0

0.0

0.0

0.0

Montana

2.0

100

0.6

0.0

0.0

0.0

0.0

0.0

0.0

Nevada

1.6

9.4

0.4

2.6

15.4

1.8

12.4

75.2

3.8

New Mexico

5.9

49.4

1.6

1.9

15.6

1.3

4.1

35.0

1.3

Oregon

3.2

61.7

0.9

0.2

4.3

0.2

1.8

33.9

0.5

Utah

2.6

39.4

0.7

1.6

24.3

1.1

2.4

36.3

0.7

19.6

58.3

5.4

1.6

4.7

1.1

12.4

37.0

3.8

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

Ohio

South

North Carolina

West Virginia
West

Washington
Wyoming

Note: These are the mean counts across all replicates.

Page 26

3.2.1 ARD law enforcement homicide reporting methodology


Primary data collection methodology was determined through interviews with the SRCs in 2012
and from information gathered during the 2011 data collection period (see table 7). Most states
(N = 26) relied solely on media to identify ARD-eligible cases. ARD program staff identified
cases in four states, also relying primarily on media sources. Eight states used direct reporting
from local law enforcement agencies, and seven states relied on direct reporting from medical
examiners/coroners offices. Four states accessed the National Violent Death Reporting System
(NVDRS). Finally, two states had a legislative mandate to report arrest-related deaths.
Table 7: Primary methodology for identifying arrest-related deaths, by state, 2011

Legislative
mandate

Direct law
enforcement
agency report

Direct medical
examiners/coroners
report

National
Violent Death
Reporting
System
KY

Media only

CA

AL

CO

TX

DC

ME

KS

MA

MO

NM

UT

NH

NC

FL

ND

NV

SC

HI

OH

TN

WA

ID

PA

IL

RI

IN

SD

IA

VA

LA

WV

MD

MS

MI

MT

VT

ArrestRelated
Death
program
staff

AK

MN

AR

OK

AZ

NE

GA

OR

CT

NY

WI

DE

NJ

WY

Note: States are listed using standard U.S. postal abbreviations.

Figure 5 shows that there was not much variability in the proportion of observed law
enforcement homicides that were found in the ARD program based on the primary identification
method. The proportion of observed cases found in ARD ranged from a low of 80% for states
that relied primarily on direct law enforcement agency reports to a high of 85% for states that
relied on reports from medical examiners or coroners offices. The lack of association between
the primary identification method and the proportion of cases found in ARD could be due to
imprecise methods for categorizing states. For example, there may be additional variability
within primary methodologies. Some states with direct law enforcement agency reports may
have received these from only a few agencies in the state, and without a backup form of
identification, were missing many law enforcement homicides. The majority of states relied on
media reports, but there was likely to be considerable variability in the number of media sources
reviewed, whether such review was proactive or reactive, and whether media-identified reports
were then verified. Furthermore, the information RTI received about primary identification
methodology may not have reflected the true methods that the SRC uses in that state, due to
incomplete or outdated information collected during the SRC interviews or during the 2011 data
collection period.

Page 27

Figure 5: Law enforcement homicides, by source and primary reporting method, 2011
ARD only

90%

SHR only

Total cases (avg across replicates)


900

828
17%

16%

17%

15%
20%

80%

19%

800
19%
700

70%

600

60%
500
50%

371

400

40%
300

30%

201
200

20%

102

10%

72

42

41

ARD program
staff
identification
only

Access to and use


of NVDRS

0%

100
-

All states

Media report
only

Legislative
mandate

Direct medical
examiner/
coroner's office
report

Direct law
enforcement
agency report

Primary identification methodology

Page 28

Total number of law enforcement homicides observed

Proportion of observed cases captured in each source

100%

ARD and SHR

3.2.2 Location of SRC


States were also categorized by the location of the SRC. The greatest number of SRCs (25) were
located in a planning, commission, or statistics agency, such as a statistical analysis center, a
research or data agency within the states department of criminal justice, or a planning
organization (see table 8). SRCs in 13 states were situated in a state-level law enforcement or
public safety department. There were three SRCs in a state prosecutors office, three in a medical
examiner/coroners office or health agency, and three in a state corrections agency.
Figure 6 describes the proportion of observed law enforcement homicides that were found in
ARD by SRC location. SRCs located in state prosecutors offices and statistical analysis or other
research centers, and those located in other locations, each had approximately 87% of the
observed cases found in ARD, and only 13% of the observed cases found only in SHR. SRCs
located in state law enforcement agencies and state corrections agencies had the lowest
proportion of observed cases found in ARD (76% and 75%, respectively). State law enforcement
agencies may not have collected data regularly from local law enforcement agencies to report to
ARD, or they may have captured data only from a subset of agencies in the state. Data reported
to the SHR, on the other hand, came directly from local law enforcement agencies. SRCs located
in state corrections agencies may not have had the direct access or necessary relationships to
regularly collect information on law enforcement homicides that occurred during the process of
arrest. Further analysis may be warranted to determine how staff in state law enforcement
agencies work with local agencies to identify and collect information about law enforcement
homicides.

Table 8: SRC agency location, 2011


State law
enforcement
agency or
public
safety dept

State
prosecutors
office

Medical
examiner/coroners
office or health
agency

CO

NH

ME

FL

ND

NC

IN

HI

TX

OR

LA

MA
MI
MN

IL

UT

State
corrections
agency
ID

Planning,
commission, or
statistics agency
AK

NM

AR

AZ

NY

GA

CA

OH

WI

CT

OK

WY

DE

PA

MO

IA

VT

RI

KS

VA

SC

KY

WV

SD

MD

AL

TN

NV

MS

WA

NE

MT

DC

NJ

Note: States are listed using standard U.S. postal abbreviations.

Page 29

ArrestRelated
Death
program
staff

Figure 6: Law enforcement homicides, by source and SRC location, 2011


ARD only

90%

SHR only

Total cases (avg across replicates)


900

828
17%

13%

13%

13%
19%

80%

24%

25%

800
700

70%

600

60%

500

451
50%

400

40%
30%

300

221

200

20%
55

10%

42

30

29

0%

100
-

All states

Statistical analysis
or other research
center

State law
enforcement
agency

State prosecutors ARD program staff State corrections


office
agency
SRC location

Page 30

Other

Total number of observed law enforcement homicides

Proportion of observed homicides found in each source

100%

ARD and SHR

3.3 Number of years at least one death was reported to ARD and SHR
Table 9 describes the number of agencies reporting at least one death for the 8 years observed in
these analyses, and the estimated number of deaths reported by those agencies. Overall, more
unique agencies reported to ARD in the years observed, 2003 through 2009 and 2011 (N =
1,261), compared to SHR (N = 881). For both systems, the majority of these unique agencies
reported a death in only 1 of the 8 years examined. Fifteen agencies reported at least 1 death to
ARD in all 8 years, and 28 agencies reported at least 1 death to SHR in all 8 years. The number
of unique agencies reporting deaths in 7 or 8 years is higher for the SHR system compared to the
ARD program. This finding suggests that the SHR is more likely to include data from agencies
that consistently have at least one law enforcement homicide each year, as compared to the ARD
program. Agencies that do not have law enforcement homicides each year are more likely to be
found in the ARD data.

Table 9: Number of years at least one death was reported to ARD and SHR
ARD
Number of years at least one
death was reported
Total

SHR

Number of
agencies

Number of
deaths

Number of
agencies

Number
of deaths

1,261

3,620

881

3,385

15

696

28

1075

13

348

17

368

23

278

20

240

23

262

18

149

42

273

40

304

81

358

77

295

189

478

167

401

875

927

514

553

Page 31

4 Conclusions and recommendations


The capture-recapture analysis described above estimates that, at best, the Arrest-Related Deaths
(ARD) program captured approximately half of all law enforcement homicides in the United
States during the study period (200309, 2011). This estimate assumes that all unobserved law
enforcement agencies had zero law enforcement homicides. If those unobserved agencies did
have a similar number of law enforcement homicides as those that did report to the ARD
program or the Supplementary Homicide Reports (SHRs), the ARD coverage estimate is reduced
to 36%. This information alone may be sufficient for BJS to determine whether to continue the
ARD data collection program. Additional considerations for BJS to explore regarding continuing
data collection through the ARD program are described below.

4.1 Relative strengths and weaknesses of the ARD program methodology


A primary strength of the ARD program is that it was the only data collection that attempted to
enumerate all arrest-related deaths in the United States. Unlike the SHR, or any other national- or
state-level database, it also included accidental and natural deaths that occurred during the
process of arrest in addition to law enforcement homicides. Although the current analyses did not
identify any specific program methodologies that were more likely to identify law enforcement
homicides than others, we did not examine their utility for identifying other types of arrestrelated deaths.

4.1.1 Types of cases that were covered well by the ARD program
Analyses of decedent characteristics did not reveal any meaningful differences in capture rates
by age, sex, or race/Hispanic origin. The 2011 data collection year did exhibit some differences
across decedent characteristics compared to previous years, but these differences were not
confined to cases found only in the ARD program.

4.1.2 Identification approaches that are associated with better coverage


The analyses described above did not identify specific methodologies that were associated with
increased coverage. There was little variation in the percentage of observed cases found in ARD
(as opposed to the SHR) based on the primary methodology employed by the SRCs. For
example, our analyses indicated that 20% of all observed homicides were found only in the SHR
for states where the SRC relied primarily on direct reports from law enforcement agencies to
identify homicides. In other words, ARD captured 80% of the observed homicides in these
states. Eighty-five percent of the observed law enforcement homicides in states that relied
primarily on data from medical examiners or coroners offices were found in the ARD program.
Among these states, 15% of the cases observed in 2011 were found only in the SHR. Examining
state-level variations in reporting and matching those directly to the time put in by the SRC, the
location of the SRC, and the primary identification methodology will likely shed more light on
effective identification practices. BJS may also further examine the policies and practices in
California and Texas. Both states have mandated reporting legislation, but our analysis suggests
that approximately a fifth of the observed cases in California in 2011 were reported to the SHR
only, while 14% of the observed cases in Texas in 2011 were reported to the SHR only.

Page 32

4.2 Recommendations
While the capture-recapture analysis estimated that approximately half of all law enforcement
homicides were not captured in the ARD program during the study period, the estimated rate of
coverage for ARD in 2009 and 2011 was higher than the earlier years examined. In 2003, we
estimated that the ARD program captured, at best, 42% of the universe of law enforcement
homicides in the United States. In 2011, that coverage estimate increased to 69%. However, the
analyses described above did not uncover any significant variations in coverage for observed
homicides based on the location of the SRC or the primary identification methodology used in
2011. It is possible that all of the states were using more rigorous methodologies in 2011 as
compared to earlier data collection years. Even states that relied primarily on media to identify
cases may have been applying a comprehensive and regular approach to identifying these cases
and following up with other agencies to verify the death. In fact, most states relied on multiple
approaches to identify cases, but only the primary method was analyzed in this report.
In general, more of the observed law enforcement homicides in 2011 were found in the ARD
program compared to the SHR. There was still considerable state-to-state variation in reporting to
one or both programs, however, and our analyses failed to identify specific reasons for this
variation, such as the SRCs location or the primary identification methodology used.
A combination of data from both the SHR and the ARD is estimated to capture, at best, 72% of all
law enforcement homicides in the United States across all years observed. In 2011, the ARD
program captured, at most, an estimated 69% of the universe. Despite demonstrated improvements
in the ARD program coverage over time, there are still a large number of law enforcement
homicides that go unreported, indicating that the ARD is not a census.
A first step to increasing the coverage of ARD is to define a consistent data collection approach
across states, as opposed to the variable SRC-driven data collection methodology employed in
2011. More guidance or even requirements for where the SRC is located, how cases are identified,
and verification processes may increase the reliability and coverage of the program. However, such
an approach would increase the burden of this voluntary program, and would need to be balanced
with the utility of the data for data providers and stakeholders at both the national and the local
level.
The analyses described above only examined one alternative source for identifying law
enforcement homicides. BJS may further study other potential alternatives to the SRC-focused data
collection methodology to determine if any provide more cost-effective or more valid and reliable
means of identifying arrest-related deaths, including primary data collection or secondary analyses
of existing data sources.
Primary data collection methods such as direct surveys of law enforcement agencies or medical
examiners/coroners offices could identify law enforcement homicides and the circumstances
surrounding those homicides in the jurisdictions surveyed. Direct surveys would involve an
additional burden than the ARD programs SRC-focused approach through the cost of fielding the
study and the time required to collect the data, as well as the burden on survey respondents. If the
survey has a sufficient response rate and demonstrated validity, however, it may be an accurate
measure of law enforcement homicides in the United States.

Page 33

Less burdensome methodologies could include data collection from open sources. Open sources
are publicly available data compiled through targeted searches, such as Google Alerts and review
of specific websites devoted to law enforcement homicides. Open sources also include public
documents such as press releases from states attorney general offices, medical examiners, or state
police agencies. For example, the New York City Police Department releases an annual report
describing all law enforcement homicides in that jurisdiction. Finally, fee-based services such as
www.webclipping.com can be used to identify information from a large number of media outlets.
Reliance on open source methods, while cost-effective, has a number of limitations. For example,
media sources are often limited to newsworthy deaths and may be biased toward higher profile
cases. Furthermore, there may be considerable regional and local variation in media coverage for
law enforcement homicide-related events. Google Alerts and web-clipping services are also
constrained by the search terms provided, which must be balanced to include all potentially eligible
law enforcement homicides while not resulting in an unmanageable number of alerts to review.
The timeliness of open source searches presents another limitation, because information on webbased news platforms may be available for a limited time. Most importantly, however, the quality
of information from open sources is unknown, and the accuracy, reliability, and type of content
provided is likely to vary over time, by source, and in comparison to official records.
Other alternative measures of law enforcement homicides include existing data maintained by the
Federal Bureau of Investigation (FBI) and the Centers for Disease Control and Prevention (CDC).
Supplementary Homicide Reports (SHRs) are collected nationally as part of the FBIs Uniform
Crime Reporting Program (UCR), where information is voluntarily provided by law enforcement
agencies to their state UCR programs and then to the FBI. The SHR has the potential to identify or
confirm ARD cases categorized as a homicide by law enforcement officer(s). Other publicly
available data sources that may also measure law enforcement homicides include the FBIs
National Incident Based Reporting System (NIBRS), and CDCs National Vital Statistics System
(NVSS) and its National Violent Death Reporting System (NVDRS) Each of these data sources
has known limitations. For example, data from NIBRS, NVDRS, and local law enforcement
agency websites are not national in scope, and are available only for certain jurisdictions. Data
from the SHR and NVSS are national in scope, but do not contain unique identifiers that would
allow precise case-level matching and comparison with the cases submitted to the ARD program.
Studies have further documented inconsistencies in reporting to the ARD, SHR, and NVSS both
within and across states (Loftin et al., 2003; Mumola, 2007; RTI International, 2012). The ability
of any of these systems to accurately capture the universe of all arrest-related deaths as defined by
the ARD program, or law enforcement homicides in particular, is currently unknown.

4.3 Study limitations


The findings described above have several limitations, some of which may be addressed in
additional analyses as indicated by BJS. First, the study is limited by the approach to including all
probable and certainty matches in the analyses reported in exhibits 8, 10, 12, and 14. BJS may
choose to limit the cases identified as a match to those with a probability score cutoff of 0.50 or
higher, or to limit them to certainty matches only. The analyses presented here represent the best
case scenario for ARD; any assumption changes will lower the number and proportion of
observed cases that overlap across the two systems and increase the number and proportion of
observed cases found in one system only.

Page 34

The most significant limitation of this study, however, is measurement discrepancies between the
SHR and ARD on matching variables, and the lack of a unique identifier to positively link cases
across the systems. The monte carlo replicate approach implemented here provides the most
rigorous means of addressing this limitation, but the shortcomings in the matching variable
measurement remain. Findings were presented here as estimates of cases, but all estimates are
based on the replicate approach, so any conclusions about the actual number of cases matched or
covered by either system are not warranted. Instead, we can only estimate the level of matching
and overlap across the two systems based on the information provided.

Page 35

References
Boriah, S., Chandola, V., & Kumar, V. 2008. Similarity measures for categorical data: A
comparative evaluation. In proceedings of the eighth SIAM International Conference on
Data Mining. 243254.
Loftin, C., Wiersema, B., McDowall, D. & Dobrin, A. (2003). Underreporting of justifiable
homicides committed by police officers in the United States, 19761998. American
Journal of Public Health, 93(7), 11171121.
Mumola, C.J. (2007). Arrest-related deaths in the United States, 20032005. Bureau of Justice
Statistics Special Report. (NCJ 219534). Washington, DC: U.S. Department of Justice,
Office of Justice Programs. Retrieved from
http://bjs.ojp.usdoj.gov/index.cfm?ty=pbdetail&iid=379.
RTI International. (May 2012). Arrest-related deaths program: Program assessment, alternative
approaches to data collection, and recommendations for program enhancement. Draft
report submitted to the Bureau of Justice Statistics. Washington, DC: U.S. Department of
Justice, Office of Justice Programs.
Wolter, K.M. (1986). Some coverage error models for census data. Journal of the American
Statistical Association, 81 (394): 338346.

Page 37

Appendix A: Similarity scoring for ARD/SHR matching


Introduction
Calculation of similarity scores in the context of matching data from the Arrest-Related Deaths
(ARD) program with the Supplemental Homicide Reports (SHRs) requires a method that
accounts for both continuous and categorical variables. Cases will be matched between data
sources and within the originating agency identifier (ORI) code and victim sex on three
measures: date of incident/date of death, age, and race/Hispanic origin, with the first two
measures being continuous and the latter categorical. Given this mix of variables, RTI proposes
calculating scores using a mixture of methods as follows.

Continuous variables
For the date and age variables, we propose use of a standardized Euclidean distance, combined
with a similarity transformation.
Let
= number of records in SHR (within a given year)

= number of records in ARD (within a given year)

= SHR record index where [1, ]

= ARD record index where [1, ]

, = SHR date (measured in number of days since January 1, 1960 5) for record i
, = ARD date (measured in number of days since January 1, 1960) for record j

, = SHR age for record i

, = ARD age for record j

(
)
, = ,( )

( )
, = ,
)
(

The midpoint of the reported SHR month will be used to calculate the number of days since January 1, 1960, the
standard date convention used by SAS statistical software.

Page 38

, =
, =

, ( )
( )

, ( )
( )

Then the standardized Euclidean distance between records i and j on date and age is given by
2
2
, = , , + , ,

This distance is then transformed to a similarity score as


;, =
where

1
1 + ,

0 < ;, 1

Categorical variables
Rather than simply counting matches on categorical variables as 1 and mismatches as 0, we
propose the use of a penalized scoring approach similar to that described in Boriah et al., 2008. 6
This approach is based on the premise that matches that have a high probability of occurring by
random chance should not count as much as those that would be observed very rarely by chance
alone. For each categorical variable (currently only race/Hispanic origin), the similarity between
two records i and j is given by the following.
Let
()

= the sample probability that =

Then the similarity between records i and j on variable X is given by


1 ( ) ( ) if =
=
0
otherwise
0 < 1

The proposed method is similar to the Goodall3 approach described by Boriah et al., but adapted for two data
sources.

Page 39

Calculating the overall score


The final similarity score for records i and j is the weighted mean of the various components and
is given by the following:
2

, = 3 ;, + 3 /;,

Page 40

Appendix B: Linking algorithm


Estimate the agency population size and agency undercoverage adjustment
factor
To ensure that coverage estimates apply to the population of law enforcement (LE) homicides
across all agencies and not only to the population of LE homicides occurring in agencies that
have been observed, we must estimate the total population size of agencies with LE homicides
(reported or unreported). To achieve this, we will use the Lincoln-Petersen (LP) estimator 7 as
follows:
(1)

= ( +1)( +1) 1

+1
||

In (1), nARD is the number of agencies in ARD, nSHR is the number of agencies in SHR, and
is given by
nARD||SHR is the number of agencies represented in both sources. The variance of N
the following:
=
(2)

+1 +1 || ||
2
|| +2
||

The number of agencies observed across ARD and SHR is given by


= + ||

and the agency undercoverage adjustment factor is given by


=

and its variance is given by


() =

)
(
2

Classify cases as matchable or unmatchable


For the purposes of record linkage, cases are separated into two groups determined by whether or
not it is possible to match them between sources, within characteristic classifications that must
be the same: originating agency identifier (ORI) code, sex, and year. Cases from either source

7
By incorporating agency-level characteristics into a stratified LP estimate, we may reduce bias induced by the
assumption that agencies are observed with constant probability across agency characteristics.

Page 41

that cannot be matched on these three characteristics are considered unmatchable and are
excluded from subsequent probabilistic linking.

Probabilistic linking between ARD and SHR


For cases considered matchable across sources, probabilistic linking is used to determine
whether a case in the Arrest-Related Deaths (ARD) program is the same as a case in the
Supplementary Homicide Reports (SHRs). Within each distinct combination of ORI, sex, and
year, probabilistic linking proceeds according to the following steps:
a. Randomly sort the ARD cases
b. For each ARD case i, calculate a similarity score, , , corresponding to each SHR
case j

c. Create a 0/1 indicator, , , indicating whether SHR case j has already been linked
to another ARD record

d. Define ,
= , (1 , )

e. Create a 0/1 indicator, , , for whether or not each SHR case is within acceptable
threshold for being considered the same as the ARD case in question (an SHR
case is considered within threshold if the difference in dates of incident and death
is less than or equal to 30 days, the ages are less than or equal to 5 years apart,
and the race/Hispanic origin matches exactly); Note that if , = 1 then , = 0
f. Treating the scores as unscaled probabilities, calculate the probability that the
ARD case is not represented in SHR as follows

, = 1 ,
(1 , )
=1

g. Determine the number of cases that are within threshold as

= ,
=1

h. If > 0 then define ,


= ,
,

i. Otherwise, if = 0, then define ,


= ,

, =

j. Define , , the probability that ARD case i is the same as SHR case j as

, + 1 ,

k. Define the probability of no match as

Page 42

, =

, + 1 ,

l. Define the cumulative probability of record linkage as

, = ,
=1

m. Select a random variate , from ~(0,1)

n. Link ARD case i with SHR case j if case j is the first case (going from 1 to k) such
that , >

o. If this condition is not satisfied, ARD case i will be linked to no SHR cases with
probability ,

Recombine cases

Once matchable cases have gone through the probabilistic linking process, they are recombined
with cases previously determined to be unmatchable due to a lack of overlap on key variables
(ORI, sex, and year). Once recombined, the number of individual homicides in ARD, in SHR,
and in both ARD and SHR are tabulated; these are denoted as , , and || ,
respectively.

Stratify by individual-level characteristics

Since homogeneity of capture probabilities cannot be assumed within or across lists, the LP
estimator of population size may contain downward bias, leading to an overestimation of
coverage (Wolter, 1986). A common method for amelioration of this biasand the one we
employis poststratification. Once record linkage has occurred, cases will be grouped into
strata, defined by the cross-classification of demographic characteristics. If these characteristics
are correlated with capture probability, bias incurred through use of the LP estimator will be
reduced the choice of characteristics is limited to information available in both sources (age,
race/Hispanic origin, and sex).

Estimate population size and variance by stratum

Within each stratum , with 1 , obtain the LP estimate of population size and its
associated variance as
(3)

= , +1(, +1) 1

+1
,||

is given by the following:


The variance of M
=
(4) M

, +1, +1, ,|| , ,||


2,|| ,|| +2

Page 43

Estimate population size


By summing the stratum-specific population sizes, we obtain the overall population size among
observed agencies:

=1

is given by the following:


The variance of

= (
)

=1

Adjust population size for agency undercoverage

To obtain an estimate of the overall population size, , among all agencies with LE homicides,
including those unobserved across lists, we apply the ratio from 1 as follows:

The variance of is given by the following:

2 () + 2

Estimate ARD coverage

ARD coverage is estimated as the number of cases observed in ARD, divided by the estimated
population size:
(5)

The variance of is given by the following:


(6)

() = 2 ()

Replication

To achieve final estimates of coverage and variance that take into account uncertainty inherent to
the probabilistic record linkage, the process above is repeated 1,000 times. In each of these
replicates the linking procedure yields different results, and this variability is incorporated as
follows:
Let be the replicate index, where 1 1,000, then the overall population size estimate is
given by
1,000

1
=

1,000
=1

The variance of is given by the following:

Page 44

1,000

1,000

=1

=1

1
1
=
+
( )2
1,000
999

The final coverage estimate and variance are obtained by replacing and () with the above
in equations (5) and (6), respectively.

Page 45

Appendix C: Race/Hispanic origin coding


Before matching and capture-recapture analyses may begin, RTI must recode race/Hispanic
origin to be comparable between the Arrest-Related Deaths (ARD) program and Supplementary
Homicide Report (SHR) data sources. This requirement presents a problem in that race/Hispanic
origin is collected in a combined measure for the ARD and in two separate variables for the
SHR. Since separating the combined ARD measure into two variables comparable to those that
exist in the SHR would yield a missing race for all records currently coded as Hispanic, it is
much more desirable to create a combined measure on the SHR file that is comparable to that in
the ARD data. Given high levels of missingness in the SHR Hispanic origin measure, this
approach is also not without problems.

Hispanic origin in the SHR


At the national level, Hispanic origin in the SHR is highly underreported, with a missingness rate
of 53% in the years 200311.
Cumulative
V22

Frequency Percent Frequency


2051

53.44

(H) Hispanic origin

2051
600

(N) Not of Hispanic origin


(U) Unknown or not reported

15.63
1101
86

2651
28.69
2.24

3752
3838

To examine whether SHR respondents may be using the Hispanic origin variable as an extra race
category, when needed (i.e., when the deceased was obviously Hispanic), we explored the
hypothesis that if the Hispanic origin responses that were not missing were mostly 1, it would be
a strong indication that this was true. As shown in the frequency above, this is not the case. So,
we dug a little deeper to see where the missing Hispanic origin cases were coming from, with the
following result:
State N Reported Hisp.

CALIF 993 0.9073515

COLO 80

ALA

CONN

10

ALASKA 18

DC

17

ARIZ 177 0.8983051

DEL

ARK

FL

373

16

Page 46

GA

82 0.4512195

HAWAII 8

IDAHO 3

N MEX 46
NY

53

NEBR

11 0.1818182

ILL 137

NEV

79

IND

OHIO 44

50

IOWA 17

OKLA 92 0.5978261

KANS

OREG 33 0.6666667

14

KY

19

PA

185 0.9621622

LA

65

RI

SC

20

MAINE 18
MASS
MD

11

169

0
0

S DAK 7

TENN 73

MICH 78 0.1153846

TEXAS 320 0.8937500

MINN 31

UTAH 23

MISS
MO

VI

103 0.6116505

VA

45

VT

MONT
NC

42 0.7619048

N DAK 1
NH

NJ

102

0
0
0

W VA

0
0
0
0

WASH 98 0.2346939
WIS
WYO

Page 47

47 0.4255319
3

Figure 7: ACS 5-year Hispanic origin, by county

Highlighted in the table at left are the states where we would expect to find the largest Hispanic
populations. Of the 2,068 cases occurring in these states, 72% come from states with high
Hispanic origin coverage. This implies that the situation may not be as bad at the national level
as it might seem based only on the frequency above.
Assuming we were to create a combined race/Hispanic origin variable for the SHR, the
distribution of white, black, and Hispanic nationally in 200309 would be 45.89%, 33.89%, and
16.28%, respectively. In the ARD, the corresponding breakdown is 42.06%, 31.03%, and
20.36%.
If we subset to states with reporting rates for Hispanic origin greater than 75% (Arizona,
California, North Carolina, Pennsylvania, and Texas), the SHR distribution of white, black, and
Hispanic becomes 35.54%, 25.20%, and 34.01%, respectively. The corresponding breakdown in
the ARD, when subset to the same states, is 32.38%, 26.39%, and 34.70%.
Given these results, we follow an approach in which we create a combined race/Hispanic origin
measure in the SHR and conduct both a national analysis as well as one subset to the states with
high Hispanic origin reporting. Using only these five states would cover half of our national
sample and would give us some insight into how sensitive the capture-recapture results are to the
issue of data regarding Hispanic origin.

Page 48

The Bureau of Justice Statistics of the U.S. Department of Justice is


the principal federal agency responsible for measuring crime, criminal
victimization, criminal offenders, victims of crime, correlates of crime,
and the operation of criminal and civil justice systems at the federal, state,
tribal, and local levels. BJS collects, analyzes, and disseminates reliable and
valid statistics on crime and justice systems in the United States, supports
improvements to state and local criminal justice information systems, and
participates with national and international organizations to develop and
recommend national standards for justice statistics. William J. Sabol is
acting director.
This report was written by Duren Banks, Lance Couzens, Caroline Blanton,
and Devon Cribb, RTI International. Erin Kennedy, Crystal Daye, Lance
G. Couzens, Devon S. Cribb, Caroline Blanton, and Duren Banks, RTI
International provided statistical review and verification of the report.
Lynne McConnell and Jill Thomas edited the report. Barbara Quinn produced
the report.
March 2015, NCJ 248543

Office of Justice Programs


Innovation Partnerships Safer Neighborhoods
www.ojp.usdoj.gov

Похожие интересы