AMIA 2018 Clinical Informatics Conference

Multiple Choice Question (MCQ) Booklet

for Self-assessment

and

Maintenance of Certification (MOC) Part II Credit

[>280 pages in length – consider downloading to printing]

 AMIA 2018 Clinical Informatics Conference Self-assessment and
Maintenance of Certification (MOC) Part II Q&A booklet

Purpose
AMIA invited the AMIA 2018 Clinical Informatics Conference workshop and session presenters to submit
multiple choice questions (MCQ) associated with their presentations. This was to encourage interactivity
as well as to comply with Maintenance of Certification guidelines from the American Board of
Preventive Medicine (ABPM) and the American Board of Pathology (ABPath).

All attendees of the AMIA 2018 Clinical Informatics Conference may use these MCQs to test their
knowledge of what they learned in these sessions.

All board-certified clinical informaticians must engage in answering the MCQs

associated with the sessions for which they will claim MOC-II credit.
American Board of Pathology-boarded clinical informaticians: acceptable performance in this self-
assessment activity is defined as answering correctly 75% of the MCQs for the sessions you attended.
Copy the text on page 4 to send an attestation of your self-assessment to pesha@amia.org.

The ACGME competencies addressed by this activity are: Patient Care, Practice-based Learning and
Improvement, Interpersonal & Communications Skills, Professionalism, and Systems-based Practice. For
a full explanation of the ACGME competencies in the context of clinical informatics, see Safran C et al.
Program requirements for fellowship education in the subspecialty of clinical informatics. J Am Med
Inform Assoc. 2009 Mar-Apr;16(2):158-66.

MOC Committee Reviewers and Disclosures:

The following MCQ reviewers disclose that they and/or their life partners do not have relevant
financial relationships with commercial interests:
Kamran Badizadegan, Laura Fochtmann, Martha Hellems, Robert Leviton, Robert C. Marshall; Lisa
Masson, Karl Poterack, Matthew Rafalski, Sandeep Soman, Amy Y. Wang, Keith Woeltje

The following MCQ reviewers disclose that they and/or their life partners do have relevant financial
relationships with commercial interests:

Name of Commercial
Name of Individual Nature of the Relationship
Interest

Philip A. Smith MedMorph, LLC Stockholder

Applied Health IT Experts,

Stockholder
LLC

2|Page
Instructions
The first half of this booklet of about 270 MCQs provides the question and answer options organized by
day and by session number. The second half provides the correct answers per question. Each answer is
followed by an explanation and reference(s).

Clinical Informatics Diplomates: How to claim MOC-II credit

• Log into your account at www.amia.org to access AMIA Central (upper right corner)
• On the right side, click “My Events”
• Click “Apply for Credits” under “AMIA 2018 Clinical Informatics Conference”
• There will be two drop-down boxes - for the first one, called “Select Credit Type” click “Physician” (Do
not use “CME” as this is a placeholder not yet in use.)
• For the second one, “Select Physician Credit Type” click “MOC-II”
• Use the radio buttons to select the MOC-II sessions you attended
--You must claim your MOC-II credit first for AMIA to keep an accurate record of your MOC-II
credits
• At bottom of page, click “Save and Add Additional Credit Type”
• Now, be sure both drop-down menus show “Physician” and claim your regular CME session (there will
be only one, the keynote speaker session)
• Click Submit
• You will be prompted to download your certificate

Claim credit by Monday, June 11, 2018. AMIA reports MOC-II credits to the ABPM and to the American
Board of Pathology.

If you are board-certified through the American Board of Pathology, please

complete and send the form on the next page to Pesha@amia.org by June 11.
2018.
AMIA pulls data to send reports only within the same year as the activity.

3|Page
Email attestation text for American Board of Pathology-boarded clinical informaticians:

To: Pesha@amia.org

From: [Use the email you have on file with the American Board of Pathology]

Subject: MOC-II credits CIC2018

I attest that I attended the live activity AMIA 2018 Clinical Informatics Conference and used the
self-assessment booklet as my Self Assessment Module (SAM) for those sessions designated as
offering MOC-II credit in the activity.

I attest that I achieved at least a 75% score on the questions I answered that were associated
with the MOC-II designated sessions that I attended.

Diplomate’s ABPath ID number

First Name

Last Name

Date of Birth (Month and Day only,

mm/dd format)

Activity Completion Date in

mm/dd/yyyy format

ACCME Activity Program & Activity 201105920

Reporting System (PARS) ID

Number MOC Credits earned from

your claim at amia.org’s AMIA
Central

4|Page
Multiple Choice Questions for CIC 2018 Self-Assessment:
Tuesday, May 8th

WS02: Organizational Issues and Informatics: Translating Theory into Practice

--Kim Unertl, Saira Haque

WS02-1: When implementing health information technology, why is it a good idea to consider
organizational theory when developing an implementation plan?

A. Academics love theory

B. To help identify concepts that are relevant for informatics practice

C. It doesn’t make a difference

D. To inform your organization as a whole

WS02-2: What paradox led to the research that established Sociotechnical Systems Theory?

A. The Profit Paradox, that profits were elevated when new technology was implemented

B. The Happiness Paradox, that workers were happier when their work was going poorly

C. The Productivity Paradox, that despite improved equipment and better working conditions,
productivity went down

D. The Despair Paradox, that the better technology got, the less happy workers became

WS02-3: Everett Rogers identified four major elements in how innovations are adopted in groups, as
part of his Diffusion of Innovations Theory. What were the four elements?

A. The Innovation, Communication Channels, Time, A Social System

B. The Bosses, The Workers, The Tools, The Organizers

C. Advertising, New Technology, Coordinators, End Users

D. Great Ideas, Software Developers, Sales Representatives, Time

5|Page
WS02-4: There are five stages in Rogers’ Innovation-Decision Process. What is a major influence on the
Persuasion stage?

A. Whether people are really interested in making changes

B. How many people are interested in adopting the innovation

C. How people perceive the innovation

D. How long people have been using their past tools

WS02-5: In applying Lewin's Change Model to an organization that is considering a major health
information technology change, what strategies might be useful in "unfreezing" an organization?

A. Making sure the opinions of people who oppose the new technology are ignored

B. Avoiding contact with the intended end users of the technology

C. Firing anyone who disagrees with the choice of new technology

D. Involving all levels of intended users, from front-line workers to management, in the decision-
making process

WS02-6: Why is the low point of the Change Curve Model sometimes referred to as the "Valley of
Despair"?

A. People like to whine

B. People worry for no reason when productivity goes down

C. When productivity dips because of new technology, people have a hard time understanding how
the technology might help them in the future

D. The implementation productivity dip does not actually exist, but is rather just a theoretical model
that has been disproven by years of technology implementation

6|Page
WS02-7: Many organizations refer to the day a new technology system is turned on as a “Go Live” date.
Why is this a term that we are better off avoiding?

A. Going “live” makes people think about death

B. Implementing technology occurs over time and “go live” implies it is a single point in time

C. Customers/patients might think we are talking about medical procedures

D. People are unnecessarily picky about language use

WS02-8: What is one reason that people worry about change?

A. No one worries about change – it’s just a part of life

B. Concerns about wasting time and effort

C. People who worry about change are irrational

D. People enjoy finding new things to worry about

WS03: How to Properly Onboard Doctors in the Age of Informatics & Digital Health

--Hamed Abbaszadegan, Nacchal Nachiappan, Shakaib Rehman

WS03-1: As a busy clinician, there is no time to learn a new EHR system in a three-day period. Patients
need to be seen. What is your approach to onboarding this busy physician:

A. Cancel Clinic

B. Reschedule patients

C. Find coverage

D. Block time 4 months in advance

7|Page
WS03-2: As a long-time provider at your local facility, you never really learned how to use the EHR nor
keep up with new changing widgets and apps. How can you attend New Provider Orientation if you are
not new?

A. Displace the new provider

B. Schedule special sessions

C. Advanced planning of participants

D. New Provider Onboarding is only for new providers

WS03-3: Every hospital has a unique set of features and health IT tools that they invested in. How is
curriculum established?

A. Given dynamic changes in the health IT world, curriculum can change at any time

B. Teaching is universal and content for onboarding is usually uniform

C. New digital toys should be brought on display for new providers

D. Once you know which providers attend, content should be tailored to fit

WS03-4: Shorthand and macros can make any process faster. Given EHRs have more than one way to do
the same thing, what approach should be taken when onboarding?

A. Show all methods of shortened processes

B. Discuss approaches to macros and let the providers decide what is best

C. Only show what works for the majority

D. Discourage using quick methods for processes

8|Page
WS03-5: Health care providers are diverse in their workflows. How can this diversity be a part of your
onboarding process?

A. Too much work flow diversity cannot be overcome

B. Customize a trainer depending on who attends class

C. Ensure representation of both in/outpatient settings for instruction

D. Any provider can teach class as long as they have EHR experience

WS03-6: Providing health care now involves utilization of advance messaging techniques such as secure
messaging through web portals. How can doctors be more efficient in processing of secure messaging?

A. Show the providers in detail how to send messages

B. Review all bells and whistles with advanced secure messaging

C. Set up teams associated with providers who can help resolve appropriate messages

D. Ensure that rules of timeliness of secure messaging is well explained

WS03-7: Navigating a new health system can be daunting with all the rules. What approach will yield a
higher likeliness of success?

A. Establish a buddy mentor and organize forms in easy-to-find locations

B. Show which forms matter to physicians and demonstrate how to complete them

C. Invite departmental secretaries to training sessions

D. Teach the providers medial assistant how to obtain forms

9|Page
WS03-8: Pharmacy prescribing practices such as opiates have recently been highly publicized. Which
approach will support providers as they start with your healthcare system?

A. Have a pharmacist team member teach how to write prescriptions in a rapid manner

B. Teach how to build quick order sets

C. Invite your pain management director along with your pharmacy team member to review
support roles and processes available

D. Show providers how to rapidly access decision support tools via phone apps

WS04: Hands-On Full Life Cycle Data Science Workshop

--Steven Johnson, Lisiane Pruinelli, Tamara Winden

WS04-1: When using EHR data in an observational study, which of the following is correct? IRB review…

A. Does not apply since this is not a clinical trial.

B. Is not needed since the data is being collected in the EHR to document care

C. Is not needed since we are only accessing the EHR and not interacting directly with patients

D. Is required since we are using patient-specific health data

WS04-2: How do data quality issues impact data science results?

A. Because EHR data is used to document healthcare, there are no data quality issues

B. Data quality can significantly impact results, making them biased or even invalid

C. Data quality issues can be ignored because machine learning and AI will take care of it

D. Poor data quality doesn’t impact results, because the data quality issues disappear with enough
data

10 | P a g e
WS04-3: A good practice for exploratory data analysis is to use a number of techniques to visualize your
dataset. Which of the following is NOT a good reason to visualize your data?

A. Data visualization allows you to better understand and ”get to know” your data

B. Humans can spot visual patterns much better than looking at descriptive statistics

C. Visualization allows you to see large amounts of data summarized and simplified

D. Visualization looks nicer than just having tables of numbers

WS04-4: After receiving a dataset for a data science project, a researcher will spend a significant amount
of the total project time performing data preparation and exploratory data analysis. Which of the
following is NOT a data preparation task?

A. Assessing missing data and imputing appropriate values or removing records

B. Adding variables, transforming data and joining data tables

C. Ensuring all data is of the appropriate type and is consistent with the data mode

D. Obtaining the appropriate level of IRB approval for the project

WS04-5: Many datasets have imbalanced data where there is a much smaller amount of data for
positive classes than negative ones. For example, the prevalence of a number of diseases is low in the
overall population (i.e. Ebola). Which is the best method to evaluate the performance of a classification
model for imbalanced data?

A. Accuracy

B. F1 Score

C. Precision

D. Recall

11 | P a g e
WS04-6: Once data are properly prepared and cleaned, researchers should select the most appropriate
modeling algorithm depending on the data and the research question. Which modeling technique does
NOT work well on categorical variables?

A. Classification

B. Linear Regression

C. Neural Network

D. Random Forest

WS04-7: Once you have a good performing model and you have tested it sufficiently on your training
data, it is time to implement it in the real world. Which tasks are essential for successful model
deployment?

A. Ensure the model is activated at the right time in the workflow, appropriately alerts the clinician and
allows specific action

B. It is not necessary to get buy-in from the clinicians, since the model embedded in the EHR will tell
them what to do

C. Such models should only be created for research purposes and should not be used to support direct
patient care

D. The model should only be implemented in the latest version of the EHR software

WS04-8: Data science is fast becoming very useful in healthcare, but success requires close collaboration
between people with clinical knowledge and people with data and computer expertise. Which of the
following is true about data science in healthcare? Data science…

A. Can be understood sufficiently by everyone in the healthcare organization

B. Is easy since we give raw EHR data to machine learning algorithms to produce highly accurate results

C. Is somewhat of a fad and it won’t have a significant impact in improving healthcare

D. Only needs to be understood by a few people in the healthcare organization

12 | P a g e
WS05: Operational and Practical Aspects of Clinical Knowledge Management

--Dominik Aronsky, Saverio Maviglia, Asli Ozdas Weitkamp, Roberto Rocha, Dirk Wenke

WS05-1: What is a logical first step in starting a clinical knowledge management program within the
iterative knowledge management framework?

A. Build standards-based interoperable knowledge assets

B. Centralize the process of creating knowledge assets

C. Create a catalog of existing knowledge assets including relationships with metadata

D. Establish a knowledge asset lifecycle

WS05-2: Which of the following activities is least likely within the scope of a clinical knowledge
management framework?

A. Build a sustainable knowledge asset framework with reusable components

B. Integrate knowledge assets into clinical workflow

C. Maintain the representation of knowledge assets

D. Provide knowledge lifecycle support

WS05-3: Which topic is least likely associated with clinical knowledge management?

A. Institutional data governance

B. Lifecycle management of knowledge assets

C. Management of knowledge meta-data

D. Single-login management for clinical workstations

E. InfoButton

13 | P a g e
WS05-4: Which process is not directly supported by a clinical knowledge management system?

A. Centralized prioritization of knowledge management implementation tasks

B. Centralized request management for changes in the knowledge base

C. Monitoring clinicians’ usage of clinical decision support systems

D. Versioning of knowledge assets

WS05-5: Which characteristic is NOT considered best practice for creating knowledge assets?

A. Interoperable

B. Modular

C. Recursive

D. Reusable

WS05-6: Which of the following is the least critical method applied in clinical knowledge management?

A. Formal representation of knowledge assets

B. Lifecycle management of knowledge assets

C. Management of knowledge asset reference material

D. Versioning of knowledge assets

14 | P a g e
WS05-7: Select the most important role(s) domain experts (or subject matter experts) fulfill
within the context of a knowledge management lifecycle?

A. Review quality measures and propose new knowledge assets

B. Create asset specifications based on existing knowledge representation standards

C. Author new knowledge assets using tools available within clinical knowledge management
systems

D. Critique asset specifications and share expertise during knowledge engineering sessions

E. Support end-user adoption of new knowledge assets

WS05-8: Which of the following is NOT correct when deploying or updating vendor-based
knowledge assets in a clinical information system environment?

A. Integration requires workflow adjustments, data mappings, and extensive testing

B. Assets cannot be easily customized to reflect local needs and priorities

C. Deploying updates of vendor-based knowledge assets prevents CDSS malfunctions

D. Local customizations might be lost when subsequent updates are deployed

E. Effective troubleshooting by vendor is associated with security vulnerabilities

WS06: CMIO Workshop - Leading Clinical Informatics Initiatives: How to Successfully Tackle Challenges
& Build Resilience

--Julie Hollberg, Richard Schreiber

WS06-1: All hospitals with an EHR should follow the principles of EHR governance. A formal, written EHR
governance plan is necessary for:

A. Only large university hospitals that have large information technology staffs

B. Hospitals that have a chief medical informatics officer who runs the EHR

C. Hospitals that manage their EHRs, but not those who contract EHR services

D. All hospitals of whatever size, leadership structure, or EHR design

15 | P a g e
WS06-2: The most critical success factor in EHR governance is:

A. Managing and leading change transformation processes

B. Assuring purchase of the newest technology

C. Establishing best infrastructure prior to and after EHR installation

D. Rewarding staff for meeting or exceeding goals

WS06-3: Effective governance requires effective communication for EHR optimization and healthcare
transformation, as well as implementation. All the following have been shown to be successful forms of
communication. The most effective is:

A. Rich channels such as personal interaction and audiovisual aids are better than email alone

B. Encouraging leaders to disseminate information to users by any means

C. Just-in-time training—not too soon, not too late; as well as at-the-elbow support

D. Involvement of clinicians as well as technical experts in creating the communication plans

WS06-4. Lack of available time, lack of political and communication skills, and lack of vision are
examples of leadership failures which can damage EHR implementation or migration. How can one avoid
such failures?

A. Have a mission statement, a long-term strategic plan, and governance

B. Create a project plan, use Gantt charts to determine time lines, and follow agile processes

C. Hire experienced managers to run the day-to-day operations of the implementation

D. Be sure to hire a chief medical informatics officer to manage the process

16 | P a g e
WS06-5. There are many factors which ensure a successful EHR implementation or transformation.
Which of the following is most predictive of success?

A. Training for computer-savvy providers or those who have used the EHR before is not necessary

B. Reducing ambulatory productivity permanently will accommodate changed provider workflows

C. Managers supporting implementation should have no other responsibilities at go-live

D. Additional staff is required to execute the new workflows and provide at-the-elbow support.

WS06-6. A recent study showed that different vendor databases vary in their categorization of drug-
drug interactions (DDIs). Different EHR vendor and home-grown products are configured differently, and
different sites differ as to whether they modify the databases to alter the number of alerts displayed.
Other studies show that despite these differences it is possible to decrease the burden of DDI alerting.
This data suggests:

A. It is too complicated and fraught with potential error to modify the vendor product’s data base

B. It is best practice to accept the vendor product’s DDI database and alerting configuration

C. Meaningful use, which requires DDI alerting, has set a new standard of care for such alerts

D. It is time for a revamping of the international standards of care for DDI alert strategy

WS06-7. The CIO calls you into her office. She tells you the hospital is going to select a new EHR
(Electronic Health Record), implement it, and then optimize it. All is to be done within the span of 24
months. She wants to hire a Chief Medical Informatics officer (CMIO). You advise her that the best
candidate is:

A. Physician with extensive EHR experience but not necessarily any informatics education

B. Clinical Informatician educated in and experienced with Informatics

C. Physician with extensive Informatics education but limited EHR experience

D. Only physicians, and absolutely no PhDs or Masters

17 | P a g e
WS06-8. Successfully passing the ABPM board exam in applied clinical informatics requires knowledge
in a diverse set of disciplines. These include:

A. Workflow analysis, health information technology, and clinical care

B. Clinical care, workflow analysis, and information system lifecycles

C. Evidence basis for clinical care, change management, and decision science

D. Health systematics, clinical decision support, and computer programming

WS06-9. Yogi Berra once said, “When you come to a fork in the road, take it.” Many of us change our
career trajectories, and this can be a difficult transition. Some are better at these transitions because:

A. Self-reliant physicians manage change well

B. Career and life coaches can benefit physicians

C. Physicians often change positions without difficulty

D. New positions are a welcome and exciting challenge

WS06-10. Book knowledge provides a solid basis for medical and informatics practice; experience
provides real-world application of that knowledge. Since young informaticians lack that experience, the
single best strategy to increase one’s knowledge is to:

A. Work long hours in the clinic and/or hospital as well as in informatics to augment their learning

B. Seek out a mentor to help with challenging situations and to help guide your career

C. Treat a transition into a new role as a temporary condition as things will work themselves out

D. Avoid revealing one’s difficulties to colleagues as this may jeopardize your job performance

18 | P a g e
WS06-11. Which of the following is not a key driver or domain of burnout and professional fulfillment?

A. Efficiency of practice

B. Practice productivity

C. A culture of wellness

D. Personal resilience

WS06-12. Tuckman’s model of team work posits that groups will develop in a predictable fashion. The
usual order of that development is:

A. Norming, storming, forming, performing

B. Storming, forming, norming, performing

C. Forming, performing, storming, norming

D. Forming, storming, norming, performing

WS06-13. The best statement regarding IT governance is:

A. A single structure for good IT governance can keep pace with all the “wants” and “needs” of an
organization

B. IT governance does not need to include research and educational needs in its structure—separate
governance provides adequate oversight

C. How an institution makes IT investment decisions is often more important that what specific decision
sare made

D. Proper leadership governance in IT is sufficient to overcome the lack of a clinical information systems
advisory group

19 | P a g e
WS06-14. When facing conflict over very important issues, and there are no time pressures for
resolution, you should first use which of the following leadership/management methods:

A. Accommodation

B. Compromise

C. Collaboration

D. Competition

WS06-15. Alert fatigue reduces efficiency of use of electronic health records. Many clinicians gloss over
alerts, often without reading them. Research reveals various ranges of override rates, but the most
widely quoted and likely most accurate overall rate is:

A. 60%

B. 75%

C. 96%

D. 99%

WS06-16. Which of the following leadership qualities would be most important for a chief medical
informatics officer to display when implementing new systems and applications?

A. Charismatic

B. Task-oriented

C. Transformational

D. Supportive

20 | P a g e
WS07: Get Your Hands on FHIR®

--Laura Heermann, Russ Leftwich

WS07-1. What does the acronym FHIR® stand for?

a. Federal Health Information Record

b. Fast Healthcare Interoperability Resources

c. Fire Hot Information Response

d. Fast Health Information Record

WS07-2. FHIR® is designed to

a. Replace enterprise healthcare records

b. Be used exclusively for personal health records

c. Enable the exchange of healthcare-related information

d. Burn up all other attempts at interoperability

WS07-3. Why is FHIR® important to clinicians?

a. It leverages the latest web standards and applies a tight focus on interoperability across care
settings and disciplines.

b. It is a standard specification specific to the needs of nursing interoperability.

c. FHIR® has specific resources that apply only to hospital care.

d. FHIR®has no implications for clinical interoperability

21 | P a g e
WS07-4. How is the Fast Healthcare Interoperability Resources specification accessed?

a. http://hl7.org/fhir

b. www.fhir.com

c. www.FIRE.org

d. http://fasthealthcareinteroperability.org/

WS07-5. What is a FHIR® Resource?

a. A system of interoperability gadgets that hook together

b. A logically discrete data concept that has a defined meaning, a known identity and location, and
has meaning in healthcare.

c. Short snippets of reusable clinical themes used to comprise a medical record.

d. Small increments of reusable code.

WS07-6. What is an extension in FHIR®?

a. A way to limit the use of a specific resource

b. An adornment to a resource to make it more interesting.

c. A reference to an additional resource that makes it more meaningful.

d. A method to implement additional valid requirements not included in the core specification.

22 | P a g e
WS07-7. What does it mean to constrain a FHIR® resource?

a. To limit a resource to a specific definition fitting the intent of the implementation (such as
limiting a pick list with the colors red, green, yellow, blue, orange and purple to red, yellow and blue
only)

b. To allow for additional items to be added to a resource

c. To compel a user to document using FHIR resources only.

d. To discourage use of specific parts of a FHIR resource but allow them if absolutely necessary.

WS07-8. What is a FHIR® Profile?

a. A sketch of a drafted FHIR® resource prior to its being released for use.

b. A specification of a FHIR® resource or resources detailing constraints and extensions necessary

for use in a particular use case.

c. A fully instantiated resource

d. A FHIR® resource in action sharing patient data from one clinical site to another.

WS07-9. Name 3 FHIR® Architectural Principles.

a. Fast, healthcare specific, reusable

b. Fast, easy to learn, scalable

c. Reusable, Scalable, Readily Implementable

d. Implementable, clinically sufficient, highly structured

23 | P a g e
WS07-10. FHIR® Resources are organized into 5 framework layers. These include:

a. Foundation, Base, Clinical, Financial and Specialized

b. A, B, C, D, E layers

c. Problems, Meds, Allergies, Orders and Labs

d. Structural, Clinical, Observations, Provider, Payer

WS07-11. Resources classified as “workflow” are meant to:

a. Support the healthcare process

b. Conform to a specific institution’s clinical workflow.

c. Be implemented using BPMN

d. Identify the users of a specific workflow

WS07-12. What is a FHIR® Implementation Guide?

a. A listing of FHIR® resources

b. A how-to guide on profiling FHIR® resources

c. A knowledgeable person available for new implementers for assistance as needed.

d. A publication outlining capabilities as defined by national standards, vendor consortiums, clinical

societies, etc., regarding how FHIR® specifications are used to solve particular problems.

24 | P a g e
WS07-13. The Observation Resource should be used for what type of data?

a. Home meds, orders, MAR

b. Vital Signs, physical exam finding, pulmonary artery pressure reading, lab results

c. CT Scan, MRI, Xrays

d. Problem List, Chief Complaint, Past Medical History

WS07-14. The Condition Resource should be used for what type of data?

a. CT Scan, MRI, Xrays

b. Vital Signs, Physical Exam Findings, Pulmonary Artery Catheter reading

c. Diagnosis, Chief Complaint, Past Medical History

d. Home meds, orders MAR

WS07-15. The clinFHIR software tool requires what configuration parameters?

a. A use case type aligned with the categories of FHIR® resources

b. Selection of a set of capability statements from a library

c. Selection of LOINC and SNOMED value sets from the NIH Value Set Access Center

d. Setting compatible data, conformance, and terminology servers

WS07-16. How do I provide feedback to the FHIR development team regarding FHIR resources?

a. Send them an email to FHIR@FHIR.org

b. Call HL7 Headquarters

c. Submit them through the gForge tool

d. Don’t bother – the specification is done and published.

25 | P a g e
WS07-17. What is a FHIR® Maturity Level?

a. A method for the FHR specification developers to inform implementers how advanced and
stable a FHIR Resource is.

b. How long the FHIR® resource has been in existence

c. How many times the FHIR® resource has been implemented

WS07-18. What is the interoperability paradigm in which FHIR® is operable?

a. The messaging paradigm, like HL7 version 2.

b. The document paradigm, like HL7 Clinical Document Architecture (CDA).

c. As a RESTful API.

d. FHIR® can be used in all of the above interoperability paradigms.

WS07-19. The Argonaut project is what type of initiative?

a. An initiative to develop FHIR® certification testing led by the federal government.

b. An initiative initially started to accelerate development of FHIR®profiles for the Common Clinical
Data Set sponsored by a vendor collaborative.

c. An initiative to develop FHIR® translations led by the HL7 Greece affiliate.

d. An initiative that is part of SMART on FHIR®.

WS07-20. What is SMART on FHIR®?

a. An ONC project to create FHIR® profiles for CDS.

b. A platform architecture for portable healthcare apps utilizing FHIR® as an API for health data.

c. The organization which has published FHIR® profiles for the Common Clinical Data Set.

d. The organization which first developed the concept of FHIR®.

26 | P a g e
WS07-21. Which of the following HL7 standards include specifications for an application
programming interface (API)?

a. HL7 Version 2.x

b. HL7 Version 3

c. HL7 CDA

d. HL7 FHIR®

WS07-22. FHIR resources can be linked to each other via what type of element in a resource
definition?

a. Code

b. Codable Concept

c. Reference

d. Identifier

WS07-23. Which of the following represent types of terminologies that can be linked to a FHIR®
Profile?

a. SNOMED and LOINC

b. SnowOwl and RF2

c. C4NNol1 and NaPoLeon

d. Java and Python

27 | P a g e
WS08: How to Effectively Negotiate the Outcomes You Really Desire?

Shakaib Rehman, Hamed Abbaszadegan

WS08-1: According to the Thomas-Kilmann Model, if your concern about your own outcome is low and
your concern for the other party's outcomes is high; your approach would be:

A. Accommodating

B. Avoiding

C. Compromising

D. Problem-solving

WS08-2: Psychologist Stephen Covey’s research has shown that a certain percentage of life is under your
control and is decided by how you react while the other portion of life is called destiny (no matter what
do you do; it will happen because it is inevitable). Which of the following option is correct about his
determination of life events decided by you over what happened to you?

A. 60/40

B. 70/30

C. 80/20

D. 90/10

WS08-3: Psychologist Abraham Maslow has described the “Human Hierarchy of Needs.” What is the
highest need of individuals, according to his model?

A. Esteem

B. Love

C. Security

D. Self-Actualization

28 | P a g e
WS08-4: What one of the following best describe “Principled Negotiations”?

A. Focus on interests not positions

B. Focus on people not problems

C. Identify subjective criteria to determine success

D. Identify Worst Alternative to a Negotiated Agreement (WATNA)

WS08-5: Which one of the following is not part of the “Principled Negotiation” model?

A. Separate people from problem

B. Invent options for mutual gain

C. Insist on using subjective criteria

D. Focus on interests, not positions

WS08-6: The “Principled negotiation” model emphasizes the importance of the “BATNA.” What does
BATNA stand for?

A. Best Action to a Negotiated Agreement

B. Best Alternative to a Negotiated Agreement

C. Best Alternative to a Nonsense Agreement

D. Best Action to a Nonsense Alternative

WS08-7: Complete the list of Thomas-Kilmann's five conflict management styles: Competing,
accommodating, avoiding, ___ and ___.

A. collaborating, and compromising.

B. colluding, and connecting.

C. communicating, and compromising.

D. cooperating, and connecting

29 | P a g e
WS08-8: If a person typically tends to delegate controversial decisions and accept others’ decisions,
which of the following styles best describes this behavior?

A. Accommodating

B. Avoiding

C. Collaborative

D. Competitive

WS09: Designing Outpatient Clinical Decision Systems for High Use Rates, Provider Satisfaction, and
Improved Care: Lessons from Meta-Analyses and Experience

--Patrick O'Connor, JoAnn Sperl-Hillen

WS09-1: Outpatient Clinical Decision Support (CDS) has been shown to be least effective at improving
which of the following clinical measures:

A. Adult Immunization rates

B. Blood pressure control in hypertensives

C. Diabetes eye exam rates

D. Lipid test rates

E. Screening Mammography rates

WS09-2: Which of the following statements is most true of the impact and cost of Clinical Decision
Support and Nurse Case Management to improve the proportion of those with diabetes who reach
recommended treatment goals:

A. Clinical Decision Support is as effective as Nurse Case Management

B. Clinical Decision Support and Nurse Case Management can reach similar proportions of adults with
uncontrolled diabetes

C. On a population basis, the cost effectiveness of Clinical Decision Support and Nurse Case
Management for diabetes care are similar

D. Nurse Case Management can reduce overall health care costs

E. Clinical Decision Support for diabetes care is cost saving to payers

30 | P a g e
WS09-3: Consider a CDS system that extracts clinical information from the EHR, sends it to a web service
where it is processed using clinical algorithms, and then transmits CDS back to the clinician and the
patient at the point of care. Barriers to broad dissemination this type of CDS system include all of the
following except:

A. Getting physician leaders from many medical groups to agree on clinical goals for BP, A1c, lipid
management, and aspirin use in various groups of patients.

B. Concerns about data security when transmitting personal health information using the cloud

C. The potential of EHR-derived clinical information to be inaccurate or incomplete

D. Desire to provide evidence-based care personalized to each patient

E. Low use of the CDS system by primary care providers

WS09-4: In a classic paper by McGlynn et al published in 2003, the number of evidence-based actions
that apply to adults in primary care settings is approximately:

A. 60

B. 300

C. 600

D. 3000

E. 6000

WS09-5: In a time-motion study of primary care physicians using a popular electronic health record
system, the number of clicks and minutes required to get 80% of relevant data on cardiovascular risk to
assess control of BP, lipids, glucose, smoking, aspirin use, and weight/BMI during an encounter was:

A. 4 clicks and 20 seconds

B. 12 clicks and 50 seconds

C. 30 clicks and 2 minutes

D. 50 clicks and 4 minutes

E. 70 clicks and 6 minutes

31 | P a g e
WS09-6: The approximate annual payment from a medical group to an EHR vendor for use and
maintenance of an EHR software system is approximately:

A. $500 per clinician per year

B. $1,000 per clinician per year

C. $2,000 per clinician per year

D. $5,000 per clinician per year

E. $50,000 per clinician per year

WS09-7: Advantages of web-based, rather than within-EMR clinical decision support systems include all
the following except:

A. Many medical groups can use a single CDS installation

B. Updating CDS algorithms when guidelines change is cheaper in a web-based service

C. Updating CDS algorithms when guidelines change is less prone to error if only one installation
requires updating

D. Data security is improved when CDS is web-based

E. The cost of shared CDS installations would likely be lower than the aggregate cost of each medial
group doing their own CDS installation

WS09-8: In many cases, there are conflicting clinical guideline recommendations from different sources.
For example, the 2018 American Diabetes Association recommends a BP goal for adults with diabetes of
< 140/90 mm Hg, while the 2017 AHA/ACC hypertension guideline recommends a BP goal of < 130/80
mm Hg for adults with diabetes. The best way to handle this type of situation in primary care CDS
systems is to:
A. Provide detailed information to clinicians at each encounter, explaining the different
recommendations
B. Insert “hard stops” to assure that clinicians must pay attention to BP goals at every patient encounter
C. Provide no CDS on BP management until expert panels resolve their differences
D. Frame treatment recommendations using words like “consider” rather than words like “should,” to
accommodate differences in guidelines
E. Give each hypertensive patient a written sheet explaining the situation

32 | P a g e
Wednesday, May 9th

S01: Presentations - Integrating Quality and Workflow

Quality in Action, Adapting Quality Measures for Realtime Guidance

--Andrew Simms

S01-1: When are providers required to report electronic Clinical Quality Measures data to CMS?

A. 2017

B. 2018

C. 2019

D. 2020

S01-2: The Merit-Based Incentive Payment System (MIPS) uses a score system for calculating
performance. What percentage of this score is based on quality improvement activities?

A. 10%

B. 15%

C. 25%

D. 60%

From Pathway to Bedside: Integrating Clinical Informatics in Quality Improvement.

--Eric Shelov

S01-3: The most valuable asset of a Clinical Informatician in a Quality Improvement project centered on
Clinical Decision Support is their knowledge of:

A. Data Standards

B. Clinical & IS workflows

C. EHR functionality

D. Database structure

33 | P a g e
S01-4: The most fundamental strength of a clinical pathway is its ability to help ________ clinical care
and best practices.

A. Improve

B. Disseminate

C. Standardize

D. Accelerate

How am I doing? Online, interactive quality measures for clinicians

--Mark Weiner

S01-5: Patients may see different providers over time, so attributing the quality of a patient’s care to a
single provider can be challenging. Which of the following is the most sensible way to ensure that the
association between patient and provider is appropriate for attribution of a quality measure?

A. Recent Provider

B. Listed Provider in EHR

C. Listed Provider on the Insurance Card

D. Most frequent Provider

S01-6: Providers who receive report cards on their quality of care express many concerns about the
validity of the measures. Which of the following statements about quality measures is true?

A. If a quality threshold is relaxed, the proportion of a provider’s panel achieving that threshold is likely
to increase

B. If a quality threshold is relaxed, a provider’s quality ranking is likely to improve

C. A higher-ranked (“better”) provider will have a greater number of patients meeting a quality
threshold then a lower-ranked (“worse”) provider.

D. Stratifying a provider’s panel by the presence of diabetes will decrease the proportion of the cohort
receiving a HBA1c test compares with the overall cohort.

34 | P a g e
S02: Presentations –- Standards and Best Practices for EHR Implementations

Come Together: Lessons Learned from a Health System EHR Migration and Proposal for Sharing
Implementation Best Practices

--John McGreevey

S02-1: All of the following are limitations to the broad applicability of existing guidance about how to
implement an EHR except:

A. Existing EHR implementation guidance is based on valid experience and research and comes from
respected health care, academic, and government institutions

B. Work describing an implementation addresses a narrow population (example: small, rural primary
care clinics)

C. Some literature is becoming less relevant with time (example: paper to EHR transition)

D. Guidance focuses on a single aspect of an implementation (example: CPOE)

E. Guidance tends to be noncontroversial and described at a high level (example: training is important)

S02-2: Which of the following combinations correctly represents lessons learned in this health system
EHR implementation?

A. Given that vendors have extensive experience with implementation, do not deviate from vendor
guidance; develop an infrastructure of point people from various domains to support the
implementation work; get an early start with change management; forego developing a strategy related
to the EHR implementation as the vendor will guide you to a successful outcome.

B. Begin an implementation with internal alignment of practices, policies, equipment so that there is
consistency throughout the organization; develop an infrastructure of point people from various
domains to support the implementation work; recognize that sometimes vendor guidance will be
appropriate and at other times, an organization may need to follow its own plan; wait to initiate change
management efforts until 4-6 weeks before go-live so that the rest of the implementation work can be
completed first.

C. Get an early start with change management; begin an implementation with internal alignment of
practices, policies, equipment so that these are consistent across the organization; develop an
infrastructure of point people from various domains to support the implementation work; maintain
humility

D. Begin an implementation with internal alignment of practices, policies, equipment so that there is
consistency throughout the organization; “no training, no access, no kidding” was an effective training
strategy in this implementation; get an early start with change management; recognize that with a
thoughtful strategy for an EHR implementation, anyone who reports problems with the system or who
participates in workarounds after go-live is likely doing so because they are resisting necessary
organizational change.
35 | P a g e
Seize the Moment: Promotion of Lean Principles in Moving from Paper to Electronic Provider
Documentation at an Academic Medical Center

--Joel Betesh

S02-3: An Advantage of the APSO (Assessment, Plan, Subjective, Objective) note over a SOAP
(Subjective, Objective, Assessment and Plan) note in the EHR is that:

a. It is supported by extensive high quality literature

b. It is easier for the reader of the note to find the key elements of decision making of the note author

c. It protects against notes being bloated with info pulled in from elsewhere in the chart

d. APSO has been used more extensively over the last few decades

S02-4: Moving from paper to electronic format alone is not enough to fully alter ingrained provider
practices and perceived documentation needs. Problems with this transition at the health system
described include all the choices below except:

a. Providers continue to feel a need to add data to their note for purposes of billing.

b. Manual chart review is still required to assess note quality

c. It is possible for individual providers to deviate significantly from the standard templates offered at a
health system level

d. The majority of notes evaluated after the transition from paper to electronic documentation used the
APSO format

Maintaining EHR Standards in a Complex Deployment

--Keith Woeltje

S02-5: All of these are important to invite to participate in the change control meetings, except:

A. CMOs/CNOs
B. Local leaders from each hospital
C. All subject matter experts (SMEs) for each specialty
D. CMIO

36 | P a g e
S02-6: Change control is an essential skill for clinical informaticists. Which of the following is TRUE about
good change control? Change control…

A. is very technical, and should really only involve trained analysts and informaticians

B. helps prioritize limited resources and ensures system standards are upheld

C. has to be done because it’s part of Meaningful Use

D. is most useful at the start of a project, and is less useful once application teams are more sure about
what they are doing.

S03: Panel - Clinical Informatics Fellows Project Ignite Talks

--Christopher Hollweg (Presenter)

S03-1: In the FHIR (Fast Healthcare Interoperability Resources) specifications, what is the difference
between “Resources” and “Extensions”?

A) Resources manage the incoming FHIR queries and Extensions the outgoing queries

B) Resources map the data back to the database while extensions map the data to SMART applications

C) Resources define the standard data types while extensions the additional data types

D) Resources deal with structured data (i.e., ICD) and extensions with unstructured data (i.e., free text)

S03-2: Which of the following is not a core principle of information system security theory?

A) Availability

B) Vulnerability

C) Confidentiality

D) Integrity

37 | P a g e
S03-3: A swim lane flowchart differs from a simple flowchart in that it:

A. Focuses on the value stream

B. Is a physical map of movements of people in the workflow

C. Visually represents the actions taken by various roles

D. Maps out the steps in the process

S03-4: Why is ELK stack a good tool set to manage healthcare data? Because it is…

A) an SQL database management system and can be easily queried

B) a NoSQL database management system and is highly efficient in querying text rich data

C) a graph database and can be easily queried, especially for imaging data

D) not scalable therefore provides security and stability

E) not open source, with proprietary technology that cannot be breached by outside hacker

S03-5: Which of following is one of the common issues encountered when clinicians are developing
SMART-on-FHIR® CDS with developers?

A. Could be hard to have access to a FHIR® database/server. The database might not have the clinical
data of your interest (e.g., lab values of the liver function test.)

B. Mock EHR might not be available even when the team has a FHIR® server - building a barrier to the
usability testing.

C. Designing UI/UX requires serious feedback from clinicians

D. Medical calculators could be the serious contenders.

E. All of Above

38 | P a g e
S04: Presentations - Evidence-based Care Improvement

Article Relationship Visualization to Support Evidence-based Practice & Knowledge Discovery

--Po-Yin Yen

S04-1: What contributes to the highest quality of evidence for clinical practice?

A. Rapid review

B. Systematic Review

C. Literature Review

D. Scoping Review

S04-2: What is the most commonly reported barrier to evidence-based clinical decision-making? Lack
of…

A. time to process new information

B. literature with high level of evidence

C. interprofessional communication

D. departmental leadership

Characterizing Cutaneous Lupus Erythematosus with Meta-Analysis Using STARGEO

--Osama El-Sayed

S04-3: A researcher is interested in performing a meta-analysis with data from the gene expression
omnibus. What is a potential barrier to using GEO in one's research?

A. There is no standard way of filling out the metadata for the study

B. There is no information included about how the samples were processed

C. GEO does not contain any data on context of the study design

D. GEO is not accessible to the public and requires a subscription to access its data

39 | P a g e
S04-4: What is one way that generating disease signatures from meta-analyses can improve clinical
care?

A. Disease signatures that are generated from meta-analysis are individual specific, which means that
we can find unique treatments for each individual

B. There is no way to associate disease signatures with potential treatments

C. Using pathway analysis, we can find treatments that are known to target genes that are upstream in
the disease signature

D. Using pathway analysis, we can determine the odds that a potential treatment will work before we
know what molecule it interacts with

Digital Health Innovation System Integration Assessment: A methodology for streamlined early-stage
evaluation

--Bailey Griffin

S04-5: While there is increased interest and investment in digital tools to improve the outcomes,
experience and cost of care, the potential impact of an early stage digital health tool can be difficult to
ascertain. Our abstract outlines our process and framework for assessing early stage digital health tools
and the results of our retrospective review of recommendations given to companies to better align their
product with system needs. The most common recommendation given to companies included:

A) Do better market research and hire consultants to help you

B) Identify individuals who have purchasing power at an organization

C) Properly identify the appropriate level of the health system at which the product is likely to be
purchased and increase value to the primary payer

D) Focus only on the end needs of patients. The needs of providers and other users are irrelevant since
we’re all most interested in improving the patient experience.

40 | P a g e
S04-6: Our Digital Health Innovation System Integration Assessment framework is focused on providing
individuals and organizations with:

A) An early stage assessment/evaluation of digital health tools to better allocate scarce resources.

B) A basis to skip validation of a digital health tool via rigorous scientific inquiry.

C) An alternative to the regulated procurement processes at an organization.

D) A way to feel engaged with the small to medium sized enterprises in digital health.

S05: Presentations - Clinical Decision Support Embedded Predictive Models

Validating Clinic Workflow Models for Prediction

--Rafael Fricks

S05-1: Which of the following statements best describes over-fitting?

a. As model complexity increases, model predictions show decreased error in validation sets while
training set error increases.

b. Over-fitting can be determined without the use of independent data sets.

c. An over-fit model provides an optimistic estimate of model quality when evaluated against the
training set.

d. Hypothesis testing (i.e. Kolmogorov-Smirnov Test) cannot be used to detect over-fitting.

S05-2: Choose the answer that best completes the statement: “K-fold cross-validation…”

a. reduces the influence of any particular set of data.

b. should divide data into five subsets.

c. requires at least 100 observations.

d. should be used over other model assessment techniques such as bootstrapping.

e. is a feature introduced in MATLAB 2017a.

41 | P a g e
Patient-Specific Explanations from Risk Prediction Models

--Amin Tajgardoon

S05-3: Machine learning models are being increasingly developed to predict clinical outcomes such as
mortality, morbidity, and adverse events. Among the following choices, which one enable clinicians to
engender trust in a sophisticated machine learning model with excellent prediction power?

A. The model achieves a high area under the ROC curve (AUROC)

B. Satisfy the assumption that training data samples are independent and identically distributed

C. Provide simple and reliable patient-specific explanations for each prediction

D. The model has a very low generalization error

S05-4: Which of the following definitions best describes “prediction explanation” in the context of a
machine learning model that predicts a clinical outcome?

A. Present to the user the model’s prediction score for a clinical outcome

B. Present an interpretation of the model to the user in terms of structure and parameters

C. Allow the user to select a model with the best accuracy in predicting the clinical outcome

D. Provide the user with an understandable interpretation of the prediction for an individual to
whom the model is applied

Leveraging Electronic Health Record Data to Predict Surgery Cancellation

--Nick Pratap

S05-5: In this study we implemented machine learning algorithms to explore variables from patient
records to predict day-of-surgery cancellation. Overfitting is one of the common problems in machine
learning which will decrease the generalizability of the model. Which machine learning algorithm has
the highest risk of overfitting?

A. Gradient boosted LR

B. Random Forest

C. Regularized logistic regression

D. Classification tree

42 | P a g e
S05-6: In this study we utilized a iterative step-forward method to do feature selection. What is the
primary objective of doing feature selection?

A. It identifies and selects key predictors to improve the interpretability of the model.

B. It increases the complexity of the model.

C. It creates new combinations of variables in order to reduce the dimensionality of the data.

D. It decreases the prediction performance of the model.

S06: Presentations - Informatics Application and Research to Enhance Surgical Practice

Adapting Technology into the Surgical Environment

-- Alex Langerman

S06-1: Information targeted towards surgical circulating nurses is best presented:

A. for the entire case at the start of a case (e.g., during time out or a preference card)

B. through a sterile interface to prevent surgical site infection (SSI)

C. in a mobile format that can be transported within and outside the operating room

D. with anatomic images to illustrate the choices between surgical techniques

E. for all of the cases of the day, to allow "anticipatory guidance"

S06-2: The most common and financially impactful result of poor communication regarding tools and
tasks in the operating room is:

A. tool damage and repair/replacement costs (RRC)

B. over-provisioning of surgical supplies

C. lapses in sterility from improper draping

D. reductions in mean turnover time (TTm)

E. over-staffing of surgical cases

43 | P a g e
EHR Needs of the Surgical Subspecialist: Insights from System Selection

--Saira Haque

S06-3: What functionality should the surgical specialist consider when selecting an EHR?

A. Reconciling long term meds from outside vendors

B. Presenting longitudinal data for chronic disease

C. Targeting information pertinent to surgical problem

D. Graphing of BMI, blood pressure and pulse over time

S06-4: When an EHR vendor demonstrates the system’s functionality, which points should be clarified?

A. the programming and coding employed in the programming software

B. the ability to adapt the system to clinical and administrative workflows

C. the ability of the system to withstand cyber-attacks from external sources

D. workflow of enterprise systems beyond the scope of the current project

Challenges of Electronic Health Record Transition for Surgical Services: A Qualitative Observational Study
of “Go Live”

--Megan Shroder

S06-5: In our study of surgical practices during an electronic health record transition, one of the most
common sources of problems and confusion was related to:

A. Admission orders

B. Phases of care

C. Operative notes

D. Inpatient documentation

44 | P a g e
S06-6: An important recommendation from our study of surgical practices during an electronic health
record transition at an academic medical center is dedicated training for:

A. Work shared across teams of residents, staff, and faculty

B. Emergency department to operating room admissions

C. Physician scheduling of procedures from the outpatient setting

D. Documentation for trauma resuscitation

S07: Panel - LEANing into the Quality Payment Program: Using Agile Strategies for Real Practice
Improvement

--Julia Skapik, et al

S07-1: Which is NOT one of the four reporting categories in the Quality Payment Program (QPP) Merit-
based Incentive Payment Program for individual providers and groups?

A. Advancing Care Information

B. Improvement Activities

C. Clinical Decision Support

D. Quality Measures

S07-2: Lean is a tool that has been around the manufacturing industry for a long time and only more
recently has it been applied to healthcare organizations. This is partially due to the healthcare
reimbursement system changing from fee for service to pay for value incentivizing more and more
organizations to undergo Lean transformations. Taking on a Lean transformation can provide an
organization significant gains in productivity, quality, and cost reduction but requires a cultural shift.
This type of a shift is driven by leadership and needs to have the right principles in place to transform
the culture. Which of the following are the most appropriate management principles to create a Lean
culture?

A) Value to the customer, open communication, participate and contribute, pursue excellence, share
knowledge, keep it simple.
B) Be a leader, exceed customer expectations, continuously improve, keep it simple, seek perfection,
respect for every individual.
C) Attitude of continuous improvement, value creation, unity of purpose, respect for front-line workers,
visual tracking, flexible regimentation.
D) Create value at every step, quantity over quality, keep it simple, seek relentless perfection, share
knowledge, listen and communicate.

45 | P a g e
S07-3: Capturing and extracting valid and reliable data from certified EHRs requires iterative
improvements to the certified EHRs, workflow, data entry screens and strategies to capture valid data
based on the measures requirements and specifications. Which of the following is most likely the case in
terms of iterative effort?

A. 80% of the effort is by the vendor to provide modifications within the certified EHR/20% is with the
clinical side to modify workflow to accommodate eCQM data capture

B. 60% of the effort is by the vendor to provide modifications within the certified EHR/40% is with the
clinical side to modify workflow to accommodate eCQM data capture

C. 50% of the effort is by the vendor to provide modifications within the certified EHR/50% is with the
clinical side to modify workflow to accommodate eCQM data capture

D. 20% of the effort is by the vendor to provide modifications within the certified EHR/80% is with the
clinical side to modify workflow to accommodate eCQM data capture

E.10% of the effort is by the vendor to provide modifications within the certified EHR/90% is with the
clinical side to modify workflow to accommodate eCQM data capture

S07-4: Which of the following strategies is most likely to improve problems with electronic quality
measure data capture and reporting?

A. Using only data from a clinical registry

B. Manual abstraction

C. Natural language processing in writing clinical notes

D. Use of a clinical data model associated with a standardized data warehouse

46 | P a g e
S08: Presentations - Bridging Analytics

POKE-R: Reducing Patient Harm

--James McGlothlin

S08-1: After the analytics have been developed, one of the most important elements to a successful
project is a good deployment strategy for user adoption. When attempting to implement POKE-R in a
specific unit, what is of the following is a key element which will help us succeed?

A. Utilizing specially trained nurses

B. Having one source of truth for all types of orders

C. Performing structured rounding

D. Meeting regularly with informatics professionals

E. Preventing duplicate orders in the EMR

S08-2: To develop a a project such as POKE-R, it is important to establish specific and attainable goals.
Which of the following is a reasonable and significant goal for a POKE-R project.

A. Reduce surgeries

B. Reduce lab draws

C. Reduce medication administrations

D. Change nutritional orders

E. Reduce mortality

47 | P a g e
The Clinical Informatics Fellowship Experience: A Qualitative Analysis (qACIF)

--Benjamin Orwoll

S08-3: You are a first-year resident and you find yourself very interested in how the information and
computer systems at your institution can be leveraged to improve care. A faculty member mentioned
the clinical informatics fellowship, but you are not sure what kinds of qualifications you might need to
apply successfully. Of the following, which profile most closely approximates the experiences prior to
fellowship from the cohort of fellows surveyed in the qACIF study?

a. Quality improvement scientists with extensive research experience who want to apply informatics
principles in their work

b. Physicians with bachelors or graduate degrees in computer science intending to apply machine
learning algorithms in health care.

c. Young physicians with interests in technology who are frustrated with the current state of health IT
and desire to be leaders toward future system change.

d. Physicians with IT leadership background who need dedicated training in clinical informatics to obtain
board certification

e. Entrepreneurial physicians who have previous business experience and need to gain a foothold for
their devices in the health information ecosystem.

S08-4: Clinical informatics fellowships, as they are currently structured, do not specify training in a
specific medical specialty or specialties as a prerequisite to application. Thus, clinical informatics can be
a subspecialty of any medical specialty. Based on the results of the qACIF survey, which of the following
medical specialties is most significantly underrepresented among clinical informatics fellows?

a. Pediatrics

b. Emergency Medicine

c. Pathology

d. General Surgery

e. Internal medicine

48 | P a g e
Interdisciplinary Development of Palliative Care EHR Decision Support

--Ruth Bush

S08-5: Palliative care is associated with more efficient resource use and decreased medical costs. The
targeted technology available in the EHR, including clinical decision support could facilitate screening for
earlier identification of patients in or about to be in distress; to provide more appropriate referrals to
specialized care; and to facilitate more rapid communication among patients, caregivers, and clinicians.

Which of the following is a good working definition of palliative care:

a. A short-term approach to providing pain control.

b. Targeted patient education designed to prevent individual chronic disease development.
c. Medical care, which prevents or treats symptoms and side effects of disease and treatment and is
designed to be patient and family-centered.
d. Engagement of the patient in their health decision making through technology such as patient portals
and tracking devices.

S08-6: Which of the following statements most accurately reflects the results of the palliative care
trigger list feasibility study:

a. The final algorithm version had several hundred variables to ensure it captured every possible
individual needing palliative care.

b. The final algorithm needed to be validated with manual chart review.

c. The trigger list had poor sensitivity; patients retrospectively identified were not likely to have been
seen prospectively by the palliative care team.

d. The algorithm relied primarily on laboratory testing as the basis for the clinical decision support.

S09: Presentations - Clinical Decision Support - Trends in Optimization

Optimize Up: Using Analytics for Clinical Decision Support Optimization

--Anwar Mohammad Sirajuddin

S09-1: Which one of the following is NOT a potential outcome of Clinical Decision Support (CDS) alerts?

A. Creation of a continuity of care document

B. Prevention of adverse drug events
C. Improvement of reimbursement and regulatory compliance
D. Provide treatment recommendations

49 | P a g e
S09-2: In order to optimize Clinical Decision Support (CDS) alerts, Memorial Hermann identified
potential areas of inefficient alerting to the clinical users. The main two criteria they reviewed were:

A. Low frequency alerts and high alert override rates

B. Low frequency alerts and low alerts override rates

C. High frequency alerts and low override rates

D. High frequency alerts and high alert override rates

An Evaluation of Clinical Order Patterns Machine-Learned From Clinician Cohorts Stratified by Observed
Versus Expected 30-Day Mortality Rates

--Jason Ku Wang

S09-3: With the emergence of data-driven clinical decision support tools, in particular those that
aggregate the collective expertise of clinicians and make recommendations based on patterns learned
from historical electronic health record data, defining a robust external reference standard is necessary.
For the case of promoting compliance with clinical practice guidelines, which of the following proves to
be the largest barrier to defining a reference standard, even for common admission diagnoses?

A. There are too few clinical practice guidelines in the literature to assemble a robust reference standard

B. In translating clinical practice guidelines for a given admission diagnosis into a finite reference
standard, subjectivity and interpretation are required

C. There is no systematic way for multiple clinicians with different opinions to reach a consensus on a
single reference standard

D. Clinical practice guidelines derived from more reputable literature source should be assigned more
weight than those derived from less reputable sources

50 | P a g e
S09-4: Clinical order patterns derived from data-mining electronic health records can be a valuable
source of decision support content. However, learned patterns can be compromised by underlying
provider experience as clinical orders of less experienced providers (trainees) can exhibit greater
variability and worsened patient outcomes than those of experienced providers (attendings). Based on
the conclusions of this podium abstract, which choice of training data will invariably yield the greatest
compliance with clinical practice guidelines?

A. Training indiscriminately on all available data

B. Always training on a cherry-picked subset of the most expert clinicians

C. Always weeding out data generated by less experienced clinicians (e.g. trainees) to improve model
accuracy

D. No single approach will invariably yield the best outcomes

Developing a Learning Electronic Medical Record System

--Shyam Visweswaran

S09-5: A learning EMR identifies and highlights EMR data that is relevant for a given patient and clinical
context. Which of the following can be enabled by such a learning EMR?

A. Faster access to patient data

B. Increased responsiveness of EMR to mouse clicks

C. Reduced time needed to assess the patient’s condition

D. More coherent display of patient data

S09-6: A learning EMR discovers what type of patterns from the EMR database?

A. Medical appointment patterns

B. Patient data access patterns

C. Patterns of comorbidities

D. Medication prescription patterns

51 | P a g e
S10: Presentations - Collaboration, Pathways and Training in EHR Implementations

Using a Cloud-based Collaboration and Dissemination Platform to Support the Implementation of an

Evidence-based Clinical Pathway Program Across a Multi-Entity Academic Healthcare System

--Emilia Flores

S10-1: Which of the following best describes a clinical pathway?

A. A graphical overview of a process

B. A document that contains criteria for guiding diagnosis and treatment of diseases or conditions

C. A method for translating evidence into local work structures and processes

D. A document that aims to standardize care for a specific clinical problem in a specific population

E. Structured and multidisciplinary plan of care, used to incorporate evidence into local practice, that
has criteria-based progression, and aims to standardize care for a specific clinical problem in a specific
population.

S10-2: Research findings have shown that patients do not always receive evidence-based care and
treatment, despite the availability of evidence and published guidelines (Agency for Healthcare Research
and Quality (2016)). Clinical pathways are one method for integrating evidence into practice; however,
the development and implementation approach can have a significant impact on provider adoption.
Which of the following approaches for clinical pathway development and implementation is most likely
to have a positive impact on provider uptake?

A. Using a rigorous process and framework for clinical pathway development and dissemination,
including identification of a clinical owner, basing the pathway on a synthesis of current guidelines, use
of a multidisciplinary stakeholder group to review the evidence and adapt based on the local setting,
and identification of barriers and facilitators to dissemination and tailoring the plan accordingly

B. Use of technology to provide centralized and easy access to clinical pathways across all practice
locations

C. Incorporating access to clinical pathways in provider workflow

D. Use of smart alerts to identify relevant clinical pathways for a given patient

E. Using a rigorous process and framework for pathway development and dissemination, having a
centralized and easy to access resource for storing pathways, and use of technology to provide access to
relevant pathways in the provider workflow

52 | P a g e
Built-in Care Pathways and Optimization of Electronic Medical Record System for Efficient Throughput at
a High-Volume Academic Medical Institution

--Rika Ohkuma

S10-3: A surgical service line A in an academic medical institution investigated their current workflow to
find opportunities to reduce clinical variance postoperatively. They found that postoperative orders
varied by attending surgeons and ordering residents. A current clinical pathway was developed four
years ago, but rotating residents were not aware of the care pathway. Which is the most appropriate
next step to reduce variance of clinical treatment by ordering physicians?

A. More education sessions for ordering physicians to think and order more carefully

B. Develop order sets aligned to the care pathway and have them embedded in EMR

C. Develop a more detailed care pathway so that they can follow each process

D. More process alerts in EMR to raise awareness of the care pathway

S10-4: A surgical service line B investigated their current workflow to find opportunities to reduce
postoperative length of stay. They found that front line care givers are not aware of benchmark and
expected length of stay. Inefficient timely preparation caused a delay in hospital discharge or a transfer.
Which is the most appropriate next step to reduce unnecessary delay to discharge patients?

A. Start a discharge plan in EMR when patients become clinically ready

B. Start a discharge plan on the day of surgery and update an estimated discharge date daily in EMR

C. Set a target length of stay for each procedure and show an expected discharge date in a patient chart
to set a same expectation in a multidisciplinary team

D. Track overall service line length of stay monthly

53 | P a g e
Evaluation of a Peer Training Model for Electronic Health Record System Implementation

--William Lancaster

S10-5: How does the peer training model (PTM) differ from the traditional training model?

a. With the PTM, providers train other providers in their same or similar specialty as to how to use
the new technology.

b. With the PTM, certified trainers train all of the providers in how to use the new technology.

c. The PTM is less costly than the traditional model.

d. The traditional model allows providers to become more engaged earlier on in the
implementation process.

e. The PTM does not require additional time away from clinical activities.

S10-6: The majority of the peer trainer survey respondents identified with which of the following
statements:

a. I do not feel that my peers learned the system better because it was taught by a peer

b. I felt more prepared for Go-Live after having served as a peer trainer

c. Fewer problems were identified earlier as a result of using the peer training model

d. I sacrificed too much clinical time in order to serve as a peer trainer.

S11: Panel - Quality Measurement: Authoring and Implementing with Clinical Quality Language and
FHIR®

--Ben Hamlin, Anne Smith, Bryn Rhodes, Carol Macumber

S11-1: Clinical Quality Language (CQL) is an HL7 standard authoring language that:

a. Defines relationships between patients and clinical concepts

b. Uses data from electronic health records or health information technology systems to measure health
care quality

c. Allows logic to be shared with both quality measures and decision support

d. Contains quality data for one patient

54 | P a g e
S11-2: Data elements specified for clinical quality measures reference the Quality Data Model (QDM)
which defines:

a. The source of the acquired data

b. The relationships between patients and clinical concepts in a standardized format

c. The codes that define each data element concept

d. SNOMED CT categories

S11-3: Clinical quality measures rely on standard terminologies for accuracy and reliability of the process
being measured. For patient reported measures and clinical observations, the preferred terminology is:

a. CPT

b. SNOMED CT

c. LOINC

d. ICD

e. HCPCS

S11-4: FHIR® (Fast Health Interoperability Resources) is a standard for exchanging healthcare
information electronically. FHIR® is useful for measurement because:

a. FHIR® is a relatively simple exchange model that maintains data integrity

b. FHIR® can be used as a standalone standard or with other existing standards

c. It is organized into modules.

d. It has patient resources.

55 | P a g e
S12: Presentations - mHealth Solutions for Screening, Communication, and Disease Management

Digital Drawing Platform for Early Dementia Screening

--Kelvin Tsoi

S12-1: Assuming that digitization of cognitive screening questionnaires is possible, which of the
following is the main benefit of digitized screening tests that can supply the dramatically expanding
elderly population?

A. Enhancement of screening accuracy

B. Easy collection of digital data of participants

C. Enabling self-administered assessment

D. Prevention of subjective judgments on the drawing images

S12-2: Which of the following best describes the difference between scoring methods for drawing tests
in paper-and-pencil form and scoring based on a digital drawing test?

A. Scoring methods for drawing tests in paper-and-pencil form are well-studied, but scoring based on
digital drawing test is not.

B. Scoring based on a digital drawing test considers participants’ drawing behaviors, but scoring
methods for drawing tests in paper-and-pencil form can only consider the final drawing image.

C. Scoring based on digital drawing test embraces advanced computing techniques is more
comprehensive than the traditional screening tests in paper-and-pencil form.

D. Scoring methods for drawing tests in paper-and-pencil form are more objective than the scoring
method on digital drawing test.

56 | P a g e
Increasing Efficiency and Enabling Innovation by Implementing Secure Text Messaging Application for
Communication and Coordinating Patient Care

--Neha Patel

S12-3: Which of the following communication tools are Health Insurance Portability and Accountability
Act [HIPAA]-complaint?

a. Pagers

b. Hospital-issued Smartphones for text

c. Hospital-issued Smartphone for voice calls

d. Hospital-issued Tablets

S12-4: CMS recently announced that texting patient orders would be allowed but if a hospital ensured
that the process is through a secure text messaging platform and complies with which of the following:

a. Secure sign-on process(s)

b. Encrypted messaging

c. Allowed use of text messaging (for example, short message service (SMS) text messaging)

d. Customized message retention time frames

Practical Apps: A Framework to Assess the Quality of Mobile Health Apps for Chronic Disease
Management

--Xinjie Yin

S12-5: Which of the following is one of the dimensions of the evaluation framework for mobile health
apps for chronic disease management?

A. Accessibility

B. Cost-benefit analysis

C. Sustainability

D. Patient networking

57 | P a g e
S12-6: Which poses the greatest challenge for evaluation development of patient-facing mobile
applications?

A. Prediction of user commitment

B. Extreme expense involved

C. Rapid change in mobile platforms

D. Medical society endorsement

S13: Presentations - Healthcare Big Data Analytics

Governing Predictive and Prescriptive Models: Adding the Right People and Processes to Technology

--Farah Ahmed

S13-1: The use of predictive models in healthcare organizations has become more common in the last
decade. According to Becker’s Hospital Review, one of the key takeaways from the 2018 Healthcare
Information Management Systems Society Conference (HIMSS) was that predictive analytics seems to be
driving both operational improvements and clinical decisions (1). However, with the rise of these
modeling methods, new challenges have arisen with interoperability and transparency within healthcare
systems. While it is clear that predictive analytics has become significant in healthcare, what is the
biggest challenge in the deployment of such models?

a. Predictive modeling is not effective when used for small healthcare organizations due to the
complexity of the algorithm(s)

b. Predictive model implementation leads to complete replacement of current clinician workflows

c. Lack of oversight in implementation and minimal stakeholder involvement hinders acceptance and
use

d. Minimal validation techniques and guidelines are available to ensure scientific validity of the models

58 | P a g e
S13-2: The term “black-box medicine” is one that describes a shortcut to answering complex questions
without fully understanding the problems or solutions (1). This shortcut most often occurs to prevent
the dissemination of proprietary algorithms in predictive models. With more hospitals utilizing
internally and externally developed predictive models that follow “black box medicine,” what guidelines
should end-users follow when using these types of prediction models?

a. End-users should not endeavor to understand the algorithm due to clinical liability and patient privacy
issues

b. End-users should override or appeal recommendations made by predictive models when they have
sound reason

c. End-users should adapt and accept all outcomes made by predictive models that are scientifically
valid.

d. End-users do not need to be educated on the possible outcomes presented by the predictive model
since they are the subject matter experts.

Using a Validated Predictive Analytics Model for Early Detection of Patients with Sepsis

--Gordon Moyer

S13-3: You've created a predictive analytics model for sepsis. In your validation of the model, you
calculate a C-statistic of 0.5. How do you interpret this result?

A. The model is no better at predicting which patients will and will not develop sepsis, compared with
chance alone.

B. The model is good at predicting which patients will and will not develop sepsis.

C. The model is strong at predicting which patients will and will not develop sepsis.

D. The model always predicts which patients will and will not develop sepsis.

E. Based on this predictive model, 50% of patients will develop sepsis and 50% will not.

59 | P a g e
S13-4: You've created a predictive analytics model for sepsis. In your validation of the model, you
calculate a C-statistic of 0.85. How do you interpret this result?

A. The model is no better at predicting which patients will and will not develop sepsis, compared with
chance alone.

B. The model is good at predicting which patients will and will not develop sepsis.

C. The model is strong at predicting which patients will and will not develop sepsis.

D. The model always predicts which patients will and will not develop sepsis.

E. Based on this predictive model, 85% of patients will develop sepsis.

Progression of a Quality Scorecard Leveraging Informatics Tools

--Mark Connolly

S13-5: Which of the following is a key quality reporting best practice?

A. Excessive consultation with subject-matter experts

B. Develop a standard validation process to ensure quality and consistency

C. Provide access to underlying data to all report users to promote transparency

D. Do not release measure definitions to minimize the risk of the Hawthorne effect

S13-6: What is the role of subject-matter experts during metric development?

A. Oversee data capture and availability

B. Ensure value of metric to warrant resources for development

B. Taking all validation responsibilities

C. Develop plan that aligns with organizational goals

D. Secure necessary resources for developing metrics

60 | P a g e
S14: Presentations - Infrastructure and Frameworks for Successful Implementations

Application of an Evidence-Based Adoption Framework: Practical Implementation Tools for Clinical

Operations

--Blake Lesselroth

S14-1: You have been asked by your leadership to improve and standardize medication reconciliation in
an ambulatory clinic. In response, your team has developed electronic health record features to
improve medication history collection, documentation of reconciliation, and patient counseling. When
rolling out the new technology, which of the following approaches is most important about improving
staff adoption?

A. A robust message campaign (e.g., educational materials, posters) is a critical ingredient in an

implementation bundle.

B. Software should include patient-facing affordances that improve patient reliability including
medication images, plain language, and simple response controls.

C. The implementation team should pay attention to the socio-technical fit of the product, including
workflow compatibility and implementation climate.

D. Executive sponsorship and clinical champions are necessary and sufficient for promoting adoption in
large practices, managed care organizations, and public systems.

S14-2: You have recently implemented new electronic health record features designed to improve the
reliability of medication reconciliation in clinic. Nurses are expected to use new software to collect and
document patient history. Physicians are expected to respond to an automated reminder confirming
medications have been reconciled. Upon review of the monthly performance dashboards reports, you
realize that nurse charting has improved but physician documentation remains unchanged. What would
be the most appropriate next step to improve physician documentation?

A. Add a new hard-stop or modal alert that requires physicians to complete documentation before
proceeding.

B. Gather observational or time-motion data to better understand physician workflow and identify
potential barriers to reminder completion.

C. Consolidate the history collection and reconciliation tasks so that nurses can complete all
documentation concurrently.

D. Meet with department leadership and script out physician performance plans that include
reconciliation as a measure of effectiveness.

61 | P a g e
Creating the Data Infrastructure to Support Mobile Health and Telehealth Initiatives

--Christopher Hollweg

S14-3: Which of the following best describes an error induced by automation bias?

A) Dr. Smith is alerted by an automatic EHR message that his patient may be developing severe sepsis,
along with a button to order intravenous (IV) fluids. However, this patient also has heart failure, which
may worsen with IV fluids. Dr. Smith assumes that the alert is appropriate and orders the fluids. The
patient later develops fluid overload and is transferred to the intensive care unit.

B) Dr. Smith is about to order antibiotics for his patient with a urinary tract infection. He chooses
ciprofloxacin because that is what he is used to using for his patients, but did not notice that this
patient's previous urine culture grew organisms resistant to ciprofloxacin

C) Dr. Smith needs to discontinue the medication metoprolol for a patient in the EHR. However, he
mistakenly discontinues the medication "metoclopramide," which was located right next to metoprolol
in the EHR and looked very similar.

D) Dr. Smith is about to admit his 10th patient on a busy call day. When placing his admission orders, he
mistakenly signs them in an open chart in the EHR for another patient whom he had seen earlier.

S14-4: Which of the following is one of the largest challenges commonly addressed in clinical risk
prediction model studies involving EHR data?

A. Evaluation metrics

B. Informative presence

C. Lead time bias

D. Missing data

S14-5: Which of the following is a recommended goal when developing and implementing an electronic
decision support tool for medication reconciliation?

A. Anticipating the users’ needs

B. Conducting usability testing in different settings

C. Focusing on thoroughness rather than efficiency

D. Redesigning the existing clinical workflow to match the tool

62 | P a g e
Use Cases for a WiFi Connected Internet of Things Button in an Academic Medical Center

--Haipeng Zhang

S14-6: The BWH Dash button is an example of using Internet of Things (IoT) devices. IoT is best defined
as:

A. A group of related ontologies within a larger database housed on a public facing network.

B. A network of physical objects that contain embedded technology to communicate and sense or
interact with their internal states or the external environment.

C. Wearable technologies that have the capability to sense biometric data from the user and
communicate with one another and across a larger network.

D. Miniaturized single function computers that communicate via physical connections to one another.

E. Low cost, single board computers that can be programmed to perform various functions.

S14-7: The BWH Dash button uses AWS IoT 1-Click technology with a custom dashboard that allows for
easy provisioning and programming of BWH Dash buttons. How many functions can BWH Dash buttons
be programmed for?

A. 1 function - single press

B. 2 functions - single press, long press

C. 3 functions - single press, double press, long press

D. 4 functions - single press, double press, triple press, long press

E. 5 functions - single press, double press, triple press, long press, long press and hold

S15: Panel - New Approaches for Improving CPOE Safety: Indications-based Prescribing and CancelRx

--Gordon Schiff, Neri Pamela, Christine Cheng, Samantha Pitts

S15-1: Physicians have long been urged to place the indication for the medication on the prescription for
drugs they are ordering. With paper prescriptions, physician acceptance has been poor, but with near
universal adoption of electronic ordering (CPOE) there are renewed opportunities to promote utilization
of this feature. Integrating indications into prescribing could pave the way for a safer, more complete

63 | P a g e
continuum of care for the patient and save time during prescribing and related tasks. Which of the
following is another significant benefit of indications-based prescribing?

A. The indication will always be put on the medication label

B. Drug costs will decrease when indications are included

C. Patients will know the reason why they are taking each of their medications

D. Brand name drugs will be selected more often

S15-2: While including the indication on the prescription is not a new idea, many providers do not use
this feature on a consistent basis. It has not become standard practice partially due to poor CPOE design.
To successfully incorporate indication into the prescribing workflow, efforts need to focus on reducing
mouse clicks, providing help at the point of care, and minimizing alert fatigue while supporting
physicians’ autonomy and flexibility. Which of the following design choice best explains how a prototype
successfully incorporates indications by allowing for physician autonomy while providing useful
guidance?

A. Allow providers to search for a drug only and do not require an indication

B. Allow providers to start with a drug or a problem/indication and see drug suggestions

C. Allow providers to search for a drug only, enter an indication and see alternative drug suggestions

D. Allow providers to search for a problem only and see drug suggestions

S15-3: Associating indications with a drug order may help prevent medication errors associated with
look-alike-sound-alike (LASA) drugs. Which of the following best describes the utility of associating high-
level indications concepts with drug pairs on the Institute for Safe Medication Practices’ list of
commonly confused drug names?

a. The majority of drug pairs that were comprised of antineoplastic drugs could be differentiated using
high-level indication concepts.

b. The majority of drug pairs that were comprised of antibacterial drugs could be differentiated using
high-level indication concepts.

c. More than half of all the evaluated drug pairs could be differentiated using high-level indication
concepts.

d. Only 20% of all evaluated drug pairs could be differentiated using high-level indication concepts.

64 | P a g e
S15-4: The National Council for Prescription Drug Programs’ SCRIPT standard supports electronic
functionality for sending prescription cancellations from electronic health records (EHRs) to pharmacies,
known as CancelRx. However, adoption of this functionality has been limited. Which of the following
presented the greatest challenge to pilot, rather than enterprise-wide,implementation?

A. Impact on pharmacy workflow

B. Impact on prescriber workflow

C. Restricted functionality of prescription cancellation in pilot implementation

D. Buy-in from key stakeholders

S16: Presentations - Care Coordination and Communication

Improving care coordination for children with autism spectrum disorder: defining care team members
information needs

--Julia O’Rourke

S16-1: Autism spectrum disorder (ASD) is a neurodevelopmental disorder diagnosed in childhood that
affects individuals throughout their lifespan. (1). As many as one in 50 children in the U.S. between the
ages of six and 17 years have a diagnosis of ASD (2), which reflects a 200-fold increase compared to the
estimates of four cases per 10,000 in 1966 (3). Individuals with ASD experience lifelong difficulties in
social interaction, communication skills, behavior, and learning.

Care for children with ASD is complex, involving on average seven different types of interventions,
programs, and therapies (4), including state organizations, local schools, medical providers, in-home
behavioral therapy, and rehabilitation services, such as physical and occupational therapies. Children
with ASD also have increased medical needs due to comorbid mental and medical conditions including
sleep problems, seizures, gastrointestinal problems, and maladaptive behaviors (5, 6). Current
treatment delivery model for children with ASD is fragmented with treatment team members delivering
care at different entities without any systematic collaboration or communication.

To better demonstrate the roles or different care team members working with a child with ASD, we
describe treatment of a hypothetical school age boy John, who attends a public elementary school. In
school John has several care team members assisting him. His teacher and school aid are working with
him daily in the classroom; his occupational therapist focuses on sensory difficulties during a weekly 45-
minute session. His physical therapist helps with building strength and endurance during another weekly
45-minute session. His speech therapist sees him three times per week and helps with articulation
difficulties. Finally, the school’s behavioral therapist sees John two times per week to teach pragmatic
and social skills. At school, John has an Individualized Educational Plan (IEP) plan.

65 | P a g e
John’s parents also bring him to a private speech therapist and occupational therapist for weekly
sessions. In addition, John sees a psychologist for weekly sessions outside of school, and another
behavioral therapist who helps with some of the troubling behaviors at home. John is also followed by a
psychiatrist, who monitors his psychotropic medications. John sees a neurologist for sleep and seizure
problems and gastroenterologist for constipation. In this example, all care team members work with
John to help him and his family to achieve a better quality of life and help him learn skills to live as an
independent and productive adult. John has caregivers at school, at home, in private offices, and in
several medical facilities, but they work independently and do not communicate across their practices
and locations.

These treatments are fragmented and it is not clear whether the therapies being delivered are the
optimal choice for John and his family.

What is the most likely care coordination scenario?

A. John has a Medical Home at his pediatrician’s office, where a case manager (social worker or nurse)
coordinates his medical care and services with other care team members

B. John has an Integrated Medical home at his psychiatrist’s office, where a case manager coordinates
his mental health care and services with other care team members

C. John has no care coordination activities

D. John’s mother performs all care coordination activities. To do this, she has had to reduce her hours
and now works part-time.

S16-2: Care coordination integrates services across settings and supports interfacing among multiple
care systems and care providers, including the following: medical, social and behavioral professionals;
the educational system; home care agencies; allied health professionals; and families (1). In a recent
opinion piece titled “Health Information Technology and Care Coordination: The Next Big Opportunity
for Informatics?”, Dr. David Bates described the care coordination needs for complex patients who are
cared for by a group of medical providers: “This entire group needs to be able to communicate better
than they do today – to see the care plan, to message each other, and to have synchronous and
asynchronous conversations about goals, and progress and lack thereof” (2).

Dr. Halamka further highlighted the need for care coordination infrastructure in 2017 NEJM: “If we want
patients to be engaged to help reduce the burden of care coordination, care plan tracking, and
communication, we need modern tools that enable patients to interact with their providers using
devices and workflows that are already part of their daily lives. Many companies are now offering such
tools” (3).

However, care coordination tools will be useful within a collaborative care model that supports care
coordination. The collaborative care model relies on information systems and case managers to link
primary care providers, patients, and mental health specialists (4). The collaborative care model has
shown to be effective in over 80 studies for in adult population for the treatment of depressive and

66 | P a g e
anxiety disorders (4, 5). Collaborative care also improves outcomes for children and adolescents with
mental health problems (6).

The collaborative care model is based on the chronic care model developed by Wagner and colleagues
nearly two decades ago to guide health services towards better management of chronic diseases (7).
According to this model, improved patient outcomes result from the interactions between an activated
patient and a proactive healthcare team.

Factors that activate patients include support for disease self-management and enabling community
resources. Factors that activate healthcare providers include adequate information systems, decision
support, and delivery system design to support longitudinal care as opposed to episodic care.

What is the greatest barrier in implementing a collaborative care model for children with ASD?

A. Severe shortage of child psychiatrists in the United States

B. No adequate payment mechanism for supporting care coordination

C. Privacy laws (The Health Insurance Portability and Accountability Act of 1996 (HIPAA) and The Family
Educational Rights and Privacy Act(FERPA))

D. Difficulties integrating with EHR systems

Challenges to Care Coordination for Children with Special Health Care Needs: Potential Solutions

--Pooja Paode

S16-3: Current efforts to improve care for children with special health care needs (CSHCN) are highly
medicalized and designed through the lens of providers, despite the fact that parents are responsible for
the vast majority of the care coordination for CSHCN. In response to this issue, we created a multi-
sector, parent-centered Stakeholder Coalition dedicated to identifying solutions. The Coalition identified
communication, community collaboration, family engagement/advocacy, and systems issues/technology
as key areas for improvement for the care coordination of CSHCN.

How should we consider the interaction between systems issues/technology and the other three areas?
We should…

A. come up with solutions for the four areas separately, then evaluate how they interact.

B. prioritize the development of technological solutions, such as applications, in each of the three other
areas.

C. consider solutions that will address gaps in each of the three areas, then identify how technology and
systems redesign can enable those solutions.

D. prioritize development on technology and systems changes that simultaneously bring together all
three areas: communication, community collaboration, and family engagement.

67 | P a g e
S16-4: Parents of children with special health care needs (CSHCN) often simultaneously operate as care
coordinators, caretakers, and key decision-makers for their children. They experience a lack of
meaningful information exchange surrounding situation management, planning, monitoring, navigating,
and resource usage.

When does this problem cause the most adverse impact for parents caring for CSHCN? During…

A. provider-parent encounters

B. a variety of “tension points” throughout the child’s lifetime

C. the time that passes between clinic visits

D. searches for external resources to aid in the care of their CSHCN

Accuracy of Patient Self-Report of 90-Day Post-Discharge Healthcare Utilization and Complications:

Relevance to Value Based Care

--Benjamin Rosner

S16-5: University Hospital is a regional center of excellence for total joint arthroplasty (TJA), and as such,
it draws patients from a large geographic area. However, post-discharge, most of these patients return
to their home area. When complications or readmissions occur, events for which University Hospital
bears financial risk under value-based reimbursement models, University Hospital is often unaware. In
reading the literature, those at University Hospital leading the post-discharge process improvement
efforts have learned that up to 65% of complications and readmissions may occur at other institutions.
To get a more complete picture of when and where readmissions are occurring among their patients,
the hospital is considering surveying patients directly about readmissions occurring within 90 days of
discharge. Assuming an unambiguous readmission question worded in lay language, University Hospital
might expect patients to have what degree of accuracy in self-reporting readmission events?

A. Excellent (kappa≥0.8)

B. Very good (kappa 0.60-0.79)

C. Moderate (kappa 0.40-0.59)

D. Fair (kappa 0.20-0.39)

E. Poor (kappa <0.20)

68 | P a g e
S16-6: James is an analyst working for a large healthcare organization trying to understand the incidence
of 90-day post-discharge complications. He has done a literature review of papers quantifying the
accuracy of patient self-report for the presence of complications using metrics such as agreement and
kappa. He believes that he has sufficient information to bring back to leadership about overall accuracy
of patient self-report to make an informed decision about the organization's planned outreach to
patients on the topic. What should he have done differently to avoid omitting an important element in
his assessment?

A. He should have looked for papers using "concordance" as a metric rather than "agreement" and
"kappa."

B. He should have limited his search to papers surveying patients at 1 year post-encounter or longer,
because recall accuracy improves with time.

C. He failed to consider the importance of also measuring accuracy of self-reporting for the absence of
events

D. He should have limited his search to papers in which analysis was limited to patients responding to
surveys, and not including analysis of non-responders (non-responder bias).

Thursday, May 10th

S17: Presentations - Decision Support for Clinical Screening

The Power of Two: Rapid Screening and Intervening Combining clinical decision support (CDS) tools and
nurse-driven protocols to effectively evaluate patients for high risk conditions and potential interventions
at hospital admission

-- John McGreevey

S17-1: All of the following topics would be a good fit for the type of universal screening assessment
described in this presentation EXCEPT:

A. Pregnancy status

B. Capturing detailed oncologic history

C. MRSA status

D. Chlorhexidine bathing indications

69 | P a g e
S17-2: Effective screening of patients for MRSA status, pregnancy status and to identify indicators for
chlorhexidine can be potentiated when clinical decision support is combined with which one of the
following?

A. Problem lists

B. Medication reconciliation

C. Pre-selected orders in order sets

D. Nurse-driven protocols and human assessment

Cancer Patient Trial Matching Assisted by An Efficient Automated Pre-Screening Application

--Hung Le

S17-3: Cancer claims the lives of half a million Americans every year. One of the major bottlenecks in
the development of anti-cancer drugs is the lack of sufficient patient enrollment in clinical trials. Which
of the following will likely happen if the enrollment goal is not met?

A. Delay or even termination of clinical trials, which, in turn, amounts to substantial loss to the
pharmaceutical companies

B. The sponsor will ask the principal investigator to refund the investment to recruit patients

C. The sponsor can extend the time frame for enrollment indefinitely until patient enrollment goal has
met

D. The investigator will be able to use synthetic patients to add to the number of actual patients to
reach the number needed for enrollment

S17-4: At the department of Investigational Cancer Therapeutics (Clinical Phase I/II program), UT MD
Anderson Cancer Center, we have developed an automated patient pre-screening application to
significant reduce manual effort to 92% and increase enrollment rate. What eligibility criteria were used
in this automated application? (select all that apply)

A. Patient age

B. Tumor type or disease

C. Number of prior therapies

D. Molecular target

E. Performance status

70 | P a g e
StartSmartTM: Decision Support Technology for Prenatal Screening

--Bonnie Gance Cleveland

S17-5: Which of the following best describes the iterative IT development process used in the creation
of StartSmart, prenatal screening application, guided by the Davis’s Technology Acceptance model?

a) Review of the literature, focus groups with pregnant women, technology development using Davis
Technology Acceptance Model, patient and provider end-user testing, refinement

b) Focus groups with providers & patients guided by the SBIRT Model, prototype development, alpha
testing, beta testing

c) Prototype development using guidelines, focus groups, and expert consultation; alpha testing with
patients and providers; revisions; beta testing with patient and providers; refinement

S17-6: Which one of the following best describes the process for applying the Davis’s Technology
Acceptance Model to the assessment of the StartSmart Prenatal Screening Technology?

a) An iterative development of the decision support/patient education application with end users’ input
at each level

b) Assessment of usability and ease of use was included in each step of the development

c) Assessment included user input at each phase/ beta testing with patient interviews that addressed
acceptability, ease of use, and intention to change; provider surveys regarding acceptability and ease of
use.

S18: Presentations - Supporting Best Practices

Tracking and Managing Abnormal Mammograms with SMART

--Caitlin Cusack

S18-1: At the Veteran’s Health Administration, the SMART project is being implemented to improve the
ability to track clinical management against a single index abnormal mammogram via the use of a
“Breast Care Cascade.” Which of the following “opens” a cascade?

a. The text of a received mammogram report is entered into the EHR.

b. An order for a mammogram is placed.

c. A woman who is due for a mammogram presents for a preventive care visit.

d. The pathology report for a breast biopsy is entered into the EHR.

71 | P a g e
S18-2: Ms. Jane Doe presents for a routine preventive care visit to her primary care physician (PCP). She
is due for a mammogram that is ordered. When completed the mammogram is recorded as a BIRAD 0:
Incomplete. The radiologist discusses the findings with the physician covering for Ms. Doe’s PCP, who
orders additional views. The followup views are done the following day and reported as BIRAD 4,
suspicious abnormality. Both reports generate SMART alerts, one to the PCP, the second to the covering
physician. When the PCP views the initial BIRAD 0 alert, which SMART template is shown?

a. BIRAD 0 SMART Template.

b. BIRAD 2 SMART Template.

c. BIRAD 4 SMART Template.

d. Multiple Tests Please Summarize SMART Template.

Integrating Evidence to Inform Lab Test Selection into a Knowledge Management Framework

--Spencer DesAutels

S18-3: An academic medical center makes evidence-based local practice decisions across multiple
content domains on a frequent, recurring basis. New external knowledge, e.g., a new guideline or meta-
analysis, is shared through various institutional channels and eventually leads to each institutional
decision being made or revised. Documenting this underlying knowledge has the potential to promote
adherence reuse, while simultaneously establishing a basis for crucial ongoing decision maintenance. To
maximize efficiency of documentation, the institution should consider which of the following?

a. Creating a shareable document that provides a running tally of all decisions by a decision-making
body.

b. Creating a single record of the underlying knowledge and linking it to a unique record of each
decision.

c. Selecting a standardized citation format for each publication type.

d. Requiring all committees to use an approved communication channel for official communications.

S18-4:There are many sources that provide relevant information on lab testing for patient care decisions
and analytics. These sources are internal and external, and refer to testing at different scales and for

72 | P a g e
different purposes. Furthermore, sources are constantly emerging, changing, or being deprecated. To
make a fully informed decision, e.g., selection of lab tests to perform or lab tests to restrict to certain
clinical scenarios, it is often necessary to consult multiple resources concurrently. How can an institution
maximize use of these heterogeneous data sources?

a. Publish a list of approved resources.

b. Negotiate contracts with vendors to present data in institutionally-approved formats.

c. Invest in mechanisms to bring informational resources to the point of care.

d. Bring all applicable data together in a consistent internal format that references each data source.

Cleaning House: EHR Order Set Innovation and Transformation at Penn Medicine

--John McGreevey

S18-5: All of the following contributed to the success of the order set transformation process described
in this presentation except:

A) Specialty Lead engagement

B) Legacy Order Set analysis

C) Creation of a modular order set framework

D) Development of a standard order set layout and naming convention

E) Replication of all legacy order sets into the new EHR

S18-6: Based on this presentation, each of the following order set sections would be a suitable core
module except:

A) Admission

B) Labs

C) Vital Signs

D) Notify Physician

E) Activity

73 | P a g e
S19: Panel - Implementing and Spreading Patient-centered, Evidence-Based Technologies: The Fall
TIPS Story

Patricia Dykes, Denise Goldsmith, Ann O’Brien

S19-1: The Fall TIPS (Tailoring Interventions for Patient Safety) program is based on over a decade of
research supporting its effectiveness with reducing falls and related injuries in hospitals. Which of the
following are most likely to prevent a fall in patients with physiological risks for falling?

A. Universal fall precautions

B. A bed alarm

C. A fall prevention plan tailored to patient-specific risk factors

D. A fall prevention bundle

S19-2: Which of the following are most likely to support evidence-based fall prevention interventions for
all patients?

A. Universal fall precautions

B. Clinical decision support

C. A fall prevention bundle

D. Infobuttons in the EHR

S19-3: Which of the following is the strategy most likely to achieve successful implementation of Fall
TIPS into an EHR?

A. Hospital leadership support

B. Integration during the design/pre-implementation phase

C. Fall prevention workflow analysis

D. Front line staff involvement

74 | P a g e
S19-4: While the Fall TIPS program is currently used in over 100 hospitals in the US, and around the
world, there are challenges to spread that can be addressed by organization leadership. What strategy
has been used to improve long term effectiveness of the Fall TIPS program.

A. Regular fall prevention education

B. Regular leadership participation in safety rounds

C. Regular leadership seminars on fall prevention

D. Regular monitoring/feedback related to patient engagement in fall prevention

S20: Presentations - Safety: Root Cause Analysis, Virtual Care, and Medical Reconciliation

Verbal Orders: Root Cause Analysis of Co-signature Authentication Timeframes

--Thomas Powell

S20-1: During a visit from the Joint Commission, the inspection team asks for additional information
regarding verbal orders (VO) usage for the respiratory therapy department. Aside from the number and
percentage of verbal orders, which additional analysis can be performed to present a more detailed
overview of order procedures?

A. Perform a dual domain analysis of time/day for threshold based similarity

B. Parse the data to find patterns in VO receiver(s) and/or author(s)

C. Compare the respiratory therapy VO statistics against other departments in the health system

D. Compare the VO records against the health care system policy, describing the data points and
timeline to authenticate the validity/authenticity/adherence of the order.

E. Identify grouping/bundling of orders to differentiate clusters from single purpose VO

75 | P a g e
S20-2: When analyzing Verbal Order (VO) records within the EHR, which data element is often
overlooked?

A. Staff scheduling

B. Discharge timelines

C. Clinic type

D. Physician inbox time

E. Mode of VO delivery

Quality Standards for Virtual Care: The Roles and Responsibilities of Stakeholders for Advancing Care
Quality in an Online Environment

--Kevin Smith

S20-3: A virtual care (telemedicine) provider group has convened a clinical quality council to provide
guidance for ongoing clinical quality monitoring and management.

Which of the options below is most appropriate for establishing quality outcome metrics?

A. Start out by creating a new virtual care framework for measuring clinical quality.

B. Partner with a statistician to analyze virtual care claims data.

C. Establish a disciplinary action and remediation program for underperforming doctors.

D. Establish outcome measure targets for the most common virtual care visit types.

76 | P a g e
S20-4: A health care system wishes to improve appropriate antibiotic prescribing practices for the
management of acute upper respiratory infections in the virtual care (e-visit) setting by improving
clinician adherence to an evidence-based clinical guideline.

Given the above information, which of the following may be the most appropriate intervention to
improve adherence to the evidence-based clinical guideline?

A. Add a link to the Infectious Disease Society of America web site on the clinician screen.

B. Incorporate contextual clinical decision support which highlights pertinent patient history of the
current illness, a brief summary of the guideline criteria with actionable guidance, all within the clinician
workflow.

C. Provide a variety of pop-up reminders and alerts reminding clinicians of the importance of antibiotic
stewardship.

D. Disable e-prescribing functionality for antibiotic medications to prevent clinician over-prescribing.

S20-5:Which of the following is most true regarding the current state of clinical quality monitoring and
management among virtual care (telemedicine) providers and vendors in the United States?

A. The CDC has been adopted as the source of truth for virtual care quality standards.

B. Any virtual care provider or vendor who can demonstrate that they have a medical reference
available for the clinicians, such as UpToDate, is considered to be compliant with quality standards.

C. There are no established standards for measuring clinical quality outcomes in virtual care in the
United States.

D. The World Health Organization, in partnership with AHRQ, is considered as the source of truth for
virtual care quality

77 | P a g e
S20-6: The quality committee has a debate about how to approach clinical quality as it relates to the
relatively new and rapidly growing model of virtual care.

What is the greatest challenge for establishing clinical quality targets for virtual care?

A. The conditions treated via virtual care are vastly different than those treated in a clinic setting.

B. The challenges for establishing clinical quality targets are essentially the same no matter the method
of care delivery.

C. The potential limitations of telemedicine equipment and internet connectivity could make it more
difficult to deliver quality care.

D. The lack of research in this area makes it impossible to establish quality targets.

Using Qualitative Techniques to Classify Medication Reconciliation Errors

--Rod Tarrago

S20-7: According to the Institute for Healthcare Improvement (IHI), the medication reconciliation
process involves three steps. Which of these is NOT one of the steps:

A. Verification

B. Clarification

C. Communication

D. Reconciliation

S20-8: The Joint Commission states that the intention of medication reconciliation is to address all of the
following except:

A. Duplications

B. Expirations

C. interactions

D. Omissions

E. Continuations

78 | P a g e
S21: Presentations - Patient Generated Data, Interventions, and Telehealth

Patient Generated Safety Data in the Hospital Setting: Trust, Transparency and Learning

--Sarah Collins

S21-1. Which of the following related to safety reporting in the hospital is true?

A. Most recognized safety concerns and incidents are reported

B. Patients and families are not considered key stakeholders in safety reporting

C. Most recognized safety concerns and incidents are not reported

D. Safety data reported by patients and families provides little insights

S21-2: Which of the following is an identified sociotechnical requirement for patient safety reporting
most related to trust from patients?

A. Availability of a real-time clinician dashboard

B. Protected anonymity when submitting
C. Comparing data to other safety data sets
D. Clear responsibility for follow up

Building and Evaluating a Patient-Facing Intervention to Deliver Implantable Cardioverter-Defibrillator

Remote Monitoring Data to Patients via MyChart: Patient and Provider Perceptions

--Romisa Rohani Ghahari

S21-3: A healthcare system wishes to send personalized information to patients based on implantable
cardiac defibrillator (ICD) remote monitoring data. Survey data from a recent study showed that most of
the participants (86%) wanted to receive more information than they currently receive from the clinic
about their remote monitoring data through the standard of care letter. Which of the following is most
appropriate based on the findings from this study?

A. Patients would be most satisfied with a standard of care letter when compared to the ICD data
summary.

B. Patients found the standard of care letter easier to access and use compared to ICD data summary.

C. Participants found the mailed letter with the ICD data summary easier to access compared to the
electronic format.

D. Either method (mailed letter vs. electronic format) of providing the ICD data summary works well for
the patients.

79 | P a g e
S21-4: A healthcare system wishes to provide patients with their device information without increasing
clinical workload and having a negative effect on providers. Prior research reported that providers were
concerned that messaging patients with their device data could potentially increase the clinical
workload. We evaluated the effect of providing patients with their device information on clinical
workload and clinician attitude towards ICD data summary. Which of the following is most appropriate
based on the findings from this study?

A. Providing patients with their device information did increase healthcare utilization.

B. Providing patients with their device information did not increase the workload of clinicians from
increased patient questions (calls or PHR messaging).

C. Clinicians believed the ICD data summary was NOT easy for patients to understand and use for
managing their health.

D. Providing patients with their device information did increase the device-specific workload for
clinicians.

Predictors of average adherence to the use of home telehealth devices in Veterans with heart failure

--Jenice Ria Guzman

S21-5: Adherence to the use of home telehealth devices in Veterans with heart failure enrolled in the VA
Home Telehealth Program ________ over time.

a. decreased

b. increased

c. reversed

d. stayed the same

S21-6: It is important to understand adherence to the use of home telehealth devices because:

a. Adherence is complicated and needs to be monitored.

b. Home telehealth devices are expensive and should only be given to patients who will use them.

c. Poor adherence to home telehealth devices over time may be a potential reason for negative results
in prospective randomized controlled trials.

d. There needs to be a conversation with patients regarding expected adherence to the use of home
telehealth devices for outcome effectiveness.

80 | P a g e
S22: Presentations - Identifying Medication Risk

Building a pharmacist prioritization tool using pharmacy-specific data and readmission risk

--Samuel Kabue

S22-1: When building a universal readmission risk prediction model for use in pharmacy, it is important
to base the modeling strategy on:

A. Commercially available 'black box' prediction models

B. A single predictor such as age or sex

C. Predictors that are readily available in electronic medical records within many health
systems/pharmacy departments

D. Pharmacists have time to intervene on every patient and therefore have little need for risk prediction
models

S22-2: A pharmacy department or health system wishes to use our readmission risk prediction model
but does not have a COPS2 score, LAPS2 score or RxDxCG score readily available. Which if the following
predictors currently available to them can be used to in place of these predictors for the best
performing model?

A. Diagnosis groupings

B. Drug subgroups

C. Admission category indicators (i.e. emergency department versus medical)

D. Number of refills on each medicine

81 | P a g e
A data mining approach to detect medication order near misses associated with electronic order sets

--Ron Li

S22-3: Which of the following best describes an error induced by automation bias?

A. Dr. Smith is alerted by an automatic EHR pop-up alert notifying that his patient may be developing
severe sepsis and includes a button that places an order for 30cc/kg of normal saline. However, this
patient also has congestive heart failure with an ejection fraction of 20%. Dr. Smith, assuming that the
alert is appropriate, does not consider this patient's heart failure diagnosis and proceeds to order the
30cc/kg of normal saline. The patient later develops acute pulmonary edema and has to be transferred
to the intensive care unit.

B. Dr. Smith is about to order antibiotics for his patient who has urinary tract infection. He chooses
ciprofloxacin because that is what he is used to using for his patients, but did not notice that this
patient's previous urine culture grew organisms resistant to ciprofloxacin

C. Dr. Smith needs to discontinue the medication metoprolol for a patient in the EHR. However, he
mistakenly discontinues the medication "metoclopramide," which was located right next to metoprolol
in the EHR and looked very similar.

D. Dr. Smith is about to admit his 10th patient on a busy call day. When placing his admission orders, he
mistakenly signs them in an open chart in the EHR for another patient whom he had seen earlier.

S22-4: Which of the following functions would you typically fail to achieve in your design of electronic
order sets?

A. Save time for clinicians for placing orders

B. Promote adherence to clinical practice guidelines

C. Provide clinical decision support

D. Increase the number of billable diagnoses for a clinical encounter

Detecting post-approval Adverse Drug Events within EHR Data: A case study correlating Lipophilic Beta-
Blockers with the subsequent diagnosis of Parkinson’s Disease.

--Cyril Spiro

82 | P a g e
S22-5: A group of researchers are studying potential adverse drug reactions (ADR) related to
prescriptions of medications and subsequent diagnosis of a specific disease. The researchers first select
all the ADRs related to the specific disease detected by an ADR detection heuristic. Then the researchers
study the ADRs by looking at their odds ratios. The goal of the researchers is to shortlist the potential
ADRs for a subsequent detailed study. A few of the potential ADRs are related to a specific class of
medications. Previous research has shown that this class of medications, let’s call it “X class,” have a
high chance of interacting with other medications and cause complications in patients. The X class of
medications have also shown conflicting results in previous research studies which were studying the
ADR in question. Assume that a patient can be on X class medication along with other medications in the
potential ADR list and the researchers want to shortlist the remaining ADRs in the list.

Which step can the researchers take to most efficiently shortlist the remaining and offset the potential
effects of X class medications on other potential ADRs that were detected?

A. Recalculate the odds ratios of only the remaining potential ADRs for the particular disease in question
by excluding patients that were on X class of medications.

B. Stop the study until a consensus is reached by clinical trials on the ADR related to X class of
medications and the specific disease in question.

C. Continue shortlisting the potential ADRs without any change in the analysis strategy.

D. Start the entire analysis from the beginning by first removing the patients who were on X class of
medications from the data and then running the ADR detection heuristic again on entire available data.

S22-6: A group of researchers are studying a particular potential adverse drug reaction (ADRs) related to
a particular disease (disease Dx) and a particular medication (medication Rx) found by using data
obtained from an EHR system. The design of the study is that of a retrospective cohort study. On
investigating the data on the detected potential ADR by using statistical tests and running confounding
analysis, the researchers find that all the statistical tests are pointing in the direction of the potential
ADR being an actual ADR. What can the researchers conclude from the results of the statistical tests?

A. There is a causal effect of the prescriptions of medication Rx to patients leading to them developing
the disease Dx. No more analysis or studies are necessary to confirm the findings

B. There is strong evidence to conclude that there is a correlation between prescription of medication Rx
to patients and the subsequent development of disease Dx in those patients. Conducting clinical trials is
necessary to establish a causation.

C. No inference can be made from the statistical tests about either causation and correlation between
drug Rx and disease Dx.

D. There is strong evidence to conclude that there is a correlation between prescription of medication
Rx to patients and the subsequent development of disease Dx in those patients. Running further
statistical tests on the EHR data (observational data) will establish causation.

83 | P a g e
S23: Panel - Closing-the-loop: Transforming Medical Information into Action

--Chancey Christenson, Kenan Katranji, Gargi Schneider

S23-1: What do experts identify as the most challenging aspect of the implementation of
pharmacogenetics into the clinic?

A. Translation of genetic information into clinical action

B. Test cost, test reimbursement or other economic issues

C. Availability of high quality genotyping test (CLIA approved)

D. Electronic medical record use, such as application of CDS

E. Clinician and/or patient resistance

S23-2: What type of genetic information does the MyCode program at Geisinger use?

A. Whole Gene

B. Proteins

C. Exons

D. Enzymes

E. Introns

S23-3: Of the following which is the most challenging aspect of Natural Language Processing?

A. Tokenization

B. Parts of Speech Tagging

C. Dealing with ambiguity

D. Parsing of a sentence

84 | P a g e
S23-4: Which of the following is most likely to be stored as structured data?

A. Video

B. Images

C. Text

D. Audio

S24: Presentations - Patient Deduplication, Clinical Phenotypes, and Collecting Social Determinants of
Health

Developing Methods for Standardized Patient Cohort Deduplication Across the Carolinas Collaborative, A
Regional Data Research Network

Robert Bradford

S24-1: A Clinical Data Research Network (CDRN) has decided on a hashing algorithm that utilizes: Last
Name, Date of Birth, and Last 4 digits of the Social Security Number (SSN), where Date of Birth and Last
4 digits of the SSN are translated directly into text (with no formatting changes). Site A has the following
data for their patient:
First Name: John
Middle Name: A
Last Name: Smith
Date of Birth: 01/31/53
SSN: 000-0000-7890
State: NC

Which of the following options of data available at Site B would cause the de-duplication process to FAIL
(hashes do not align).

A. SSN: 123-456-7890

B. Middle Name: 'null'

C. First Name: "John A"

D. Date of Birth: 31-JAN-1953

E. State: FL

85 | P a g e
S24-2: One of the methods to success in developing the Carolina's Collaborative was the ability to test
the de-duplication process internally between two different data sources. If another institution were to
join the Collaborative but did not have a legacy EMR to use as an internal gold standard in testing the
de-duplication process, what would be the best validation approach to attempt first?

A. Implement the process without additional checks, using previous validation results from other sites as
justification

B. Use only dummy data as a validation mechanism

C. Use another internal patient-centered data source, such as claims or registry data

D. Obtain an IRB approval and execute a Data Use Agreement with a collaborating site to share patient
data for purposes of testing the de-duplication process

Practical Clinical Phenotype Definitions with SNOMED CT

--DuWayne Willett

S24-3: A health care system wishes to construct diagnosis groupers for a large number of clinical
conditions (clinical phenotypes) to support chronic disease registries and population health initiatives.
To streamline grouper construction and maintenance, they wish to explore ways to build diagnosis
groupers besides a one-dimensional list of terms. Hierarchy-based groupers are possible when using
SNOMED CT because:

A. SNOMED has parent-child ("is a") relationships between concepts that relate specific concepts to
more general concepts

B. SNOMED has attribute relationships between concepts that specify additional defining characteristics
of one of the concepts

C. SNOMED has fewer concepts than ICD

D. Clinical terminologies in EHRs are only mapped to SNOMED, and not to ICD

S24-4: Compared with groupers listing all individual codes, hierarchy-based groupers:

A. require more frequent updating when new codes or concepts are added or deprecated
B. more closely match clinical thinking about disease subtypes to include or exclude from a grouper
definition
C. take more time to construct and maintain
D. are more likely to be missing relevant diagnosis clinical terms

86 | P a g e
Supporting Disparities Research by Increasing the Collection of Social Determinants of Health Data
through EHRs

--Lauren Hovey

S24-5: To facilitate the standardized collection of REaL data, which of the minimum data sets provides
an optimal level of granularity?

A. OMB minimum standard

B. HHS race and ethnicity standards

C. CDC Race and Ethnicity Code Set Version 1.0

D. The optimal data standards are those that are fit for purpose and the context in which the data are
collected.

S24-6: Even when providers and/or health systems are motivated to collect and use REaL/SOGI data for
clinical purposes, their vendor products may not be optimized for the task. Conversely, vendors may
offer robust functionalities that are unintuitive or that providers are not well versed in using. For EHR
products designed to facilitate REaL/SOGI collection and use, which of the following is an optimal
solution to reducing provider burden during information capture?

A. Pre-populated fields related to gender/sex-specific history

B. Free text fields that allow providers to enter relevant information that may not be captured

C. Capture of the information in a pre-assessment (e.g., via portal or registration questionnaire)

D. Structured fields for REaL/SOGI that occur in a consistent place in the EHR

87 | P a g e
S25: Presentations - Methods for Enhancing EHR Usability

Categorization of EHR Design Problems Using Usability Heuristics

--Benjamin Duncan

S25-1: The main goal of the heuristic categorization methodology introduced here is intended to
accomplish which of the following?

A) Allows for grouping of the most significant issues in usability and to prioritize them based on
their effect on workflow

B) Make suggestions to users to improve the usability of the overall system

C) Have usability experts review a chosen interface and compare it against accepted usability
principles

D) Evaluators work through a series of tasks and ask a set of questions from the perspective of the
user.

S25-2: Why should workflow be studied in advance of an enterprise-wide EHR conversion?

A) Make improvements to the current clinical systems being used

B) Perform log file analysis on user interactions to develop a standard workflow for EHR-mediated tasks

C) Understand variation across different clinical sites and systems in the healthcare settings

D) Determine how well clinicians can perform their specific tasks through various EHRs based off
interactive behavior measures and human-computer interaction frameworks.

Disrupt and Adjust: Insights from a Contextual Inquiry of A Novel Documentation and Workflow Tool for
Inpatient Providers

--Subha Airan-Javia

88 | P a g e
S25-3: Your hospital identifies a safety event in which an immunocompromised patient with recent
history of extended-spectrum beta-lactamase resistant E. coli bacteremia was not started on sufficiently
broad antibiotics when she developed a fever overnight, and effective antibiotics were not instituted
until the error was caught 18 hours later. During root cause analysis, your team identifies that the
patient’s progress notes made clear that the patient should be started on vancomycin, imipenem, and
amikacin in the setting of a new fever, however the written handoff did not include this
information.Which of the following systems would be most likely to prevent this error?

A. Instituting a policy in which all febrile, immunocompromised patients are immediately transferred to
the intensive care unit.

B. An easily accessible documentation template for fever in immunocompromised patients.

C. Educating providers about the importance of situational awareness in the written handoff.

D. Populating the progress note & written handoff from a single source of truth

S25-4: A healthcare organization aims to increase the number of problems added to the problem list in
the inpatient setting. They design a system that forces clinicians to enter each problem on the problem
list before being able to document the plan related to that problem. Which of the following unintended
consequences is MOST likely?

A. Clinicians will document multiple problems under a single problem heading.

B. Case-mix index will go down, leading to reduced hospital billing.

C. Clinicians will enter fewer problems on the problem list for each patient.

D. The hospital will be at higher risk of Medicare fraud from inaccurate problem lists.

S26: Presentations - Workflow Enhancements for the HER

Improving Point of Care Information Workflow with EHR Embedded Search

--Bhrandon Harris

S26-1: On average, what is the number of unanswered clinical questions that clinicians have after a
patient encounter?

A. 0.25 questions per encounter

B. 0.5 questions per encounter

C. 1 question per encounter

D. 2 questions per encounter

89 | P a g e
S26-2: What are the two most commonly cited reasons that clinicians do not pursue answers to clinical
questions that have arisen during an encounter?

A. Lack of time and forgetting the question

B. Forgetting the question and a lack of urgency of the question

C. Lack of time and a lack of urgency of the question

D. Lack of time and doubt that an answer exists

E. Forgetting the question and viewing question as unimportant

Integration of Patient Controlled Analgesic Infusion Forms into the Electronic Medical Record (the eAIF
Project).

--MaryAnn Connor

S26-3: Opioids are a class of medication that confer great and immediate benefit but also carry
significant immediate and long-term risks. As pain increases and doses escalate, side effects ensue, and
an opioid rotation becomes necessary. What is the best method that ensures opioid conversion and
rotation are done safely?

a. Converting between opioids by hand calculation can be done in a standardized manner by clinicians

b. Electronic calculations performed using dedicated smartphone applications, or online resources

c. Use of a consistent conversion algorithm, and automated calculation embedded within the electronic
medical record.

d. Conversion first to morphine, a universal opioid, and then secondary conversion to another opioid.

90 | P a g e
S26-4: The Rapid Application Development (RAD) model should be used in software development:

a. To address gaps by disruption to remove the status quo legacy system and build from the ground up
focusing only on fundamentals.

b. Only when you are unable to use the preferred approach of Structured Systems Analysis and Design
Method (SSADM).

c. When there is limited time and availability for users (e.g., physicians and nurses) to provide input in
the design.

d. To allow users to interact with systems analysts and develop models and prototypes that represent all
system processes, inputs, and outputs using a multidisciplinary team with the model changing as
stakeholders weigh in.

Preoperative Nursing Workflow and Problem Solving

--Lu Zheng

S26-5: A health care system wishes to reduce the cognitive burden of clinicians working in preoperative
care to ensure the quality of patient-care delivery and enhance the patient safety. However, the
clinicians are not typically trained in human factors or cognitive science. They may not be able to
pinpoint the sources of burden. Which method is best suited to characterize cognitive burden in clinical
work?

A. Semi-structured interview

B. Video-screen capture of the EHR use

C. Ethnographic study of workflow

D. Think-aloud protocol

E. A multi-method approach that triangulates evidence from different sources.

S26-6: The PreOp nurse needs to help coordinate the activities of several surgery team members, such
as the anesthesiologist, the resident, the physician assistant or the surgeon, when they stop by and talk
to the patient during the PreOp assessment process. The nurse needs to seamlessly and efficiently
communicate to clinicians that some action is needed before the patient can be transferred to the OR.
Which strategy do you think is the best way to handle this coordination process with minimal disruption
given the constraints of the 30 minutes Preop assessment?

91 | P a g e
A. Write it on the blue sheet (checklist) and provide access to all clinicians.

B. Use a symbolic system (e.g., color coded magnetic strips on the exterior of the patient room)
that signals to team members that certain actions are needed.

C. Call, email or text the team member before they come to see the patient, so that they will know
before they enter the room.

D. Wait until they come and talk to the member directly at the patient bedside.

S27: Panel - Clinical Informatics Efforts to Combat the Opioid Crisis – Experiences from the Front Line

Barry Blumenfeld, Patrick Burns, James Huizenga, Steven Labkoff, Despina-Maria Michaels

S27-1: What major event transpired in 2016 that had a material effect on how primary care doctors
think about prescribing opioid analgesics:

A. The Trump Administration published a new law on the limits on daily opioid MMEs

B. The CDC published guidelines to inform primary care and other clinicians on the most appropriate
way to prescribe opioids

C. The FDA provided new guidance to prescribe abuse deterrent opioids whenever possible

D. A new formulation of an abuse deterrent opioid was approved for sale and use in the US

S27-2: With regard to the opioid crisis, the use of big data analytics is currently in use to:

A. Identify clinicians who are appropriately writing opioid prescriptions in order to single them out as
good practice examples

B. Identify patients at risk for opioid overdose by use of a predictive model

C. Enhance the Prescription Drug Monitoring Programs (PDMP’s), which is a requirement in all states

D. Support integrated systems, which are now wide-spread in Health Information Exchanges across the
US to help deter inappropriate opioid prescribing

92 | P a g e
S27-3: With regard to the Opioid Crisis, Clinical Decision Support tools are

A. Currently in wide scale use to deal with the opioid crisis

B. Currently not showing significant impact in cutting down the rate of opioid prescribing
C. Can bring together natural triggers inside the clinical encounter to present in-the-moment guidance
D. Generally easy to implement across a wide swath of the US due, in part, to the adoption of EHRs in
the last 5 years.

S27-4: The opioid crisis presents challenges for the entire healthcare ecosystem. Specifically, the use of
informatics tools with regards to the role of primary care clinicians should be to:

A. Do primary research to identify informatics tools that can be used to curb the opioid crisis
B. Provide specific data on every patient that has an encounter to ensure that registries and databases
have adequate information to draw upon when making policy decisions around opioid use
C. Devote a substantial part of their practice to work with research efforts from organizations like
SAMHSA, NIDA, and APA to help curb the opioid crisis
D. Leverage as many existing tools as is practical from their EHR vendors to integrate appropriate
screening and documentation during routine clinical care

S28: Presentations - The Learning Health System, Optimization and Leveraging Technology - All to
Improve Care

Implementation of a Learning Healthcare System for Sickle Cell Disease

-- Erin Crowgey

S28-1: A division within a health care system wishes to improve the meaningful use of data entered into
the electronic healthcare records (EHR). The majority of EHR systems handle coding and liability issues
well, but are not robust for clinical research and data visualization at the bedside. Which of the
following options is technologically possible to enhance the downstream use and analysis of data
entered directly into the EHR?

A. Have providers enter more data via free text in note sections.
B. Encourage providers to work independently on improving data entry into EHR.
C. Create a team consisting of providers and health informaticians to implement a process that supports
structured data entry at the bedside.
D. Request that the health informatics department implement solutions per each individual clinician.
E. Enhance the use of IC-9/10 coding for data analysis techniques.

93 | P a g e
S28-2: A common data model (CDM) enables a systematic analysis of disparate databases. The
Observational Medical Outcomes Partnership (OMOP) is one type of CDM that transforms data
contained within EHR databases into a common format, or data model, with common representation of
the data elements, terminologies, vocabularies, and coding schemes. Which of the following options is
solution for a department looking to enhance the meaningful use of EHR data?

A. Recommend providers focus on analytical techniques that can only be deployed directly in EHR
system and that are independent of any data transformation processes.

B. Create a custom CDM, or use a publicly available model, to map EHR data into a standardized format.

C. Require health informaticians to analyze EHR independently and provide PDF reports to providers.

D. Have providers within a department create their own individual lists of important data elements, and
capture them inside excel.

E. Recommend to the clinicians within the department that this is a health informatics issue and does
not require their effort or input.

The Veterans Health Administration’s Reproductive Health Project

--Caitlin Cusack

S28-3: Clinical Decision Support Systems (CDS) within Electronic Health Records (EHR) to reduce
medication errors are widely used. Medication CDS systems range from basic systems that conduct
drug-allergy, drug-interaction, and duplicate drug checking. More advanced systems incorporate data
from the EHR such as conditions, laboratory results, and gender. CDS that extends to drug-pregnancy
checking have limited clinical usefulness because of the lack of specificity of these alerts. One practice
used with these alerts that does not improve their clinical relevancy is:

a. Remove from the cohort for rule-checking those women who have system evidence an inability to
conceive, such as those with coding consistent with hysterectomy or menopause.

b. Limit alerts for women using highly effective contraceptive or those who are unable to conceive with
their current partner.

c. Build a rules engine that utilizes medications classified with the former FDA categories of C, D, or X.

d. Capture as discrete data pregnancy status including “yes,” “no” or “unknown”.

94 | P a g e
S28-4: At the Veteran’s Health Administration (VHA), the Reproductive Health project uses logic on the
backend to classify women as pregnant, medically unable to conceive, low likelihood of becoming
pregnant, high likelihood of becoming pregnant, and unknown status. The due date of the VHA’s
Reproductive Health (RH) Clinical Reminder Dialog is based on these classifications. Which of the
following women is not due now to have the RH dialog completed by her physician?

a. Ms. Smith age 38 is new to the VHA, uses an IUD for contraception, and presents for her first
preventive care visit.

b. Ms. Jones has a recorded estimated due date (EDD) in the system from 4 weeks ago, but has not been
seen within the VHA during her pregnancy.

c. Ms. Noonan has a history of a hysterectomy that has been recorded in the system.

d. Ms. Johnson has a completed RH dialog from 2 months ago. She is using Depo-Provera for
contraception, but is late for her next injection.

Leveraging Technology to Reduce Unnecessary Post Procedure Office Visits for Chemo Port Placement

--Christine VanZandbergen

S28-5: A provider wishes to replace a post-procedure in-person follow-up visit with a virtual visit.
Assuming the technology is in place to support virtual workflow, what is the most appropriate next
step?

A. Tell all patients to register for a patient portal account and start sending their wound sites photos.

B. Engage the appropriate clinical, regulatory and information systems partners to validate the workflow
and ensure appropriate support/monitoring.

C. Tell patients to take a photo of their wound and text it to you

D. Set up the visits as billable encounters

S28-6: Which of the following is a potential benefit of replacing post-procedure follow-up visits with
virtual visits?
A. Increased revenue through new billable services
B. Decreased infection rates
C. Improved follow-up with post-procedure wound checks
D. Increased provider satisfaction

95 | P a g e
S29: Presentations - Translating Clinical Decision Support to Practice

Translating evidence into practice: Effect of an Evidence-based Electronic Clinical Decision Support Tool
on Provider Ordering for Venous Catheters

--Emilia Flores

S29-1: The 5 Rights for clinical decision support (CDS) interventions (right information, to the right
person, in the right format, through the right channel, at the right time in the workflow) are a
framework for designing and deploying CDS interventions. Catheter-associated urinary tract infection
(CAUTI) is a common device-associated infection in the acute care setting. However, misdiagnosis of
catheter-associated asymptomatic bacteriuria as CAUTI can lead to inappropriate use of antibiotics and
related consequences, including medication adverse effects and antibiotic resistance. Appropriate urine
culturing is dependent on a clinician’s ability to recognize a specific set of symptoms. You notice an
increase in inappropriate urine culture ordering in your organization and wish to implement a CDS to
ensure that urine culturing is aligned with evidence-based recommendations. Which CDS type would be
the most appropriate for this clinical issue?

A. A predictive analytics tool to identify patients at high risk for CAUTI

B. A report containing a summary of patients that meet culturing criteria for CAUTI is available in a
reporting module or tab and is accessible to all clinicians
C. An order set that leads the provider through a series of questions based on CAUTI culturing criteria
and provides recommendations based on provider responses
D. Order set that defaults to a reflex-urine culture order if the patient has a urinary catheter. A reflex
urine culture order is one in which the lab will first conduct a urine analysis and if criteria are met (e.g.
urine analysis with >= 10-20 white blood cell count/hpf), the lab will then perform a urine culture to
assess for CAUTI.

S29-2: Zhang and Walji (2011) have proposed 14 evidence-based interface design principles that are
important to consider for any electronic health record (EHR) embedded clinical decision support (CDS),
as they can have a significant impact on user acceptance and satisfaction. (Middleton et al., 2013) Which
of the following interface features is an example of a design principle?

A. A CDS tool to promote antibiotic stewardship asks users to manually type in the patient diagnosis
associated with a medication order rather than select from a drop-down list

B. A CDS tool for vascular access selection displays relevant patient information and calculations within
the CDS interface to aid in decision making, thus avoiding having the user reference and remember
information from other screens.

C. A CDS development group organizes and displays data entry fields in all CDS tools alphabetically

D. A CDS tool for vascular access selection utilizes drop down lists rather than check boxes for the five
most common medications infused through venous access devices

96 | P a g e
Clinical Pathway Regret Risk Differences Between Affluent and Underserved Populations

--Robert Aarhus

S29-3: Decision Regret is a well-studied patient-centered outcome measuring the quality of the medical
decision-making process. Using validated query instruments, researchers have determined key
characteristics of decision regret in attempts to understand, retrospectively, the origin and trajectory of
patient dissatisfaction with those decisions that have produced disappointing or unexpected healthcare
results. In particular, many randomized controlled trials have examined effects of enhanced patient
information, or conformity to the level of physician participation in the process expected by patients.
According to the literature, which of the following is a characteristic of treatment decision regret?

A. It is negatively correlated with decisional conflict

B. It often increases over time in breast cancer cases

C. It sometimes differs according to the patient's race and insurance status

D. It is uniformly decreased through the use of decision aids

E. It is seen in a majority of complex cancer cases

S29-4: To prospectively assess the likelihood of future regret, a model has been developed to include
estimates of regret risk. A novel term in this context, regret risk is a statement of the likelihood of
future decision regret driven by both differences between patient expectation and outcome, as well as
the curse of the counterfactual narrative -- a seemingly certain and more positive alternative future that
might have happened had past choices been different.Key assumptions, based on decision regret
research, have driven regret risk formulation and expression. According to the presented model's
hypothesis, which of the following could explain differences in long-term cancer regret risk between
socioeconomic groups?

A. An emphasis on aleatory rather than epistemic uncertainty

B. A change in action space

C. Linear dependence on measurable decision regret

D. Cultural differences between patients and health care providers

97 | P a g e
S30: Panel - The Use, Usability, and Experience of Implementing a Systemwide Inpatient Portal (IPP)

Jennifer Hefner, Timothy Huerta, Ann McAlearney, Daniel Walker

S30-1: When implementing an inpatient portal that includes secure messaging between care team
members and patients, which of the following should be included in front-line staff training?

A. Allow care team members to engage with the tool from the perspective of the patient.

B. Inform care team members that this is the most effective way to communicate with patients now and
the health system will be using it exclusively.

C. Provide them with a training manual to read during breaks.

D. Frontline staff will not be using the tool as patients do so they do not need training in this manner.

S30-2: Studying patient and provider use of an inpatient portal requires analysis of log files - server-side
records of actions taken by a user - to assess how, when, and which features of the inpatient portal
were used. Which of the following is not an issue to consider when analyzing log files?

A. How to find the full-text of the electronic messages sent between patients and the care team.

B. The application may create administrative data (e.g., pushed data to ensure that the tablet does not
display stale data) that is unrelated to patient action, resulting in artifacts in the data.

C. The application itself does not offer an indicator of a Session and as such the research team must
construct an operational definition to represent a patient’s single continuous use of the tablet for the
purpose of using MCB.

D. It may take several days for a patient to get their tablet, and as such length of stay may suffer from
distortions associated with provisioning. As a result, patient length of stay (LOS) may not be an
appropriate framing for use, because of real-world demand on provisioning practices.

S30-3: What approach(es) to learning do users prefer to familiarize themselves with the inpatient
portal?

A. Free exploration

B. Video tutorial

C. In-person training

D. Blended – including free exploration, video tutorial, in-person training

98 | P a g e
S30-4: Which of the following theoretic constructs is not a valid approach to measuring use of patient
portal use.

A. Comprehensiveness

B. Coherence

C. Duration

D. Frequency

S31: Presentations - Analysis of Medical Data and Implication for Value-based Care

A clinical informatics consult service to use aggregate patient data at the bedside

--Saurabh Gombar

S31-1: The ideal clinical decision is supported by large multicenter randomized control trials. However,
less than 20% of clinical decisions are based on RCT data. Furthermore, an estimated 20% of clinical
decisions are not based on published evidence at all. Given this limitation, physicians have long wanted
to untap the electronic medical record (EMR) to quickly generate practice-based evidence relevant to
the clinical decision they are facing. To date which of the following has NOT been a limitation to rapidly
assimilating knowledge from the EMR?

A) Tools to build a cohort of patients who faced a similar clinical scenario are not widely available.

B) The multitude of data types in the EMR (ie ICD, CPT, Notes) require physicians to be proficient with a
multitude of medical nomenclature syntaxes

C) Analyzing large datasets is too time intensive for busy physicians

D) Physicians are uncomfortable using evidence generated from EMR data.

99 | P a g e
S31-2: Stanford’s approach to enable precision medicine through aggregate patient data utilizes a
consult service. This service is offered by a multidisciplinary team comprised of an informatics physician,
an EMR data specialist to create cohorts from the EMR, and a data scientist to perform statistical
analysis. Which of the following is NOT a reason utilize a consult model to generate aggregate patient
data?

A) The pathway from clinical question to useful information requires a multitude of disparate steps best
handled by a team with complimentary skills.

B) The consult model is a model that physicians are comfortable getting subspecialist information from.

C) An asynchronous model allows for quick turnaround time and can be scaled to ever increasing
demand.

D) A consult model is already billable as long as a note is generated for the patient.

100 | P a g e
The Winning Health IT Engagement Model of an ACO in a Value-based Setting

--Nathan Riggle

The presenter will provide two multiple choice questions at the end of the presentation.

101 | P a g e
S32: Presentations - Patient Portals: Challenges and Opportunities

Patient Portal Proxy Access for Adolescents: Preserving Confidential Care While Enhancing Parent Proxy
Access

--Margaret Punch

S32-1: The parent of a 13-year-old would like full access to their child’s medical information available
through an on-line portal at their local health center. What considerations should be given to this
request?

a. The parent always has full rights to access the minor’s medical record.

b. Minor patients can’t consent to their own health care under any circumstances.

c. In all 50 states in the United States there are varying statutes that allow minors to seek confidential
care and that information can legally be withheld from the parent or legal guardian.

d. In all 50 states in the United States 13-year-olds have the right to confidential care to access
contraceptive services.

S32-2: When enabling proxy access to the parent of a minor child which portal feature may need
restriction?

a. Demographics

b. Vaccine records

c. Records from birth to age 10

d. Appointment schedules

e. Allergies

102 | P a g e
Do Community Clinic Patients Have Internet Access for Patient Portal Use?

--Ruth Bush

S32-3: Previously identified barriers to patient portal adoption include differences in internet access,
computer literacy, and internet proficiency. This digital divide is one explanation for underserved
groups being less likely to use portals, with strong correlations to utilization by race and household
income.

In the multi-ethnic, lower income community clinic in this study, which of the following was significantly
associated with previous patient portal use?

a. Having a “smart” phone

b. Education level

c. Health status

d. Race/ethnicity

S32-4: When reporting past use as well as likely future use of the patient portal, participants reported
using all of the following functions of the portal EXCEPT:

a. Contacting provider

b. Renewing prescription

c. Obtaining health information

d. Scheduling appointments

Development of a Taxonomy of Secure Messages from an Inpatient Portal

--Cynthia Sieck

S32-5: On what dimensions can secure messages be classified?

A. Sender, Type and Topic

B. Type, Inpatient and Outpatient

C. Urgent, Non-urgent and Medical

D. Dietary, Medical and Non-medical

103 | P a g e
S32-6: Which of the following would be applications of a secure message taxonomy?

A. To improve the functionality of the secure messaging feature of the portal.

B. To help patients view their lab results.

C. To help care team prioritize responses.

D. A and C.

S33: Panel - The Clinical Informatics Fellowship: A Fellow’s Perspective

--Christian Dameff, William Lancaster, Chase Parsons, Melissa Van Cain

S33-1: Clinical Informatics Fellowships are accredited by which organization?

A. The National Institute of Health (NIH)

B. The Accreditation Council for Graduate Medical Education (ACGME)

C. The Association of American Medical Colleges (AAMC)

D. The Center for Medicare and Medicaid Services (CMS)

S33-2: Clinical Informatics Fellowship requires the completion of what primary residency?

A. Internal Medicine

B. Emergency Medicine

C. Pediatrics

D. Any ACGME-recognized primary specialty

104 | P a g e
S33-3: The responsibilities of Clinical Informatics fellows include:

A. Research in informatics, patient care, implementing Informatics solutions

B. Patient care, research in informatics, and working in a basic science lab

C. Implementing informatics solutions, research in informatics, and serving as a medical

librarian

D. Research in informatics without any dedicated time to patient care

S33-4: The duration of the clinical informatics fellowship is:

A. 1 year

B. 2 years

C. 3 years

D. 4 years

S34: Presentations - Tool Based Analytics

Institution of a Pop Up Reminder Decreases Opioid Prescriptions for Pediatric Patients in the Emergency
Department

--Melissa Van Cain

S34-1: The opioid epidemic has had a serious effect on the US population in recent years and this is true
of adolescent patients as well as adults. What is the approximate number of adolescents who are
addicted to prescription pain relievers?

A. 1,500

B. 15,000

C. 150,000

D. 1.5 million

105 | P a g e
S34-2: Massachusetts (MA) enacted a law in 2016 that providers are required to check the MA
prescription drug monitoring program before prescribing opioids. We achieved the biggest change in
opioid prescriptions with this intervention:

A. Adding a link to the chart

B. Reminder e-mails

C. Physician training

D. Pop-up reminder

E. Hard stop

Radiation Therapy Data Curation: Utilizing Interdisciplinary Expert Knowledge via an Iterative Tool-based
Process

--Thilo Schuler

S34-3: Manual data curation is currently the most common approach to retrospectively improving data
quality in clinical datasets.

Which of the following options is the greatest challenge with this approach?

A. Unsuitability for inter-institutional data exchange

B. Poor scalability

C. Poor accuracy

D. Lack of contextual knowledge

106 | P a g e
S34-4: Using a fully automated curation approach for large clinical datasets would be ideal, as it is
scalable and requires minimal clinician input. However, in most settings this is at the moment not
achievable.

Which of the following strategies is most successful in reducing the time imposed on clinical expert
curators?

A. Using non-expert curators

B. Focus on subset of data

C. Technology support

D. More training for curators

Developing a visual annotation tool to rapidly collect expert-annotated weight errors in pediatric growth
charts

--PJ Van Camp

S34-5: The clinical importance of a child’s weights being entered correctly into the electronic health
record is to:

A) correctly classify children as obese or malnourished

B) avoid potential medication dosing errors

C) avoid useless information being stored and taking up valuable disk space

D) ensure correct reimbursement from insurance companies

S34-6: What is a major benefit of creating a software application to let experts annotate a large dataset
instead of manual annotation without the use of a specific tool?

A) The process of gathering, annotating and saving the data can be sped up and it allows annotators
unfamiliar with data analysis software to interact with the data in an intuitive manner.

B) Data annotated digitally is more credible than data annotated by hand because the latter is more
prone to error

C) Once a tool has been developed, it can be used for many other projects that require different dataset
annotations

D) Only experts can annotate data manually, while with a tool anyone can

107 | P a g e
Closing Session and Keynote Panel: Building Clinical Decision Support that Works

--Adam Wright, Joan Ash, Dean Sittig, Bill Galanter, Sarah Collins

S35-1: A healthcare system wants to reduce the time it takes to clinical decision support system
malfunctions. Because the organization participates in several quality programs, the organization’s
leadership is particularly concerned about “false negative” situations, where CDS is supposed to fire, but
doesn’t. Which of these strategies is most likely to quickly detect “false negatives”?

A. User reporting through the help desk

B. Ongoing monitoring with anomaly detection

C. Routine clinical use of the system by informatics leaders

D. Sentiment analysis of override reasons

S35-2: A few years after implementation of an Alert type of CDS, or "rule," there are a series of
complaints about how it is functioning. The rule was designed by a requesting physician and IS staff. The
requesting physician explained what they were looking to have built in a few paragraphs and then it was
approved by the CDS committee and built by an IS analyst. A new IS analyst is having a hard time
determining if the rule is doing what it was intended to do as the paragraph describing the intent is not
clear and does not provide sufficient detail.

What is the best strategy to prevent this type of difficulty?

a) Use clinical informaticists to investigate potential rule malfunctions.

b) Require rule requests to be formatted as flowcharts with sufficient details to allow IS to interpret
unambiguously and future clinicians and IS to understand the intent.

c) Require the initial requesting clinician to test all CDS and be responsible for any future queries once
the CDS receives approval.

d) Check the override rate of the rule, if greater than 85%, turn the rule off.

108 | P a g e
S35-3: Leaders of a large healthcare system want to optimize their EHR by providing more effective CDS
with minimal malfunctions. Which of the following “people” strategies might help the most?

a) Set up a CDS committee of subject matter experts.

b)Involve clinical staff and encourage them to report malfunctions such as alerts not firing when they
should.

c) Assure that staff members in informatics, quality improvement, IT, and elsewhere liaise with one
another and with clinicians in multiple ways.

d)Hire additional staff to do measurement and monitoring because only they can consistently identify
malfunctions.

S35-4: You are leading the governance committee for clinical decision support at your institution. A new
request for development of CDS that is nurse-facing was submitted. The request is to implement an
evidence-based risk assessment for peripheral nerve damage in the operating room with guidance to
support appropriate interventions. Your organization leverages the Data-Information-Knowledge-
Wisdom (DIKW) Conceptual Framework to prioritize strategic development of CDS to support nursing
practice. Which component of the framework best categorizes this CDS request?

a. Data

b. Information

c. Knowledge

d. Wisdom

109 | P a g e
Tuesday, May 8th

ANSWERS, EXPLANATIONS, REFERENCES

WS02: Organizational Issues and Informatics: Translating Theory into Practice

--Kim Unertl, Saira Haque

WS02-1: When implementing health information technology, why is it a good idea to consider
organizational theory when developing an implementation plan?

A. Academics love theory

B. To help identify concepts that are relevant for informatics practice

C. It doesn’t make a difference

D. To inform your organization as a whole

Answer: B. To help identify concepts that are relevant for informatics practice

Explanation: Understanding theory is crucial to avoiding past mistakes and developing new insights.

Reference: Workshop presentation materials

WS02-2: What paradox led to the research that established Sociotechnical Systems Theory?

A. The Profit Paradox, that profits were elevated when new technology was implemented

B. The Happiness Paradox, that workers were happier when their work was going poorly

C. The Productivity Paradox, that despite improved equipment and better working conditions,
productivity went down

D. The Despair Paradox, that the better technology got, the less happy workers became

Answer: C. The Productivity Paradox, that despite improved equipment and better working conditions,
productivity went down

Explanation: The Tavistock Institute was tasked with identifying the reason that productivity and morale
decreased in the British coal industry in the 1940s, despite the fact that there was better equipment,
higher wages, and amenities like housing provided for workers. They referred to this situation as the
Productivity Paradox.

Reference: Fox W, Sociotechnical System Principles and Guidelines: Past and Present, Journal of Applied
Behavioral Science 1995;31(1):91-105.

110 | P a g e
WS02-3: Everett Rogers identified four major elements in how innovations are adopted in groups, as
part of his Diffusion of Innovations Theory. What were the four elements?

A. The Innovation, Communication Channels, Time, A Social System

B. The Bosses, The Workers, The Tools, The Organizers

C. Advertising, New Technology, Coordinators, End Users

D. Great Ideas, Software Developers, Sales Representatives, Time

Answer: A. The Innovation, Communication Channels, Time, A Social System

Explanation: Rogers analysis of adoption of new farming technologies categorized components of

innovation according to the four groups in the answer. Each component was a crucial piece of Diffusions
of Innovation Theory.

Reference: Everett Rogers. Diffusion of Innovations. 3rd Edition, The Free Press, 1983

WS02-4: There are five stages in Rogers’ Innovation-Decision Process. What is a major influence on the
Persuasion stage?

A. Whether people are really interested in making changes

B. How many people are interested in adopting the innovation

C. How people perceive the innovation

D. How long people have been using their past tools

Answer: C. How people perceive the innovation

Explanation: Although the other answers are potential influences, the major influence is how people
understand and interpret what the innovation is and how they can use it in their own work.

Reference: Everett Rogers. Diffusion of Innovations. 3rd Edition, The Free Press, 1983

111 | P a g e
WS02-5: In applying Lewin's Change Model to an organization that is considering a major health
information technology change, what strategies might be useful in "unfreezing" an organization?

A. Making sure the opinions of people who oppose the new technology are ignored

B. Avoiding contact with the intended end users of the technology

C. Firing anyone who disagrees with the choice of new technology

D. Involving all levels of intended users, from front-line workers to management, in the decision-
making process

Answer: D. Involving all levels of intended users, from front-line workers to management, in the
decision-making process to get their buy-in

Explanation: The "unfreezing" stage requires examining the environment and creating the conditions
needed for change. By listening to end users at all different levels of the organization in the decision-
making process, it builds organizational momentum for change and helps to get people who might be
hesitant on board with making changes.

Reference: Burnes B. Kurt Lewin and the Planned Approach to Change: A Re-appraisal. Journal of
Management Studies 2004;41:977–1002.

WS02-6: Why is the low point of the Change Curve Model sometimes referred to as the "Valley of
Despair"?

A. People like to whine

B. People worry for no reason when productivity goes down

C. When productivity dips because of new technology, people have a hard time understanding how
the technology might help them in the future

D. The implementation productivity dip does not actually exist, but is rather just a theoretical model that
has been disproven by years of technology implementation

Answer: C. When productivity dips because of new technology, people have a hard time understanding
how the technology might help them in the future

Explanation: The implementation productivity dip has actually been proven repeatedly in technology
implementations across organizations and fields. Depending on how a new technology is discussed
before implementation, end users might not understand that a loss of productivity is an expected part of
an organization coming back up to speed after a major change. This causes end users to lose confidence
in the technology and to worry that it's not possible to actually get back to where they were before the
new technology.

Reference: Elrod PD, Tippett DD, The “death valley” of change, Journal of Organizational Change
Management, 2002, 15(3):273 – 291

112 | P a g e
WS02-7: Many organizations refer to the day a new technology system is turned on as a “Go Live” date.
Why is this a term that we are better off avoiding?

A. Going “live” makes people think about death

B. Implementing technology occurs over time and “go live” implies it is a single point in time

C. Customers/patients might think we are talking about medical procedures

D. People are unnecessarily picky about language use

Answer: B. Implementing technology occurs over time and “go live” implies it is a single point in
time

Explanation: Technology-related change is a process, not an event. By using the term "go live," we imply
that a technology change is happening at a point in time. However, the moment a system is turned on is
just a fraction of the change new technology creates. It is unnecessary jargon that means something
different to technology people than to end users.

Reference: Presentation materials and class discussion

WS02-8: What is one reason that people worry about change?

A. No one worries about change – it’s just a part of life

B. Concerns about wasting time and effort

C. People who worry about change are irrational

D. People enjoy finding new things to worry about

Answer: B. Concerns about wasting time and effort

Explanation: Although sometimes people may appear to be irrational when it comes to change, often
these concerns are rooted in basic ideas, like worrying about loss of productivity. Given the large body
of evidence supporting the idea that new technology often leads to a productivity dip, people have
reasons to be concerned about wasted time and effort.

Reference: Lorenzi NM, Riley RT, Blyth AJC, Southon G, Dixon BJ. Antecedents of the People and
Organizational Aspects of Medical Informatics: Review of the Literature. JAMIA 1997;4(2):79-93.

113 | P a g e
WS03: How to Properly Onboard Doctors in the Age of Informatics & Digital Health

--Hamed Abbaszadegan, Nacchal Nachiappan, Shakaib Rehman

WS03-1: As a busy clinician, there is no time to learn a new EHR system in a three-day period. Patients
need to be seen. What is your approach to onboarding this busy physician:

A. Cancel Clinic
B. Reschedule patients
C. Find coverage
D. Block time 4 months in advance

Answer: D. Block time 4 months in advance

Explanation: Advance preparation is ideal for properly onboarding providers. Additionally, having the
providers have some initial workflow experience before attending class makes the experience stronger.

References: Kirk VG, Currie G, Dewey D, Howlett A, Lemiux L. Innovation in physician recruitment &
wellness: a pilot orientation program. J Pediatr. 2013 Apr 162 (4):664-665.e2. doi:
10.1016/j.peds.2013.01.001.

Wright M, Mankey CG, Miller BW. Improving upon the ‘July effect’: a collaborative, interdisciplinary
orientation for internal medicine interns. Med Educ Online. 2013 Dec 23; 18:23249. doi:
10.3402/meo.v18i0.23249.

WS03-2: As a long-time provider at your local facility, you never really learned how to use the EHR nor
keep up with new changing widgets and apps. How can you attend New Provider Orientation if you are
not new?

A. Displace the new provider

B. Schedule special sessions
C. Advanced planning of participants
D. New Provider Onboarding is only for new providers

Answer: C. Advanced planning of participants

Explanation: New Provider Orientation is not just for new providers. As technology changes and the EHR
platforms evolve, you can plan in advance and invite any current outstanding provider in your health
system.

References: Kirk VG, Currie G, Dewey D, Howlett A, Lemiux L. Innovation in physician recruitment &
wellness: a pilot orientation program. J Pediatr. 2013 Apr 162 (4):664-665.e2. doi:
10.1016/j.peds.2013.01.001.

114 | P a g e
Wright M, Mankey CG, Miller BW. Improving upon the ‘July effect’: a collaborative, interdisciplinary
orientation for internal medicine interns. Med Educ Online. 2013 Dec 23; 18:23249. doi:
10.3402/meo.v18i0.23249.

WS03-3: Every hospital has a unique set of features and health IT tools that they invested in. How is
curriculum established?

A. Given dynamic changes in the health IT world, curriculum can change at any time
B. Teaching is universal and content for onboarding is usually uniform
C. New digital toys should be brought on display for new providers
D. Once you know which providers attend, content should be tailored to fit

Answer: A. Given dynamic changes in the health IT world, curriculum can change at any time

Explanation: In the informatics and digital age, content changes as rapidly as the technology advances.
This is why you must be flexible in how you prepare onboarding.

References: Kirk VG, Currie G, Dewey D, Howlett A, Lemiux L. Innovation in physician recruitment &
wellness: a pilot orientation program. J Pediatr. 2013 Apr 162 (4):664-665.e2. doi:
10.1016/j.peds.2013.01.001.

Wright M, Mankey CG, Miller BW. Improving upon the ‘July effect’: a collaborative, interdisciplinary
orientation for internal medicine interns. Med Educ Online. 2013 Dec 23; 18:23249. doi:
10.3402/meo.v18i0.23249.

WS03-4: Shorthand and macros can make any process faster. Given EHRs have more than one way to do
the same thing, what approach should be taken when onboarding?

A. Show all methods of shortened processes

B. Discuss approaches to macros and let the providers decide what is best
C. Only show what works for the majority
D. Discourage using quick methods for processes

Answer: B. Discuss approaches to macros and let the providers decide what is best

Explanation: There are many ways to do the same things in all aspects of life. EHRs and informatics
platforms are no different. This is often a robust discussion with showcasing of only a few methods. Each
provider can subsequently decide what their preferred process is.

115 | P a g e
References: Kirk VG, Currie G, Dewey D, Howlett A, Lemiux L. Innovation in physician recruitment &
wellness: a pilot orientation program. J Pediatr. 2013 Apr 162 (4):664-665.e2. doi:
10.1016/j.peds.2013.01.001.

Wright M, Mankey CG, Miller BW. Improving upon the ‘July effect’: a collaborative, interdisciplinary
orientation for internal medicine interns. Med Educ Online. 2013 Dec 23; 18:23249. doi:
10.3402/meo.v18i0.23249.

WS03-5: Health care providers are diverse in their workflows. How can this diversity be a part of your
onboarding process?

A. Too much work flow diversity cannot be overcome

B. Customize a trainer depending on who attends class
C. Ensure representation of both in/outpatient settings for instruction
D. Any provider can teach class as long as they have EHR experience

Answer: C. Ensure representation of both in/outpatient settings for instruction

Explanation: Healthcare providers live in different domains. It is not common any longer to have both
inpatient and outpatient providers covering both domains. Thus, needs and workflow processes are
completely different. A tag-team approach will ensure both settings are represented.

References: Kirk VG, Currie G, Dewey D, Howlett A, Lemiux L. Innovation in physician recruitment &
wellness: a pilot orientation program. J Pediatr. 2013 Apr 162 (4):664-665.e2. doi:
10.1016/j.peds.2013.01.001.

Wright M, Mankey CG, Miller BW. Improving upon the ‘July effect’: a collaborative, interdisciplinary
orientation for internal medicine interns. Med Educ Online. 2013 Dec 23; 18:23249. doi:
10.3402/meo.v18i0.23249.

WS03-6: Providing health care now involves utilization of advance messaging techniques such as secure
messaging through web portals. How can doctors be more efficient in processing of secure messaging?

A. Show the providers in detail how to send messages

B. Review all bells and whistles with advanced secure messaging
C. Set up teams associated with providers who can help resolve appropriate messages
D. Ensure that rules of timeliness of secure messaging is well explained

Answer: C. Set up teams associated with providers who can help resolve appropriate messages

116 | P a g e
Explanation: Teaching is an important part of onboarding, but also establishing profiles for providers will
do wonders for efficiencies. Setting up clinic profiles with additional individuals who can respond to
secure messaging will decrease the load of work off of your new provider.

References: Kirk VG, Currie G, Dewey D, Howlett A, Lemiux L. Innovation in physician recruitment &
wellness: a pilot orientation program. J Pediatr. 2013 Apr 162 (4):664-665.e2. doi:
10.1016/j.peds.2013.01.001.

Wright M, Mankey CG, Miller BW. Improving upon the ‘July effect’: a collaborative, interdisciplinary
orientation for internal medicine interns. Med Educ Online. 2013 Dec 23; 18:23249. doi:
10.3402/meo.v18i0.23249.

WS03-7: Navigating a new health system can be daunting with all the rules. What approach will yield a
higher likeliness of success?

A. Establish a buddy mentor and organize forms in easy-to-find locations

B. Show which forms matter to physicians and demonstrate how to complete them
C. Invite departmental secretaries to training sessions
D. Teach the providers medial assistant how to obtain forms

Answer: A. Establish a buddy mentor and organize forms in easy-to-find locations

Explanation: If professional athletes have numerous mentors and coaches, then your provider can also
benefit from mentors and coaches. Partnering providers with mentors who have experience in
navigating your health system will help them feel they can reach to resolve problems as they arise.

References: Kirk VG, Currie G, Dewey D, Howlett A, Lemiux L. Innovation in physician recruitment &
wellness: a pilot orientation program. J Pediatr. 2013 Apr 162 (4):664-665.e2. doi:
10.1016/j.peds.2013.01.001.

Wright M, Mankey CG, Miller BW. Improving upon the ‘July effect’: a collaborative, interdisciplinary
orientation for internal medicine interns. Med Educ Online. 2013 Dec 23; 18:23249. doi:
10.3402/meo.v18i0.23249.

WS03-8: Pharmacy prescribing practices such as opiates have recently been highly publicized. Which
approach will support providers as they start with your healthcare system?
A. Have a pharmacist team member teach how to write prescriptions in a rapid manner
B. Teach how to build quick order sets
C. Invite your pain management director along with your pharmacy team member to review support
roles and processes available

117 | P a g e
D. Show providers how to rapidly access decision support tools via phone apps

Answer: C. Invite your pain management director along with your pharmacy team member to review
support roles and processes available

Explanation: Involving hot-topic related guests to your onboarding class can help put new providers at
ease. This will show your health care system has a strong support structure and that you understand
pain points in healthcare systems.

References: Kirk VG, Currie G, Dewey D, Howlett A, Lemiux L. Innovation in physician recruitment &
wellness: a pilot orientation program. J Pediatr. 2013 Apr 162 (4):664-665.e2. doi:
10.1016/j.peds.2013.01.001.

Wright M, Mankey CG, Miller BW. Improving upon the ‘July effect’: a collaborative, interdisciplinary
orientation for internal medicine interns. Med Educ Online. 2013 Dec 23; 18:23249. doi:
10.3402/meo.v18i0.23249.

WS04: Hands-On Full Life Cycle Data Science Workshop

--Steven Johnson, Lisiane Pruinelli, Tamara Winden

WS04-1: When using EHR data in an observational study, which of the following is correct? IRB review…

a. Does not apply since this is not a clinical trial.

b. Is not needed since the data is being collected in the EHR to document care

c. Is not needed since we are only accessing the EHR and not interacting directly with patients

d. Is required since we are using patient-specific health data

Answer: d. is required since we are using patient-specific health data

Explanation: IRB review is always required when identifiable patient information is used in a study, even
if the patient isn’t directly involved.

Reference: https://aspe.hhs.gov/report/feasibility-using-electronic-health-data-research-small-
populations/privacy-and-security-conditions-required-research-using-ehr-and-other-electronic-health-
data

118 | P a g e
WS04-2: How do data quality issues impact data science results?

a. Because EHR data is used to document healthcare, there are no data quality issues

b. Data quality can significantly impact results, making them biased or even invalid

c. Data quality issues can be ignored because machine learning and AI will take care of it

d. Poor data quality doesn’t impact results, because the data quality issues disappear with enough data

Answer: b. Data quality can significantly impact results making them biased or even invalid

Explanation: It is critical to understand how complete, consistent and correct your data is for the
purpose that you have for the data. Having more data may just give you more errors and using any data
science technique on incorrect data will likely produce incorrect results.

Reference: Brown J, Kahn M, Toh S. Data quality assessment for comparative effectiveness research in
distributed data networks. Medical care. 2013;51(8 0 3):S22-S29. doi:10.1097/MLR.0b013e31829b1e2c.

WS04-3: A good practice for exploratory data analysis is to use a number of techniques to visualize your
dataset. Which of the following is NOT a good reason to visualize your data?

a. Data visualization allows you to better understand and ”get to know” your data

b. Humans can spot visual patterns much better than looking at descriptive statistics

c. Visualization allows you to see large amounts of data summarized and simplified

d. Visualization looks nicer than just having tables of numbers

Answer: d. Visualization looks nicer than just having tables of numbers

Explanation: The reason visualization is so important is that it engages the visual pattern matching
center of our brain to allow you to see relationships and trends in data that are hard to discern from
tables of numbers.

Reference: Tufte, E., & Graves-Morris, P. (2014). The visual display of quantitative information.; 1983.

119 | P a g e
WS04-4: After receiving a dataset for a data science project, a researcher will spend a significant amount
of the total project time performing data preparation and exploratory data analysis. Which of the
following is NOT a data preparation task?

a. Assessing missing data and imputing appropriate values or removing records

b. Adding variables, transforming data and joining data tables

c. Ensuring all data is of the appropriate type and is consistent with the data mode

d. Obtaining the appropriate level of IRB approval for the project

Answer: d. Obtaining the appropriate level of IRB approval for the project

Explanation: Obtaining the appropriate level of IRB approval for the project should be done before you
request and receive your data. There will be significant time spent in preparing your data for analytics
including dealing with missing information, inconsistent formats and creating additional variables from
the underlying data. Data preparation, visualization and exploration can consume 80% of the time spent
on a data science project.

Reference: Pyle, D. (1999). Data preparation for data mining (Vol. 1). Morgan Kaufmann.

WS04-5: Many datasets have imbalanced data where there is a much smaller amount of data for
positive classes than negative ones. For example, the prevalence of a number of diseases is low in the
overall population (i.e. Ebola). Which is the best method to evaluate the performance of a classification
model for imbalanced data?

a. Accuracy

b. F1 Score

c. Precision

d. Recall

Answer: b. F1 Score

Explanation: Imbalanced data has a smaller amount of positive class data. Precision, which is TP / (TP +
FP) doesn’t take into account FN and TN and Recall, which is TP / (TP + FN) doesn’t take into account FP
and TN which can lead to biased results.

F1 Score is the weighted average of Precision and Recall. Therefore, this score takes both false positives
and false negatives into account. Intuitively it is not as easy to understand as accuracy, but F1 is usually
more useful than accuracy, especially if you have an uneven class distribution.

References: https://www.kdnuggets.com/2016/12/best-metric-measure-accuracy-classification-
models.html

http://blog.exsilio.com/all/accuracy-precision-recall-f1-score-interpretation-of-performance-measures/
120 | P a g e
WS04-6: Once data are properly prepared and cleaned, researchers should select the most appropriate
modeling algorithm depending on the data and the research question. Which modeling technique does
NOT work well on categorical variables?

a. Classification

b. Linear Regression

c. Neural Network

d. Random Forest

Answer: b. Linear Regression

Explanation: Linear regression is appropriate for continuous variables. Categorical variables can be
directly used in random forest models. Techniques, such as “one hot encoding, ” can be used to
incorporate categorical variables into neural networks and other classifiers.

Reference: Witten, I. H., Frank, E., Hall, M. A., & Pal, C. J. (2016). Data Mining: Practical machine learning
tools and techniques. Morgan Kaufmann.

WS04-7: Once you have a good performing model and you have tested it sufficiently on your training
data, it is time to implement it in the real world. Which tasks are essential for successful model
deployment?

a. Ensure the model is activated at the right time in the workflow, appropriately alerts the clinician
and allows specific action

b. It is not necessary to get buy-in from the clinicians, since the model embedded in the EHR will tell
them what to do

c. Such models should only be created for research purposes and should not be used to support direct
patient care

d. The model should only be implemented in the latest version of the EHR software

Answer: a. Ensure the model is activated at the right time in the workflow, appropriately alerts the
clinician and allows specific action

Explanation: Models not only have to be predictive, but the prediction needs to be presented to a
clinician at a time in a clinical workflow when they can use the information to perform an action or
intervention to potentially improve an outcome. Clinicians must be intimately involved in developing
and deploying the model for it to be successful.
121 | P a g e
Reference: Bates, D. W., Kuperman, G. J., Wang, S., Gandhi, T., Kittler, A., Volk, L., ... & Middleton, B.
(2003). Ten commandments for effective clinical decision support: making the practice of evidence-
based medicine a reality. Journal of the American Medical Informatics Association, 10(6), 523-530.

WS04-8: Data science is fast becoming very useful in healthcare, but success requires close collaboration
between people with clinical knowledge and people with data and computer expertise. Which of the
following is true about data science in healthcare? Data science…

a. Can be understood sufficiently by everyone in the healthcare organization

b. Is easy since we give raw EHR data to machine learning algorithms to produce highly accurate results

c. Is somewhat of a fad and it won’t have a significant impact in improving healthcare

d. Only needs to be understood by a few people in the healthcare organization

Answer: a. Can be understood sufficiently by everyone in the healthcare organization

Explanation: If everyone involved in a data science initiative understands the basic concepts of how data
science works, the project will have a higher probability of success. Some of the team members will
have more clinical knowledge and some will have more technical knowledge, but everyone can speak
the same “data science” language and can help reduce the knowledge gap. Having many knowledgeable
people participate in date science initiatives improves the chances for success.

Reference: https://www.cio.com/article/3234353/analytics/the-secrets-of-highly-successful-data-
analytics-teams.html

WS05: Operational and Practical Aspects of Clinical Knowledge Management

--Dominik Aronsky, Saverio Maviglia, Asli Ozdas Weitkamp, Roberto Rocha, Dirk Wenke

WS05-1: What is a logical first step in starting a clinical knowledge management program within the
iterative knowledge management framework?

a) Build standards-based interoperable knowledge assets

b) Centralize the process of creating knowledge assets
c) Create a catalog of existing knowledge assets including relationships with metadata
d) Establish a knowledge asset lifecycle

122 | P a g e
Answer: c) Create a catalog of existing knowledge assets including relationships with metadata

Explanation: An initial assessment of all institutionally available knowledge assets is desirable

understand the level of existing knowledge, identify redundancies, identify gaps, and responsibilities.

Reference: Clinical Decision Support - The Road to Broad Adoption. Edited by Robert A. Greenes,
Academic Press ISBN 978-0-12-398476-0

WS05-2: Which of the following activities is least likely within the scope of a clinical knowledge
management framework?

a) Build a sustainable knowledge asset framework with reusable components

b) Integrate knowledge assets into clinical workflow

c) Maintain the representation of knowledge assets

d) Provide knowledge lifecycle support

Answer: b) Integrate knowledge assets into clinical workflow

Explanation: Knowledge management deals with the creation and maintenance of an institution’s
knowledge assets, including sustaining the knowledge assets, maintain the representation, and provide
strict lifecycle support. The aspects of (clinical) workflow integration are commonly not represented in a
clinical knowledge management environment.

Reference: Clinical Decision Support - The Road to Broad Adoption. Edited by Robert A. Greenes,
Academic Press ISBN 978-0-12-398476-0

WS05-3: Which topic is least likely associated with clinical knowledge management?

a) Institutional data governance

b) Lifecycle management of knowledge assets
c) Management of knowledge meta-data
d) Single-login management for clinical workstations
e) InfoButton

Answer: d) Single-login management for clinical workstations

123 | P a g e
Explanation: Single-login management is least likely modeled in a clinical knowledge management
environment - the other topics are typical KM activities.

Reference: Clinical Decision Support - The Road to Broad Adoption. Edited by Robert A. Greenes,
Academic Press ISBN 978-0-12-398476-0

WS05-4: Which process is not directly supported by a clinical knowledge management system?

a) Centralized prioritization of knowledge management implementation tasks

b) Centralized request management for changes in the knowledge base
c) Monitoring clinicians’ usage of clinical decision support systems
d) Versioning of knowledge assets

Answer: c) Monitoring clinicians’ usage of clinical decision support systems

Explanation: Once a knowledge asset, such as a rule, an order set, or pathway, is implemented for
operational use in the target system, its monitoring is accomplished in other environments, such as the
target system itself (e.g. EMR, LIS), a Business Intelligence (BI) or Enterprise Data Warehouse (EDW)
environment, or similar.

Reference: Clinical Decision Support - The Road to Broad Adoption. Edited by Robert A. Greenes,
Academic Press ISBN 978-0-12-398476-0

WS05-5: Which characteristic is NOT considered best practice for creating knowledge assets?

a) Interoperable
b) Modular
c) Recursive
d) Reusable

Answer: c) recursive

Explanation: Modular, reusable and interoperable are desirable key characteristics for best practices in
the creation of knowledge assets. Recursive patterns are unrelated to knowledge asset creation and are
best known in programming and describe a function or procedure that can repeat itself.

Reference: Clinical Decision Support - The Road to Broad Adoption. Edited by Robert A. Greenes,
Academic Press ISBN 978-0-12-398476-0

124 | P a g e
WS05-6: Which of the following is the least critical method applied in clinical knowledge management?

a) Formal representation of knowledge assets

b) b) Lifecycle management of knowledge assets
c) Management of knowledge asset reference material
d) Versioning of knowledge assets

Answer: c) management of knowledge asset reference material

Explanation: Management of the knowledge asset reference material is an added benefit to document
the source, but is not essential as the other methods are which are critical.

Reference: Clinical Decision Support - The Road to Broad Adoption. Edited by Robert A. Greenes,
Academic Press ISBN 978-0-12-398476-0

WS05-7: Select the most important role(s) domain experts (or subject matter experts) fulfill
within the context of a knowledge management lifecycle?
a) Review quality measures and propose new knowledge assets
b) Create asset specifications based on existing knowledge representation standards
c) Author new knowledge assets using tools available within clinical knowledge management
systems
d) Critique asset specifications and share expertise during knowledge engineering sessions
e) Support end-user adoption of new knowledge assets

Answer: d) Critique asset specifications and share expertise during knowledge engineering
sessions
Explanation: Domain experts are difficult to find and, once identified, typically have very
limited time availability. Considering these restrictions, domain experts are primarily needed
during the knowledge engineering phase of the lifecycle, where their expertise is essential to
enable proper knowledge acquisition and elucidation. The roles described in the other options
can more easily be fulfilled by other professionals.
Reference: Clinical Decision Support - The Road to Broad Adoption. 2nd Edition. Edited by
Robert A. Greenes, Academic Press ISBN 978-0-12-398476-0

125 | P a g e
WS05-8: Which of the following is NOT correct when deploying or updating vendor-based
knowledge assets in a clinical information system environment?
a) Integration requires workflow adjustments, data mappings, and extensive testing
b) Assets cannot be easily customized to reflect local needs and priorities
c) Deploying updates of vendor-based knowledge assets prevents CDSS malfunctions
d) Local customizations might be lost when subsequent updates are deployed
d) Effective troubleshooting by vendor is associated with security vulnerabilities

Answer: c) Deploying updates of vendor-based knowledge assets prevents CDSS malfunctions.

Explanation: The deployment and maintenance of outside, vendor-based content represents a
major threat to CDSS and other malfunctions within an institution as code changes and
mappings may not be updated or not updated correctly leading to malfunctions. The
deployment of assets created by vendors requires detailed planning, testing, documentation,
etc. with dedicated resources.

WS06: CMIO Workshop - Leading Clinical Informatics Initiatives: How to Successfully Tackle Challenges
& Build Resilience

--Julie Hollberg, Richard Schreiber

WS06-1: All hospitals with an EHR should follow the principles of EHR governance. A formal, written EHR
governance plan is necessary for:
A. Only large university hospitals that have large information technology staffs
B. Hospitals that have a chief medical informatics officer who runs the EHR
C. Hospitals that manage their EHRs, but not those who contract EHR services
D. All hospitals of whatever size, leadership structure, or EHR design

Answer: D. All hospitals of whatever size, leadership structure, or EHR design

Explanation: A fundamental learning objective of this workshop is that EHR governance applies to
healthcare environments universally.

126 | P a g e
WS06-2: The most critical success factor in EHR governance is:

E. Managing and leading change transformation processes

F. Assuring purchase of the newest technology

G. Establishing best infrastructure prior to and after EHR installation

H. Rewarding staff for meeting or exceeding goals

Answer: A. managing and leading change transformation processes

Explanation: The best answer is a) because use of an EHR depends on supporting the workflow
requirements of users. The latest and greatest technology (b) is only as good as the ability of personnel
to use it. Although solid infrastructure is necessary (c), it is not sufficient. Rewarding staff (d) has merit,
but is also not sufficient.

WS06-3: Effective governance requires effective communication for EHR optimization and healthcare
transformation, as well as implementation. All the following have been shown to be successful forms of
communication. The most effective is:

A. Rich channels such as personal interaction and audiovisual aids are better than email alone

B. Encouraging leaders to disseminate information to users by any means

C. Just-in-time training—not too soon, not too late; as well as at-the-elbow support

D. Involvement of clinicians as well as technical experts in creating the communication plans

Answer: D. Involvement of clinicians as well as technical experts in creating the communication plans

Explanation: Although most successful programs use all the above, the most effective plan is to involve
subject matter experts and the impacted end-users to plan and execute the communication of
anticipated changes. Hoping for dissemination of information is not as reliable (a or b). It is critical to
loop in clinicians and others who miss meetings. Depending on “top-down” communication is risky.
Face-to-face interactive methods are better than email or other indirect means.

127 | P a g e
WS06-4. Lack of available time, lack of political and communication skills, and lack of vision are
examples of leadership failures which can damage EHR implementation or migration. How can one avoid
such failures?

A. Have a mission statement, a long-term strategic plan, and governance

B. Create a project plan, use Gantt charts to determine time lines, and follow agile processes

C. Hire experienced managers to run the day-to-day operations of the implementation

D. Be sure to hire a chief medical informatics officer to manage the process

Answer: A. Have a mission statement, a long-term strategic plan, and governance

Explanation: Answer A clearly addresses the overall structure for a large, long-term effort such as EHR
implementation. Project plans (b) are for shorter term, more focused goals. Although having
experienced managers is crucial to the success of an implementation, even the best managers cannot
overcome leadership failures (answer c). Although having a CMIO is likely to improve implementation,
one leader alone is not sufficient to ensure success. EHR implementation requires 100% leadership
support.

Reference: Lorenzi NM, Riley RT. Managing change: an overview. J Am Med Inform
Assoc 2000; 7(2): 116-124.

WS06-5. There are many factors which ensure a successful EHR implementation or transformation.
Which of the following is most predictive of success?

A. Training for computer-savvy providers or those who have used the EHR before is not necessary

B. Reducing ambulatory productivity permanently will accommodate changed provider workflows

C. Managers supporting implementation should have no other responsibilities at go-live

D. Additional staff is required to execute the new workflows and provide at-the-elbow support.

Answer: C. Managers supporting implementation should have no other responsibilities at go-live

Explanation: A best practice for effective go-live support is to ensure that key managers and superusers
are able to focus on implementation activities during the go-live. Training is essential for all providers,
although one-size-fits-all training programs are not optimal. Residents must also receive training,
especially if working across multiple care settings in which other EHR systems are used. Typically, with
effective training, additional staff is not required for new workflows; it is when users try to preserve old
workflows in parallel with new workflows that there may be increased staffing requests.

128 | P a g e
WS06-6. A recent study showed that different vendor databases vary in their categorization of drug-
drug interactions (DDIs). Different EHR vendor and home-grown products are configured differently, and
different sites differ as to whether they modify the databases to alter the number of alerts displayed.
Other studies show that despite these differences it is possible to decrease the burden of DDI alerting.
This data suggests:

A. It is too complicated and fraught with potential error to modify the vendor product’s data base

B. It is best practice to accept the vendor product’s DDI database and alerting configuration

C. Meaningful use, which requires DDI alerting, has set a new standard of care for such alerts

D. It is time for a revamping of the international standards of care for DDI alert strategy

Answer: D. It is time for a revamping of the international standards of care for DDI alert strategy.

Explanation: Altering the DDI data base is not complicated as it is a function of changing one field in
most DDI tables. If informaticians, pharmacists, and technical experts work together, it is possible to
minimize errors. Best practice has yet to be determined: the most salient point in the researchers’
results is that enormous variation exists between institutions, including those which use the same
vendor product. Meaningful use does require DDI alerting, but this alone does not represent a standard
of care. The authors do advocate for “Developing a standard that covers which DDIs to alert on and how
to display those alerts [which] will likely require the collaboration of clinical informaticians, pharmacists,
and physicians” and “recommend creating an officially approved, standardized DDI knowledge base, and
considering a possible safe harbor or other legal protection for sites that implement that KB.”

References:
McEvoy DS, Sittig DF, Hickman T-T, Aaron K, Ai A, Amato M, Bauer DW, Fraser GM, Harper J, Kennemer
A, Krall MA, Lehmann CU, Malhotra S, Murphy DR, O’Kelley B, Samal L, Schreiber R, Singh H, Thomas EJ,
Vartian CV, Westmorland J, McCoy AB, Wright A. Variation in High Priority Drug-Drug Interaction Alerts
Across Institutions and Electronic Health Records. J Am Med Inform Assoc. 2017;24(2):331-338.
DOI: http://dx.doi.org/10.1093/jamia/ocw114 First published online: 28 August 2016. PMID: 27570216.

Schreiber R, Shaha J, Gregoire J, Shaha S. Think Time: A Novel Approach to Analysis of Clinicians'
Behavior After Reduction of Pharmacologic Interaction Alerts. Int J Med Inform. 2017;97:59-67.
Published online September 30, 2016. http://dx.doi.org/10.1016/j.ijmedinf.2016.09.011 PMID:
27919396.

WS06-7. The CIO calls you into her office. She tells you the hospital is going to select a new EHR
(Electronic Health Record), implement it, and then optimize it. All is to be done within the span of 24

129 | P a g e
months. She wants to hire a Chief Medical Informatics officer (CMIO). You advise her that the best
candidate is:

A. Physician with extensive EHR experience but not necessarily any informatics education

B. Clinical Informatician educated in and experienced with Informatics

C. Physician with extensive Informatics education but limited EHR experience

D. Only physicians, and absolutely no PhDs or Masters

Answer: B. Clinical Informatician educated in and experienced with Informatics

Explanation: The correct answer is B. A and C are wrong as a CMIO needs both education and
experience to succeed. D is wrong because PhDs with clinical exposure or experience have and can be
CMIOs. Only answer B is correct as it includes both education and experience and all types of
candidates.

WS06-8. Successfully passing the ABPM board exam in applied clinical informatics requires knowledge
in a diverse set of disciplines. These include:

A. Workflow analysis, health information technology, and clinical care

B. Clinical care, workflow analysis, and information system lifecycles

C. Evidence basis for clinical care, change management, and decision science

D. Health systematics, clinical decision support, and computer programming

Answer: C. Evidence basis for clinical care, change management, and decision science

Explanation: While those who achieve board certification should have clinical care knowledge, it is not a
core content area covered on the boards (answers a and b are wrong). Awareness of the evidence basis,
including guidelines of care, is in the core curriculum as are change management and decision science.
Computer programming is neither a prerequisite for informatics, nor part of the core curriculum (answer
d is wrong).

Reference: Gardner R, Overhage JM, Steen EB, Munger BS, Holmes JH, Williamson J, Detmer DE. Core
Content for the Subspecialty of Clinical Informatics. J Am Med Inform Assoc. 2009;16:153–157. DOI
10.1197/jamia.M3045.

130 | P a g e
WS06-9. Yogi Berra once said, “When you come to a fork in the road, take it.” Many of us change our
career trajectories, and this can be a difficult transition. Some are better at these transitions because:

A. Self-reliant physicians manage change well

B. Career and life coaches can benefit physicians

C. Physicians often change positions without difficulty

D. New positions are a welcome and exciting challenge

Answer B. Career and life coaches can benefit physicians

Explanation: Although many physicians prefer to manage their own affairs they frequently stumble on
non-medical decisions (answer a is not always true). Although it is true that physicians change positions
frequently, especially in academic settings, not all transitions are successful (answer c is false). Answer d
can be true, but physicians need to be aware of the imposter syndrome (“was I really cut out for this
job?”), upheavals to one’s personal and family life, and the not uncommon situation that a change was
not a willing transition, all of which exact emotional and professional stress. Answer b is the most
correct answer—physicians should be willing to ask for and seek help from family, friends, colleagues,
and career change professionals.

Reference: Babbott SF. Career Transitions: Traversing the on and off ramps. American College of
Physicians, Internal Medicine Meeting 2018. New Orleans, LA.

WS06-10. Book knowledge provides a solid basis for medical and informatics practice; experience
provides real-world application of that knowledge. Since young informaticians lack that experience, the
single best strategy to increase one’s knowledge is to:

A. Work long hours in the clinic and/or hospital as well as in informatics to augment their learning

B. Seek out a mentor to help with challenging situations and to help guide your career

C. Treat a transition into a new role as a temporary condition as things will work themselves out

D. Avoid revealing one’s difficulties to colleagues as this may jeopardize your job performance

Answer B. Seek out a mentor to help with challenging situations and to help guide your career

Physicians are strongly independent, but asking for help is not only ok, it helps guide us and improve our
care as physicians and in the practice of informatics. There may be nothing wrong with putting in long
hours (although it may have family and social costs), but it is not a substitute for wise guidance. A
transition is not a temporary condition; it is on-going, and needs active intervention to succeed.
Avoidance is a common response to difficulties but is rarely successful.

131 | P a g e
References:

Babbott SF. Career Transitions: Traversing the on and off ramps. American College of Physicians, Internal
Medicine Meeting 2018. New Orleans, LA.

Gawande A. Personal best. Top athletes and singers have coaches. Should you? The New Yorker. 2011
(Oct 3).

WS06-11. Which of the following is not a key driver or domain of burnout and professional fulfillment?

A. Efficiency of practice

B. Practice productivity

C. A culture of wellness

D. Personal resilience

Answer: B. Practice productivity

Explanation: Professional fulfillment and burnout need to be assessed together as the absence of
burnout doesn’t automatically lead to physician fulfillment. While personal resilience is important, it
must be connected to larger system focus on institutional and community support for physicians.

Reference: Bryan Bohman, MD, Liselotte Dyrbye, MD, MHPE, Christine A. Sinsky, MD, Mark Linzer, MD,
FACP, Kristine Olson, MD, MSc, Stewart Babbott, MD, Mary Lou Murphy, MS, Patty Purpur deVries, MS,
Maryam S. Hamidi, PhD & Mickey Trockel, MD, PhD. Physician Well-Being: The Reciprocity of Practice
Efficiency, Culture of Wellness, and Person Resilience. https://catalyst.nejm.org/physician-well-being-
efficiency-wellness-resilience/

WS06-12. Tuckman’s model of team work posits that groups will develop in a predictable fashion. The
usual order of that development is:

A. Norming, storming, forming, performing

B. Storming, forming, norming, performing

C. Forming, performing, storming, norming

D. Forming, storming, norming, performing

Answer D. Forming, storming, norming, performing

Explanation: Tuckman states that groups pull together (form), followed by a period of dispute (storm).
Next the group works out the disagreements (normalizes), before being able to perform.

132 | P a g e
Reference: Tuckman’s stages of group development. Found at:
https://en.wikipedia.org/wiki/Tuckman%27s_stages_of_group_development Accessed 26 April 2018.

WS06-13. The best statement regarding IT governance is:

A. A single structure for good IT governance can keep pace with all the “wants” and “needs” of an
organization

B. IT governance does not need to include research and educational needs in its structure—separate
governance provides adequate oversight

C. How an institution makes IT investment decisions is often more important that what specific
decisions are made

D. Proper leadership governance in IT is sufficient to overcome the lack of a clinical information systems
advisory group

Answer: C. How an institution makes IT investment decisions is often more important that what specific
decisions are made

Explanation: No single governance can provide proper structure for disparate functions and
departments (answer a is incorrect). That said, research and education has information technology
requirements, and IT governance must include those fields, including data governance (answer b is
incorrect). Answer d presupposes that IT personnel can function as clinical decision makers; research
shows that this often fails. Answer c is the best answer as it is the most flexible, allowing for the rapid
change in technology.

References:

Leviss J (ed). HIT or Miss. Chicago: AHIMA press. 2013. See especially chapters 10, 12, and 19.

Shortliffe EH, Cimino JJ. Biomedical informatics. London: Springer-Verlag. Page 459.

WS06-14. When facing conflict over very important issues, and there are no time pressures for
resolution, you should first use which of the following leadership/management methods:

A. Accommodation

B. Compromise

C. Collaboration

D. Competition

Answer: C. Collaboration

133 | P a g e
Explanation: There are a few key variables that define conflict management situations and determine
which conflict management strategies are likely to be effective. Time pressure is an important variable -
if there were never any time pressures, one approach might always be the best to use. When time is not
a variable influencing decisions, and the issues are important, collaboration is the best method of
conflict management to employ.

Reference: AMIA Clinical Informatics Board Review Course (CIBRC). 2014.

WS06-15. Alert fatigue reduces efficiency of use of electronic health records. Many clinicians gloss over
alerts, often without reading them. Research reveals various ranges of override rates, but the most
widely quoted and likely most accurate overall rate is:

A. 60%

B. 75%

C. 96%

D. 99%

Answer: C. 96%

Explanation: The most oft-quoted range comes from Van der Sijs H, Aarts J, Vulto A, Berg M. Overriding
of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006 (Mar-
Apr);13(2):138-147. It has been widely replicated in numerous subsequent studies as recently as this
year. The range has been broad (46 – 98%). 99% is a slight exaggeration.

WS06-16. Which of the following leadership qualities would be most important for a chief medical
informatics officer to display when implementing new systems and applications?

A. Charismatic

B. Task-oriented

C. Transformational

D. Supportive

Answer: C. Transformational

134 | P a g e
Explanation: Leadership qualities have the appearance of being related to traits or behaviors, but the
key difference is the interpersonal application as leaders interact with various constituencies of the
organization. Of the responses, only charismatic and transformational are leadership qualities. Of those
two, transformational is the most important for the change management required when implementing
new Health IT systems.

Reference: AMIA Clinical Informatics Board Review Course (CIBRC). 2014.

WS07: Get Your Hands on FHIR®

--Laura Heermann, Russ Leftwich

WS07-1. What does the acronym FHIR® stand for?

a. Federal Health Information Record

b. Fast Healthcare Interoperability Resources

c. Fire Hot Information Response

d. Fast Health Information Record

Answer: b. Fast Healthcare Interoperability Resources

Reference: http://hl7.org/fhir/summary.html

WS07-2. FHIR® is designed to

a. Replace enterprise healthcare records

b. Be used exclusively for personal health records

c. Enable the exchange of healthcare-related information

d. Burn up all other attempts at interoperability

Answer: c. Enable the exchange of healthcare-related information

Explanation: FHIR® is designed to enable information exchange to support the provision of healthcare in
a wide variety of settings.

Reference: http://hl7.org/fhir/overview-dev.html

135 | P a g e
WS07-3. Why is FHIR® important to clinicians?

a. It leverages the latest web standards and applies a tight focus on interoperability across care
settings and disciplines.

b. It is a standard specification specific to the needs of nursing interoperability.

c. FHIR® has specific resources that apply only to hospital care.

d. FHIR®has no implications for clinical interoperability

Answer: a. It leverages the latest web standards and applies a tight focus on interoperability across care
settings and disciplines.

Reference: http://hl7.org/fhir/summary.html

WS07-4. How is the Fast Healthcare Interoperability Resources specification accessed?

a. http://hl7.org/fhir

b. www.fhir.com

c. www.FIRE.org

d. http://fasthealthcareinteroperability.org/

Answer: a. http://hl7.org/fhir

WS07-5. What is a FHIR® Resource?

a. A system of interoperability gadgets that hook together

b. A logically discrete data concept that has a defined meaning, a known identity and location,
and has meaning in healthcare.

c. Short snippets of reusable clinical themes used to comprise a medical record.

d. Small increments of reusable code.

Answer: b. A logically discrete data concept that has a defined meaning, a known identity and location,
and has meaning in healthcare.

136 | P a g e
Explanation. Although option D is close to correct, it is not complete.

Reference: http://hl7.org/fhir/summary.html

WS07-6. What is an extension in FHIR®?

a. A way to limit the use of a specific resource

b. An adornment to a resource to make it more interesting.

c. A reference to an additional resource that makes it more meaningful.

d. A method to implement additional valid requirements not included in the core specification.

Answer: d. A method to implement additional valid requirements not included in the core
specification.

Reference: http://hl7.org/fhir/extensibility.html

WS07-7. What does it mean to constrain a FHIR® resource?

a. To limit a resource to a specific definition fitting the intent of the implementation (such as
limiting a pick list with the colors red, green, yellow, blue, orange and purple to red, yellow and blue
only)

b. To allow for additional items to be added to a resource

c. To compel a user to document using FHIR resources only.

d. To discourage use of specific parts of a FHIR resource but allow them if absolutely necessary.

Answer: a. To limit a resource to a specific definition fitting the intent of the implementation (such
as limiting a pick list with the colors red, green, yellow, blue, orange and purple to red, yellow and blue
only)

Reference: http://hl7.org/fhir/profiling.html#resources

137 | P a g e
WS07-8. What is a FHIR® Profile?

a. A sketch of a drafted FHIR® resource prior to its being released for use.

b. A specification of a FHIR® resource or resources detailing constraints and extensions necessary

for use in a particular use case.

c. A fully instantiated resource

d. A FHIR® resource in action sharing patient data from one clinical site to another.

Answer: b. A specification of a FHIR® resource or resources detailing constraints and extensions

necessary for use in a particular use case.

Reference: http://hl7.org/fhir/profilelist.html

WS07-9. Name 3 FHIR® Architectural Principles.

a. Fast, healthcare specific, reusable

b. Fast, easy to learn, scalable

c. Reusable, Scalable, Readily Implementable

d. Implementable, clinically sufficient, highly structured

Answer: c. Reusable, Scalable, Readily Implementable

Reference: http://hl7.org/fhir/overview-arch.html#framework

WS07-10. FHIR® Resources are organized into 5 framework layers. These include:

a. Foundation, Base, Clinical, Financial and Specialized

b. A, B, C, D, E layers

c. Problems, Meds, Allergies, Orders and Labs

d. Structural, Clinical, Observations, Provider, Payer

Answer: a. Foundation, Base, Clinical, Financial and Specialized

Reference: http://hl7.org/fhir/overview-arch.html#framework

138 | P a g e
WS07-11. Resources classified as “workflow” are meant to:

a. Support the healthcare process

b. Conform to a specific institution’s clinical workflow.

c. Be implemented using BPMN

d. Identify the users of a specific workflow

Answer: a. Support the healthcare process

Reference: http://hl7.org/fhir/resourceguide.html

WS07-12. What is a FHIR® Implementation Guide?

a. A listing of FHIR® resources

b. A how-to guide on profiling FHIR® resources

c. A knowledgeable person available for new implementers for assistance as needed.

d. A publication outlining capabilities as defined by national standards, vendor consortiums,

clinical societies, etc., regarding how FHIR® specifications are used to solve particular problems.

Answer: d. A publication outlining capabilities as defined by national standards, vendor

consortiums, clinical societies, etc., regarding how FHIR® specifications are used to solve particular
problems.

Reference: http://www.fhir.org/guides/registry

WS07-13. The Observation Resource should be used for what type of data?

a. Home meds, orders, MAR

b. Vital Signs, physical exam finding, pulmonary artery pressure reading, lab results

c. CT Scan, MRI, Xrays

d. Problem List, Chief Complaint, Past Medical History

Answer: b. Vital Signs, physical exam finding, pulmonary artery pressure reading, lab results

Reference: http://hl7.org/fhir/resourceguide.html#clinical

139 | P a g e
WS07-14. The Condition Resource should be used for what type of data?

a. CT Scan, MRI, Xrays

b. Vital Signs, Physical Exam Findings, Pulmonary Artery Catheter reading

c. Diagnosis, Chief Complaint, Past Medical History

d. Home meds, orders MAR

Answer: c. Diagnosis, Chief Complaint, Past Medical History

Reference: http://hl7.org/fhir/resourceguide.html#clinical

WS07-15. The clinFHIR software tool requires what configuration parameters?

a. A use case type aligned with the categories of FHIR® resources

b. Selection of a set of capability statements from a library

c. Selection of LOINC and SNOMED value sets from the NIH Value Set Access Center

d. Setting compatible data, conformance, and terminology servers

Answer: d. Setting compatible data, conformance, and terminology servers

Reference: www.clinfhir.com

WS07-16. How do I provide feedback to the FHIR development team regarding FHIR resources?

a. Send them an email to FHIR@FHIR.org

b. Call HL7 Headquarters

c. Submit them through the gForge tool

d. Don’t bother – the specification is done and published.

Answer: C. Submit them through the gForge tool

Reference: http://gforge.hl7.org/gf/

140 | P a g e
WS07-17. What is a FHIR® Maturity Level?

a. A method for the FHR specification developers to inform implementers how advanced and
stable a FHIR Resource is.

b. How long the FHIR® resource has been in existence

c. How many times the FHIR® resource has been implemented

Answer: a. A method for the FHR specification developers to inform implementers how advanced and
stable a FHIR Resource is.

Reference: http://hl7.org/fhir/versions.html#maturity

WS07-18. What is the interoperability paradigm in which FHIR® is operable?

a. The messaging paradigm, like HL7 version 2.

b. The document paradigm, like HL7 Clinical Document Architecture (CDA).

c. As a RESTful API.

d. FHIR® can be used in all of the above interoperability paradigms.

Answer: d. FHIR® can be used in all of the above interoperability paradigms.

References:

https://www.hl7.org/fhir/messaging.html

https://www.hl7.org/fhir/documents.html

WS07-19. The Argonaut project is what type of initiative?

a. An initiative to develop FHIR® certification testing led by the federal government.

b. An initiative initially started to accelerate development of FHIR®profiles for the Common

Clinical Data Set sponsored by a vendor collaborative.

c. An initiative to develop FHIR® translations led by the HL7 Greece affiliate.

d. An initiative that is part of SMART on FHIR®.

141 | P a g e
Answer: b. An initiative initially started to accelerate development of FHIR profiles for the Common
Clinical Data Set sponsored by a vendor collaborative.

Reference: http://argonautwiki.hl7.org/index.php?title=Main_Page

WS07-20. What is SMART on FHIR®?

a. An ONC project to create FHIR® profiles for CDS.

b. A platform architecture for portable healthcare apps utilizing FHIR® as an API for health data.

c. The organization which has published FHIR® profiles for the Common Clinical Data Set.

d. The organization which first developed the concept of FHIR®.

Answer: b. A platform architecture for portable healthcare apps utilizing FHIR® as an API for health data.

Reference: https://smarthealthit.org

WS07-21. Which of the following HL7 standards include specifications for an application
programming interface (API)?

a. HL7 Version 2.x

b. HL7 Version 3

c. HL7 CDA

d. HL7 FHIR®

Answer: d. HL7 FHIR®

References:

http://www.hl7.org/implement/standards/index.cfm?ref=nav

http://www.hl7.org/implement/standards/product_matrix.cfm?ref=nav

http://www.hl7.org/implement/standards/product_brief.cfm?product_id=449

142 | P a g e
WS07-22. FHIR resources can be linked to each other via what type of element in a resource
definition?

a. Code

b. Codable Concept

c. Reference

d. Identifier

Answer: c. Reference

Reference: http://hl7.org/fhir/references.html

WS07-23. Which of the following represent types of terminologies that can be linked to a FHIR®
Profile?

a. SNOMED and LOINC

b. SnowOwl and RF2

c. C4NNol1 and NaPoLeon

d. Java and Python

Answer: a. SNOMED and LOINC

Reference: http://hl7.org/fhir/terminologies-systems.html

143 | P a g e
WS08: How to Effectively Negotiate the Outcomes You Really Desire?

--Shakaib Rehman, Hamed Abbaszadegan

WS08-1: According to the Thomas-Kilmann Model, if your concern about your own outcome is low and
your concern for the other party's outcomes is high; your approach would be:

A. Accommodating

B. Avoiding

C. Compromising

D. Problem-solving

Answer: A. Accommodating

Explanation: According to the Thomas-Kilmann Model if your concern about your own outcome is low
and your concern for the other party's outcomes is high, your approach would be considered
accommodating.

References: 1. Negotiation and Mediation Spiral-bound – 1991 by Peter Pruitt

2. http://www.kilmanndiagnostics.com/catalog/thomas-kilmann-instrument-one-assessment-person

WS08-2: Psychologist Stephen Covey’s research has shown that a certain percentage of life is under your
control and is decided by how you react while the other portion of life is called destiny (no matter what
do you do; it will happen because it is inevitable). Which of the following option is correct about his
determination of life events decided by you over what happened to you?

A. 60/40

B. 70/30

C. 80/20

D. 90/10

Answer: D. 90/10

Explanation: According to Psychologist Stephen Covey, 90% of life is decided by how you react versus
10% of life is made of what happened to you. it is called 90/10 rule.

Reference: Covey, Stephen R. The 7 Habits of Highly Effective People: Restoring the Character Ethic.
New York: Free Press, 2004.

144 | P a g e
WS08-3: Psychologist Abraham Maslow has described the “Human Hierarchy of Needs.” What is the
highest need of individuals, according to his model?

A. Esteem
B. Love
C. Security
D. Self-Actualization

Answer: D. Self-Actualization

Explanation: According to psychologist Abraham Maslow's Hierarchy of Needs; Self-Actualization is the

highest level of human motivation which cannot be addressed until more fundamental needs are
addressed.

Reference: https://www.reference.com/world-view/maslow-s-hierarchy-needs-96f513d6ddfedf6d

WS08-4: What one of the following best describe “Principled Negotiations”?

A. Focus on interests not positions

B. Focus on people not problems
C. Identify subjective criteria to determine success
D. Identify Worst Alternative to a Negotiated Agreement (WATNA)

Answer: A. Focus on interests not positions

Explanation: Five fundamentals of negotiation: 1. Separate people from the problem. 2. Focus on
interests, not positions. 3. Invent opportunities together for mutual gain. 4. Identify objective criteria to
determine success. 5. Know your BATNA (Best Alternative to a Negotiated Agreement), and calculate the
other party's BATNA.

Reference: Fisher, R., Ury, W., & Patton, B. (1991). Getting to yes: Negotiating agreement without giving
in. New York, N.Y: Penguin Books.

WS08-5: Which one of the following is not part of the “Principled Negotiation” model?

A. Separate people from problem

B. Invent options for mutual gain
C. Insist on using subjective criteria
D. Focus on interests, not positions

145 | P a g e
Answer: C. Insist on using subjective criteria

Explanation: The model calls for insisting on using objective criteria.

Reference: Fisher, R., Ury, W., & Patton, B. (1991). Getting to yes: Negotiating agreement without giving
in. New York, N.Y: Penguin Books.

WS08-6: The “Principled negotiation” model emphasizes the importance of the “BATNA.” What does
BATNA stand for?

A. Best Action to a Negotiated Agreement

B. Best Alternative to a Negotiated Agreement

C. Best Alternative to a Nonsense Agreement

D. Best Action to a Nonsense Alternative

Answer: B. Best Alternative to a Negotiated Agreement

Reference: Fisher, R., Ury, W., & Patton, B. (1991). Getting to yes: Negotiating agreement without giving
in. New York, N.Y: Penguin Books.

WS08-7: Complete the list of Thomas-Kilmann's five conflict management styles: Competing,
accommodating, avoiding, ___ and ___.

A. collaborating, and compromising.

B. colluding, and connecting.

C. communicating, and compromising.

D. cooperating, and connecting

Answer: A. collaborating, and compromising.

Explanation: The five conflict management style orientations are competing, accommodating, avoiding,
collaborating, and compromising.

Reference: http://www.kilmanndiagnostics.com/catalog/thomas-kilmann-instrument-one-assessment-
person

146 | P a g e
WS08-8: If a person typically tends to delegate controversial decisions and accept others’ decisions,
which of the following styles best describes this behavior?

A. Accommodating

B. Avoiding

C. Collaborative

D. Competitive

Answer: B. Avoiding

Explanation: People who tend to use an avoiding style "seek to evade the conflict entirely." They avoid
controversial decisions. This style can be appropriate when "victory is impossible, when the controversy
is trivial, or when someone else is in a better position to solve the problem."

Reference: http://www.kilmanndiagnostics.com/catalog/thomas-kilmann-instrument-one-assessment-
person

WS09: Designing Outpatient Clinical Decision Systems for High Use Rates, Provider Satisfaction, and
Improved Care: Lessons from Meta-Analyses and Experience

--Patrick O'Connor, JoAnn Sperl-Hillen

WS09-1: Outpatient Clinical Decision Support (CDS) has been shown to be least effective at improving
which of the following clinical measures:

A. Adult Immunization rates

B. Blood pressure control in hypertensives
C. Diabetes eye exam rates
D. Lipid test rates
E. Screening Mammography rates

Answer: B. Blood pressure control in hypertensives

Explanation: Most meta-analyses of outpatient CDS indicate a positive impact on testing rates,
screening rates, and immunization rates, but much less impact on actual levels of BP, glucose, or lipid
control.

Reference: Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systems: a systematic
review. Ann Intern Med. 2012;157(1):29-43.

147 | P a g e
WS09-2: Which of the following statements is most true of the impact and cost of Clinical Decision
Support and Nurse Case Management to improve the proportion of those with diabetes who reach
recommended treatment goals:

A. Clinical Decision Support is as effective as Nurse Case Management

B. Clinical Decision Support and Nurse Case Management can reach similar proportions of adults with
uncontrolled diabetes
C. On a population basis, the cost effectiveness of Clinical Decision Support and Nurse Case
Management for diabetes care are similar
D. Nurse Case Management can reduce overall health care costs
E. Clinical Decision Support for diabetes care is cost saving to payers

Answer: C. On a population basis, the cost effectiveness of Clinical Decision Support and Nurse Case
Management for diabetes care are similar

Explanation: Compared to outpatient diabetes CDS, Nurse Case Management (a) reaches a smaller
proportion of diabetes patients because many decline to engage with the case manager, (b) costs more
per patient than outpatient CDS, and (c) has a bigger impact on measures such as BP level and glycated
hemoglobin (A1c) level.

Reference: Gilmer TP, O'Connor PJ, Sperl-Hillen JM, et al. Cost-effectiveness of an electronic medical
record based clinical decision support system. Health services research. 2012;47(6):2137-2158.

WS09-3: Consider a CDS system that extracts clinical information from the EHR, sends it to a web service
where it is processed using clinical algorithms, and then transmits CDS back to the clinician and the
patient at the point of care. Barriers to broad dissemination this type of CDS system include all of the
following except:

A. Getting physician leaders from many medical groups to agree on clinical goals for BP, A1c, lipid
management, and aspirin use in various groups of patients.
B. Concerns about data security when transmitting personal health information using the cloud
C. The potential of EHR-derived clinical information to be inaccurate or incomplete
D. Desire to provide evidence-based care personalized to each patient
E. Low use of the CDS system by primary care providers

Answer: D. Desire to provide evidence-based care personalized to each patient

Explanation: Most primary care clinicians are trained to personalize care to each patient based on
intuition and clinical judgement. Most providers prefer that treatments be evidence-based.

148 | P a g e
Reference: Vermunt NP, Harmsen M, Elwyn G, Westert GP, Burgers JS, Olde Rikkert MG, Faber MJ. A
three-goal model for patients with multimorbidity: A qualitative approach. Health Expect.
2018;21(2):528-538.

WS09-4: In a classic paper by McGlynn et al published in 2003, the number of evidence-based actions
that apply to adults in primary care settings is approximately:

A. 60
B. 300
C. 600
D. 3000
E. 6000

Answer: C. 600

Explanation: McGlynn, et al, measured delivery of 600 evidence-based clinical services to a sample of
adult patients, and reported that only 55% were regularly delivered to the patients that they studied.
One such patient needed over 100 such services. These data suggest the need to prioritize evidence-
based clinical services based on their potential benefit to a given patient.

Reference: McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the
United States. The New England journal of medicine. 2003;348(26):2635-2645.

WS09-5: In a time-motion study of primary care physicians using a popular electronic health record
system, the number of clicks and minutes required to get 80% of relevant data on cardiovascular risk to
assess control of BP, lipids, glucose, smoking, aspirin use, and weight/BMI during an encounter was:

A. 4 clicks and 20 seconds

B. 12 clicks and 50 seconds
C. 30 clicks and 2 minutes
D. 50 clicks and 4 minutes
E. 70 clicks and 6 minutes

Answer: D. 50 clicks and 4 minutes

Explanation: It took clinicians 53 clicks and 4 minutes to get 80% of the information needed to assess
control of BP, lipids, glucose, smoking, aspirin use, and weight/BMI in adults when that information was
desired. Only about 80% of what was found was accurate and up to date. This suggests the opportunity
to have algorithms retrieve these data and present them in an organized format, in order to save
clinician time during visits, and likely increase the accuracy of the elicited information.

149 | P a g e
Reference: Koopman RJ, Kochendorfer KM, Moore JL, Mehr DR, Wakefield DS, Yadamsuren B, Coberly
JS, Kruse RL, Wakefield BJ, Belden JL. A diabetes dashboard and physician efficiency and accuracy in
accessing data needed for high-quality diabetes care. Annals of family medicine. 2011;9(5):398-405.

WS09-6: The approximate annual payment from a medical group to an EHR vendor for use and
maintenance of an EHR software system is approximately:

A. $500 per clinician per year

B. $1,000 per clinician per year
C. $2,000 per clinician per year
D. $5,000 per clinician per year
E. $50,000 per clinician per year

Answer: D. $5,000 per clinician per year

Explanation: The actual charges vary from one medical group to another, and are often confidential per
contractual agreements. However, many experts estimate that the costs are well in excess of $10,000
per physician per year, and may be up to $20,000 per physician per year in some cases. These figures
are useful to contextualize the added cost and potential benefits of add-on clinical decision support
systems to improve care.

Reference: Adler-Milstein J, Green CE, Bates DW. A survey analysis suggests that electronic health
records will yield revenue gains for some practices and losses for many. Health Aff (Millwood).
2013;32(3):562-570.

WS09-7: Advantages of web-based, rather than within-EMR clinical decision support systems include all
the following except:

A. Many medical groups can use a single CDS installation

B. Updating CDS algorithms when guidelines change is cheaper in a web-based service
C. Updating CDS algorithms when guidelines change is less prone to error if only one installation
requires updating
D. Data security is improved when CDS is web-based
E. The cost of shared CDS installations would likely be lower than the aggregate cost of each medial
group doing their own CDS installation

Answer: D. Data security is improved when CDS is web-based

150 | P a g e
Explanation: Medical groups have varying programmer capacity and expertise, which is needed to
maintain and update CDS algorithms. Maintaining a large CDS installation that can serve many medical
groups lowers costs, reduces programming errors, and enables more rapid dissemination of new
guidelines. However, transmission of data form the EMR to the web service and back requires additional
measures to assure data security.

Reference: Schmittdiel JA, Adams SR, Segal J, et al. Novel use and utility of integrated electronic health
records to assess rates of prediabetes recognition and treatment: brief report from an integrated
electronic health records pilot study. Diabetes care. 2014;37(2):565-568.

WS09-8: In many cases, there are conflicting clinical guideline recommendations from different sources.
For example, the 2018 American Diabetes Association recommends a BP goal for adults with diabetes of
< 140/90 mm Hg, while the 2017 AHA/ACC hypertension guideline recommends a BP goal of < 130/80
mm Hg for adults with diabetes.The best way to handle this type of situation in primary care CDS
systems is to:

A. Provide detailed information to clinicians at each encounter, explaining the different

recommendations
B. Insert “hard stops” to assure that clinicians must pay attention to BP goals at every patient encounter
C. Provide no CDS on BP management until expert panels resolve their differences
D. Frame treatment recommendations using words like “consider” rather than words like “should,” to
accommodate differences in guidelines
E. Give each hypertensive patient a written sheet explaining the situation

Answer: D. Frame treatment recommendations using words like “consider” rather than words like
“should,” to accommodate differences in guidelines

Explanation: In situations where legitimate and credible guidelines present conflicting

recommendations, it is advisable to soften the language and encourage clinicians to individualize care
plans to the needs and preferences of each patient.

Reference: Ismail-Beigi F, Moghissi E, Tiktin M, Hirsch IB, Inzucchi SE, Genuth S. Individualizing glycemic
targets in type 2 diabetes mellitus: implications of recent clinical trials. Annals of Internal Medicine.
2011;154(8):554-559.

151 | P a g e
Wednesday, May 9th

S01: Presentations - Integrating Quality and Workflow

Quality in Action, Adapting Quality Measures for Realtime Guidance

--Andrew Simms

S01-1: When are providers required to report electronic Clinical Quality Measures data to CMS?

A. 2017
B. 2018
C. 2019
D. 2020

Answer: B. 2018

Explanation: The Centers for Medicare and Medicaid Services have mandated this for 2018.

Reference: Clinical quality measures basics [Internet].; 2018 [updated Feb 06,; cited Mar 26, 2018].
Available from: https://www.cms.gov/Regulations-and-
Guidance/Legislation/EHRIncentivePrograms/ClinicalQualityMeasures.html.

S01-2: The Merit-Based Incentive Payment System (MIPS) uses a score system for calculating
performance. What percentage of this score is based on quality improvement activities?

A. 10%
B. 15%
C. 25%
D. 60%

Answer: D. 60%

Explanation: Although measures vary by specialty, quality improvement is the category with the highest
contribution to total score.

Reference: Resources by provider type - centers for medicare & medicaid services. Retrieved Mar 26,
2018, from
https://www.cms.gov/Medicare/Quality-Payment-Program/Resource-Library/Resources-by-provider-
type.html

152 | P a g e
From Pathway to Bedside: Integrating Clinical Informatics in Quality Improvement.

--Eric Shelov

S01-3: The most valuable asset of a Clinical Informatician in a Quality Improvement project centered on
Clinical Decision Support is their knowledge of:

A. Data Standards

B. Clinical & IS workflows

C. EHR functionality

D. Database structure

Answer: B. Clinical & IS workflows

Explanation: All of the above are important for a well-rounded Clinical Informatician. The asset of
greatest value, however, in the potential success and minimizing unintended consequences of a QI
project is their knowledge of workflows. Even if they are intimately familiar with the exact clinical
workflow at hand, even a somewhat generic clinical workflow knowledge coupled with an
understanding of technical workflows are a critical contribution to a successful implementation.

Reference: Bates, D. W., Kuperman, G. J., Wang, S., Gandhi, T., Kittler, A., Volk, L., et al. (2003). Ten
commandments for effective clinical decision support: making the practice of evidence-based medicine
a reality. Journal of the American Medical Informatics Association : JAMIA, 10(6), 523–530.
http://doi.org/10.1197/jamia.M1370

S01-4: The most fundamental strength of a clinical pathway is its ability to help ________ clinical care
and best practices.

A. Improve

B. Disseminate

C. Standardize

D. Accelerate

Answer: C. Standardize

Explanation: All of these are potentially a strength of a clinical pathway, but A, B and D are all
dependent on C, the standardization of care that a clinical pathway affords.

153 | P a g e
Reference: Osheroff, J. A., Teich, J., Society, H. I. A. M. S., Levick, D., Saldana, L., Velasco, F., et al. (2012).
Improving Outcomes with Clinical Decision Support: An Implementer's Guide. HIMSS.

How am I doing? Online, interactive quality measures for clinicians

--Mark Weiner

S01-5: Patients may see different providers over time, so attributing the quality of a patient’s care to a
single provider can be challenging. Which of the following is the most sensible way to ensure that the
association between patient and provider is appropriate for attribution of a quality measure?

a. Recent Provider
b. Listed Provider in EHR
c. Listed Provider on the Insurance Card
d. Most frequent Provider

Answer: d. Most frequent Provider

Explanation: The listed provider in the EHR and the Listed Provider on the Insurance Card (b and c) are
often administratively the most straightforward, and indeed, insurance carriers will often attribute
quality to whomever they think is taking care of the patient. However, in many cases these listed
providers are not the actual person caring for the patient. The Recent Provider (a) is a tempting choice
since that person may have taken over the care of the patient from providers who have seen the patient
in the past. However, the recent provider may not have had the duration of exposure to the patient to
impact the patient’s care. The most frequent provider (d) is the person the patient has seen most often
in a time period and therefore is likely to have the greatest impact on the quality of the patient’s care.

S01-6: Providers who receive report cards on their quality of care express many concerns about the
validity of the measures. Which of the following statements about quality measures is true?

a. If a quality threshold is relaxed, the proportion of a provider’s panel achieving that

threshold is likely to increase
b. If a quality threshold is relaxed, a provider’s quality ranking is likely to improve
c. A higher-ranked (“better”) provider will have a greater number of patients meeting a quality
threshold then a lower-ranked (“worse”) provider.
d. Stratifying a provider’s panel by the presence of diabetes will decrease the proportion of
the cohort receiving a HBA1c test compares with the overall cohort.

Answer: a. If a quality threshold is relaxed, the proportion of a provider’s panel achieving that
threshold is likely to increase

154 | P a g e
Explanation: If a threshold for quality is relaxed, it becomes easier for all providers to achieve that
threshold (such as changing a HBA1c threshold from 7 to 8). Therefore, A is correct because it is likely
that the proportion of patients achieving the “easier” threshold will increase. B is incorrect since the
proportion of all providers’ panels achieving the threshold will increase, but some providers may have
greater increases than others, so rankings will drop for many providers. C is incorrect because the
ranking is based on the proportion of patients in a panel achieving a threshold. Therefore, a lower
ranked (worse) provider with a larger panel size may have a larger absolute number of patients
achieving a threshold than a higher-ranked (better) provider with a smaller panel. D is incorrect since
the proportion of patients receiving an HBA1c test will be more diluted in the overall panel, compared
with the smaller set of patients with diabetes on whom the HBA1c test is more likely to be ordered.

S02: Presentations –- Standards and Best Practices for EHR Implementations

Come Together: Lessons Learned from a Health System EHR Migration and Proposal for Sharing
Implementation Best Practices

--John McGreevey

S02-1: All of the following are limitations to the broad applicability of existing guidance about how to
implement an EHR except:

A. Existing EHR implementation guidance is based on valid experience and research and comes from
respected health care, academic, and government institutions

B. Work describing an implementation addresses a narrow population (example: small, rural primary
care clinics)

C. Some literature is becoming less relevant with time (example: paper to EHR transition)

D. Guidance focuses on a single aspect of an implementation (example: CPOE)

E. Guidance tends to be noncontroversial and described at a high level (example: training is important)

Answer: A. Existing EHR implementation guidance is based on valid experience and research and comes
from respected health care, academic, and government institutions

Explanation: Answers B, C, D, and E are all limitations to the broad applicability of existing guidance
about how to implement an EHR. Specifically, limitations include narrow populations or settings,
decreasing relevance of paper to electronic implementation literature as more systems have established
EHRs in place, focus on implementation of a single aspect of an EHR rather an entire EHR
implementation, and high level recommendations that may not instruct readers about the specifics of

155 | P a g e
how to execute such recommendations. Despite these limitations, the existing work is based on valid
real-world experiences and research by respected individuals and institutions. Thus the correct answer is
A.

S02-2: Which of the following combinations correctly represents lessons learned in this health system
EHR implementation?

A. Given that vendors have extensive experience with implementation, do not deviate from vendor
guidance; develop an infrastructure of point people from various domains to support the
implementation work; get an early start with change management; forego developing a strategy related
to the EHR implementation as the vendor will guide you to a successful outcome.

B. Begin an implementation with internal alignment of practices, policies, equipment so that there is
consistency throughout the organization; develop an infrastructure of point people from various
domains to support the implementation work; recognize that sometimes vendor guidance will be
appropriate and at other times, an organization may need to follow its own plan; wait to initiate change
management efforts until 4-6 weeks before go-live so that the rest of the implementation work can be
completed first.

C. Get an early start with change management; begin an implementation with internal alignment of
practices, policies, equipment so that these are consistent across the organization; develop an
infrastructure of point people from various domains to support the implementation work; maintain
humility

D. Begin an implementation with internal alignment of practices, policies, equipment so that there is
consistency throughout the organization; “no training, no access, no kidding” was an effective training
strategy in this implementation; get an early start with change management; recognize that with a
thoughtful strategy for an EHR implementation, anyone who reports problems with the system or who
participates in workarounds after go-live is likely doing so because they are resisting necessary
organizational change.

Answer: C. Get an early start with change management; begin an implementation with internal
alignment of practices, policies, equipment so that these are consistent across the organization; develop
an infrastructure of point people from various domains to support the implementation work; maintain
humility

Explanation: Answer C contains correct lessons learned as reported in this health system
implementation experience. All other answers have at least one incorrect lesson. Incorrect lessons listed
include:

1) given that vendors have extensive experience with implementation, do not deviate from vendor
guidance

156 | P a g e
2) forego developing a strategy related to the EHR implementation as the vendor will guide you to a
successful outcome

3) wait to initiate change management efforts until 4-6 weeks before go-live so that the rest of the
implementation work can be completed first

4) recognize that, with a thoughtful strategy for an EHR implementation, anyone who reports problems
with the system or who participates in workarounds after go-live is likely doing so because they are
resisting necessary organizational change.

Seize the Moment: Promotion of Lean Principles in Moving from Paper to Electronic Provider
Documentation at an Academic Medical Center

--Joel Betesh

S02-3: An Advantage of the APSO (Assessment, Plan, Subjective, Objective) note over a SOAP
(Subjective, Objective, Assessment and Plan) note in the EHR is that:

a. It is supported by extensive high quality literature

b. It is easier for the reader of the note to find the key elements of decision making of the note author

c. It protects against notes being bloated with info pulled in from elsewhere in the chart

d. APSO has been used more extensively over the last few decades

Answer: b. It is easier for the reader of the note to find the key elements of decision making of the note
author

Explanation: Having the assessment and plan the top of the note brings the thought process of the
author directly to the reader's attention.This is true in a paper note and even more so in an electronic
note which tends to be longer.

For the other incorrect options:

We did a literature search and found little literature on this topic.

SOAP has been the dominant format of doctors notes over the past few decades.

Although we wanted the notes be lean there is nothing about the APSO format that protects it against
note bloat.

Reference: Lin CT, McKenzie M, Pell J, Caplan L. Health care provider satisfaction with a new electronic
progress note format: SOAP vs APSO format. JAMA Intern Med. 2013 Jan 28;173(2):160-2.

157 | P a g e
S02-4: Moving from paper to electronic format alone is not enough to fully alter ingrained provider
practices and perceived documentation needs. Problems with this transition at the health system
described include all the choices below except:

a. Providers continue to feel a need to add data to their note for purposes of billing.

b. Manual chart review is still required to assess note quality

c. It is possible for individual providers to deviate significantly from the standard templates offered at a
health system level

d. The majority of notes evaluated after the transition from paper to electronic documentation used
the APSO format

Answer: d. The majority of notes evaluated after the transition from paper to electronic documentation
used the APSO format

Explanation: D is correct because that is the main positive finding of our study (see table on page 2 of
our presentation)

A, B and C are all problems we noted after our go-live with provider documentation. See the first 2
paragraphs of our discussion section.

Maintaining EHR Standards in a Complex Deployment

--Keith Woeltje

S02-5: All of these are important to invite to participate in the change control meetings, except:

A. CMOs/CNOs

B. Local leaders from each hospital

C. All subject matter experts (SMEs) for each specialty

D. CMIO

Answer: C. All subject matter experts (SMEs) for each specialty

Explanation: It is important to have input from leaders across the organization to arrive at consensus
decisions. While SME input is essential for developing appropriate content, their participation can be ad
hoc when their particular specialty is under discussion.

Reference: Project Management Institute. A Guide to the Project Management Body of Knowledge
(PMBOK® Guide)–Sixth Edition. Newtown Square, PA: Project Management Institute; 2017.

158 | P a g e
S02-6: Change control is an essential skill for clinical informaticists. Which of the following is TRUE about
good change control? Change control…

A. is very technical, and should really only involve trained analysts and informaticians

B. helps prioritize limited resources and ensures system standards are upheld

C. has to be done because it’s part of Meaningful Use

D. is most useful at the start of a project, and is less useful once application teams are more sure about
what they are doing.

Answer: B. [Change control] helps prioritize limited resources and ensures system standards are upheld

Explanation: A good change control process does involve technical staff, but also involves other
stakeholders. The process ensures that the highest priorities are addressed first, and that changes meet
enterprise operational and technical standards. Change control is not part of the “Meaningful Use”
standards. Change control processes are essential at all stages of a project, but become even more
essential as a project advances to ensure that previous work is not adversely affected inadvertent

Reference: Project Management Institute. A Guide to the Project Management Body of Knowledge
(PMBOK® Guide)–Sixth Edition. Newtown Square, PA: Project Management Institute; 2017.

S03: Panel - Clinical Informatics Fellows Project Ignite Talks

--Christopher Hollweg (Presenter)

S03-1: In the FHIR (Fast Healthcare Interoperability Resources) specifications, what is the difference
between “Resources” and “Extensions”?
A) Resources manage the incoming FHIR queries and Extensions the outgoing queries
B) Resources map the data back to the database while extensions map the data to SMART applications
C) Resources define the standard data types while extensions the additional data types
D) Resources deal with structured data (i.e., ICD) and extensions with unstructured data (i.e., free text)

Answer: C) Resources define the standard data types while extensions the additional data types

Explanation: The FHIR specifications have 'Resources,' which are standard data types that cover about
80% of all data. If additional data types are needed, they can be added on as 'Extensions.'

Reference: https://www.hl7.org/fhir/resource.html

159 | P a g e
S03-2: Which of the following is not a core principle of information system security theory?

A) Availability
B) Vulnerability
C) Confidentiality
D) Integrity

Answer: B) Vulnerability

Reference: United States Code, Title 44, Chapter 35, Subchapter III, § 3542—Definitions.
https://www.law.cornell.edu/uscode/text/44/chapter-35/subchapter-III

S03-3: A swim lane flowchart differs from a simple flowchart in that it:
A. Focuses on the value stream
B. Is a physical map of movements of people in the workflow
C. Visually represents the actions taken by various roles
D. Maps out the steps in the process

Answer: C. Visually represents the actions taken by various roles

Reference: 3 Types of Flowcharts You Can Use for Workflow Management. July 10, 2017.
https://gravityflow.io/3-types-flowcharts-can-use-workflow-management/

S03-4: Why is ELK stack a good tool set to manage healthcare data? Because it is…

A) an SQL database management system and can be easily queried

B) a NoSQL database management system and is highly efficient in querying text rich data
C) a graph database and can be easily queried, especially for imaging data
D) not scalable therefore provides security and stability
E) not open source, with proprietary technology that cannot be breached by outside hacker

Answer: B) a NoSQL database management system and is highly efficient in querying text rich data

160 | P a g e
S03-5: Which of following is one of the common issues encountered when clinicians are developing
SMART-on-FHIR® CDS with developers?

A. Could be hard to have access to a FHIR® database/server. The database might not have the clinical
data of your interest (e.g., lab values of the liver function test.)
B. Mock EHR might not be available even when the team has a FHIR® server - building a barrier to the
usability testing.
C. Designing UI/UX requires serious feedback from clinicians
D. Medical calculators could be the serious contenders.
E. All of Above

Answer: E. All of Above

S04: Presentations - Evidence-based Care Improvement

Article Relationship Visualization to Support Evidence-based Practice & Knowledge Discovery

--Po-Yin Yen

S04-1: What contributes to the highest quality of evidence for clinical practice?

A. Rapid review
B. Systematic Review
C. Literature Review
D. Scoping Review

Answer: B. Systematic Review

Explanation: Systematic reviews are carefully synthesized research evidence designed to answer
focused clinical questions, and provides the best evidence for Evidence-Based Practice (EBP).

Reference: Stevens KR. Systematic reviews: the heart of evidence-based practice. AACN Clin Issues. 2001
Nov;12(4):529-38.

Melnyk BM. Achieving a high-reliability organization through implementation of the ARCC model for
systemwide sustainability of evidence-based practice. Nursing administration quarterly. 2012;36(2):127-
135.

161 | P a g e
S04-2: What is the most commonly reported barrier to evidence-based clinical decision-making? Lack
of…

A. time to process new information

B. literature with high level of evidence
C. interprofessional communication
D. departmental leadership

Answer: A. time to process new information

Explanation: Although it is well-established that EBP improves healthcare quality and patient outcomes
and reduces morbidities, mortality, medical errors, costs, and geographic variation of healthcare
services, it is not standard practice by the majority of clinicians across the United States. A major barrier
to EBP is that EBP involves time-intensive steps that are burdensome and lead to information overload.

Primary resources for identifying evidential studies include health science bibliographic databases such
as MEDLINE, CINAHL, and PsycINFO. Over 12,000 studies are published weekly, including more than 300
randomized trials. This current volume of evidence is overwhelming and unmanageable to clinicians.

References: Melnyk BM, Fineout-Overholt E. Evidence-based practice in nursing & healthcare: A guide
to best practice. Lippincott Williams & Wilkins; 2011.

Melnyk BM. The Evidence‐Based Practice Mentor: A Promising Strategy for Implementing and Sustaining
EBP in Healthcare Systems. Worldviews on Evidence‐Based Nursing. 2007;4(3):123-125.

Beckett M, Quiter E, Ryan G, et al. Bridging the gap between basic science and clinical practice: the role
of organizations in addressing clinician barriers. Implementation Science. 2011;6(1):35.

Prorok JC, Iserman EC, Wilczynski NL, Haynes RB. The quality, breadth, and timeliness of content
updating vary substantially for 10 online medical texts: an analytic survey. J Clin Epidemiol.
2012;65(12):1289-1295.

162 | P a g e
Characterizing Cutaneous Lupus Erythematosus with Meta-Analysis Using STARGEO

--Osama El-Sayed

S04-3: A researcher is interested in performing a meta-analysis with data from the gene expression
omnibus. What is a potential barrier to using GEO in one's research?

A. There is no standard way of filling out the metadata for the study
B. There is no information included about how the samples were processed
C. GEO does not contain any data on context of the study design
D. GEO is not accessible to the public and requires a subscription to access its data

Answer: A. There is no standard way of filling out the metadata for the study

Explanation: While most studies included the necessary information on sample processing as well as a
link to the paper that it is associated with, this data is not necessarily included under the same header
for each study. This means that it may take longer to figure out whether studies are relevant or not. GEO
is a public repository.

Reference: GEO Overview - GEO - NCBI. National Center for Biotechnology Information.
https://www.ncbi.nlm.nih.gov/geo/info/overview.html (accessed April 27, 2018).

S04-4: What is one way that generating disease signatures from meta-analyses can improve clinical
care?

A. Disease signatures that are generated from meta-analysis are individual specific, which means that
we can find unique treatments for each individual
B. There is no way to associate disease signatures with potential treatments
C. Using pathway analysis, we can find treatments that are known to target genes that are upstream
in the disease signature
D. Using pathway analysis, we can determine the odds that a potential treatment will work before we
know what molecule it interacts with

Answer: C. Using pathway analysis, we can find treatments that are known to target genes that are
upstream in the disease signature

Explanation: Running a pathway analysis on the disease signature allows us to see the upstream
regulators that are most likely to be differentially regulated. We can then use this information to find
potential treatments based on the information we have on which treatments interact with those
upstream regulators.

163 | P a g e
Reference: Ingenuity Pathways Analysis (IPA): Using the Ingenuity Knowledgebase understand genes
and disease. Center for Human Health and the Environment.
https://chhe.research.ncsu.edu/wordpress/wp-content/uploads/2015/10/IPA-Training-slides-Search-
and-Explore-2016_04.pdf (accessed April 26, 2018).

Digital Health Innovation System Integration Assessment: A methodology for streamlined early-stage
evaluation

--Bailey Griffin

S04-5: While there is increased interest and investment in digital tools to improve the outcomes,
experience and cost of care, the potential impact of an early stage digital health tool can be difficult to
ascertain. Our abstract outlines our process and framework for assessing early stage digital health tools
and the results of our retrospective review of recommendations given to companies to better align their
product with system needs. The most common recommendation given to companies included:

A) Do better market research and hire consultants to help you

B) Identify individuals who have purchasing power at an organization

C) Properly identify the appropriate level of the health system at which the product is likely to be
purchased and increase value to the primary payer

D) Focus only on the end needs of patients. The needs of providers and other users are irrelevant since
we’re all most interested in improving the patient experience.

Answer: C) Properly identify the appropriate level of the health system at which the product is likely to
be purchased and increase value to the primary payer

Explanation:

A) Is not correct as our recommendations do not outline the need to hire a consultant to help any
company with determining value proposition framing

B) While it is important to identify your target customer, this Answer is incomplete. Additionally, simply
identifying someone with purchasing power at the organization level may not be enough, there are
many other system stakeholders who have decision making power that will impact your product.

C) This answer is correct as it accurately reflects the most often recommendation given to the
companies assessed in our program.

D) Is wrong since the needs of all users of any system need to be considered in the development and
implementation of a tool.

Reference: Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, Court CA, et al. Beyond adoption: A
new framework for theorizing and evaluating nonadoption, abandonment, and chellenges to the scale
up, spread, and sustainability of health and care technologies. J Med Internet Res. 2017;19(11):e367.

164 | P a g e
S04-6: Our Digital Health Innovation System Integration Assessment framework is focused on providing
individuals and organizations with:

A) An early stage assessment/evaluation of digital health tools to better allocate scarce resources.

B) A basis to skip validation of a digital health tool via rigorous scientific inquiry.

C) An alternative to the regulated procurement processes at an organization.

D) A way to feel engaged with the small to medium sized enterprises in digital health.

Answer: A) An early stage assessment/evaluation of digital health tools to better allocate scarce
resources.

Explanation:

A) was our intended purpose of the framework.

B) while the assessment using the framework may signal whether or not a product is sufficiently
validated or ready to be further validated, the output of the framework will never be sufficient to skip
validation of a digital health tool (e.g. a mobile app to manage depression).

C) unfortunately standard procurement processes will still apply regardless of the output of the
assessment. However, the framework could be a very complimentary tool to the scoring rubric in
procurement activities.

D) feeling engaged with the digital health enterprise community is not the goal of this tool.

Reference: Burke RE, Shojania KG. Rigorous evaluations of evolving interventions: can we have our cake
and eat it too? BMJ Qual Saf. 2018;27:251–4.

S05: Presentations - Clinical Decision Support Embedded Predictive Models

Validating Clinic Workflow Models for Prediction

--Rafael Fricks

S05-1: Which of the following statements best describes over-fitting?

a. As model complexity increases, model predictions show decreased error in validation sets while
training set error increases.
b. Over-fitting can be determined without the use of independent data sets.
c. An over-fit model provides an optimistic estimate of model quality when evaluated against the
training set.
d. Hypothesis testing (i.e. Kolmogorov-Smirnov Test) cannot be used to detect over-fitting.

165 | P a g e
Answer: c. An over-fit model provides an optimistic estimate of model quality when evaluated against
the training set.

Explanation: Over-fitting, in plain language, is when a model becomes so well adapted to nuances in a
particular data set that it loses accuracy in predicting similar data in a different sample. Over-fitting is
characterized by an increase in validation set error even as training error decreases with increasing
model complexity, the opposite of choice (a). Since the prediction error tends to decrease in the training
set, choice (c) is correct as typically an over-fit model will perform worse than expected against new
data. To determine over-fitting, it is necessary to compare performance on an independent data set,
usually referred to as a validation or test set in context (choice (b) is incorrect). Finally, while it is more
common to use measures such as squared error or absolute error, it is possible to use test statistics from
hypothesis testing as a measure of error (choice (d)). For more reading on model testing, see(1).

Reference: Hastie T. The elements of statistical learning : data mining, inference, and prediction.
Friedman JH, Tibshirani R, editors. New York: Springer; 2009.

S05-2: Choose the answer that best completes the statement: “K-fold cross-validation…”

a. reduces the influence of any particular set of data.

b. should divide data into five subsets.

c. requires at least 100 observations.

d. should be used over other model assessment techniques such as bootstrapping.

e. is a feature introduced in MATLAB 2017a.

Answer: a. reduces the influence of any particular set of data.

Explanation: K-fold cross-validation subdivides a set into k subsets, where each set is used as the
evaluation set once (1). This is an improvement over standard cross-validation that selects one
evaluation set, as it puts less emphasis on the selection of the evaluation set (choice (a)). There are
however several choices for k, where five is a common choice but not appropriate in all situations
(choice (b)). Similarly, while it is desirable to have ‘enough’ data to make subsets, ‘enough’ is
determined in context. For instance, few observed subjects (<100) with several fields of data per subject
may be suitable for subdivision (choice (c)). Other methods may be appropriate in evaluating certain
types of model, and variations such as leave-one-out or bootstrapping exist (choice (d)). Each method
provides trade-offs in model evaluation and can be chosen based on the application (1). Finally, cross-
validation is not specific to any software package (choice (e)), though several packages may have
convenient functions implemented.

Reference: Hastie T. The elements of statistical learning : data mining, inference, and prediction.
Friedman JH, Tibshirani R, editors. New York: Springer; 2009.

166 | P a g e
Patient-Specific Explanations from Risk Prediction Models

--Amin Tajgardoon

S05-3: Machine learning models are being increasingly developed to predict clinical outcomes such as
mortality, morbidity, and adverse events. Among the following choices, which one enable clinicians to
engender trust in a sophisticated machine learning model with excellent prediction power?

A. The model achieves a high area under the ROC curve (AUROC)
B. Satisfy the assumption that training data samples are independent and identically distributed
C. Provide simple and reliable patient-specific explanations for each prediction
D. The model has a very low generalization error

Answer: C. Provide simple and reliable patient-specific explanations for each prediction

Explanation: Sophisticated predictive models with excellent performance are reported in the literature
at an increasing pace. These models in most cases are regarded as black boxes that produce a prediction
for an outcome, along with a probability reflecting the certainty, from the features of a patient case.
However, for such models to be practically useful in prevention, diagnosis, prognosis, treatment and
other activities in clinical care, it is critical to provide clinicians with simple and reliable patient-specific
explanations for each prediction. Such explanations will enable clinicians to engender trust in the
predictions, interpret them in the clinical context, and even influence the clinical decisions they make.

References: 1. Štrumbelj E, Bosnić Z, Kononenko I, Zakotnik B, Kuhar CG. Explanation and reliability of
prediction models: the case of breast cancer recurrence. Knowledge and information systems. 2010 Aug
1;24(2):305-24.

2. Luo G. Automatically explaining machine learning prediction results: a demonstration on type 2

diabetes risk prediction. Health information science and systems. 2016 Dec 1;4(1):2.

S05-4: Which of the following definitions best describes “prediction explanation” in the context of a
machine learning model that predicts a clinical outcome?

A. Present to the user the model’s prediction score for a clinical outcome

B. Present an interpretation of the model to the user in terms of structure and parameters

C. Allow the user to select a model with the best accuracy in predicting the clinical outcome

D. Provide the user with an understandable interpretation of the prediction for an individual to
whom the model is applied

Answer: D. Provide the user with an understandable interpretation of the prediction for an individual to
whom the model is applied

167 | P a g e
Explanation: There is a distinction between model explanation and prediction explanation.

Model explanation provides an interpretation of the model to the user in terms of structure and
parameters, and is used in discovery. Some predictive models are more easily interpretable such as
decision trees and linear models. Often interpretable models have poorer predictive performance than
more abstract models such as random forests, support vector machines and neural networks.

Prediction explanation provides an interpretation of the prediction for an individual to whom a model is
applied, and will potentially be different from individual to individual. Acceptable prediction
explanations possess two properties: 1) explanation use concepts that are understandable to the user
such as variables that are not modified or transformed; and 2) the explanation is simple or parsimonious
so that it is readily and rapidly grasped by the user.

Prediction explanations tend to be simpler than model explanations.

Reference: Lipton ZC. The mythos of model interpretability. arXiv preprint arXiv:1606.03490. 2016 Jun
10.

Leveraging Electronic Health Record Data to Predict Surgery Cancellation

--Nick Pratap

S05-5: In this study we implemented machine learning algorithms to explore variables from patient
records to predict day-of-surgery cancellation. Overfitting is one of the common problems in machine
learning which will decrease the generalizability of the model. Which machine learning algorithm has
the highest risk of overfitting?
A. Gradient boosted LR
B. Random Forest
C. Regularized logistic regression
D. Classification tree

Answer: D. Classification tree

Explanation: Classification trees are non-parametric models that have more flexibility and are prone to
overfit training data. The other three algorithms use regularization or ensemble techniques to avoid
overfitting.

Reference: Kuhn M, Johnson K. Applied predictive modeling. New York: Springer; 2013 May 17.

S05-6: In this study we utilized a iterative step-forward method to do feature selection. What is the
primary objective of doing feature selection?
A. It identifies and selects key predictors to improve the interpretability of the model.

168 | P a g e
B. It increases the complexity of the model.
C. It creates new combinations of variables in order to reduce the dimensionality of the data.
D. It decreases the prediction performance of the model.

Answer: A. It identifies and selects key predictors to improve the interpretability of the model.

Explanation: Feature selection methods reduce the complexity of the model and make it easier to
understand and explain. It chooses a subset of original variables to reduce the dimensionality of the data
without creating or changing them. Moreover, doing feature selection can improve the performance of
the model if the optimal subset is chosen.

Reference: Kuhn M, Johnson K. Applied predictive modeling. New York: Springer; 2013 May 17.

S06: Presentations - Informatics Application and Research to Enhance Surgical Practice

Adapting Technology into the Surgical Environment

-- Alex Langerman

S06-1: Information targeted towards surgical circulating nurses is best presented:

A. for the entire case at the start of a case (e.g., during time out or a preference card)

B. through a sterile interface to prevent surgical site infection (SSI)

C. in a mobile format that can be transported within and outside the operating room

D. with anatomic images to illustrate the choices between surgical techniques

E. for all of the cases of the day, to allow "anticipatory guidance"

Answer: C. in a mobile format that can be transported within and outside the operating room

Explanation: Circulating nurses are the most mobile team members within the room, and often leave
the room to get additional supplies or deliver specimens. They are not scrubbed in or sterile. They can
benefit from anatomic knowledge, but do not participate in moment to moment decisions regarding
surgical technique (the scrub and particularly the surgeon do this). For situational awareness,
information should be given on a step-by-step basis, not all at once at the beginning (non-surgeon team
members change throughout a case and therefore not all members were present at the time out); these
step data should include guidance on upcoming tasks within a case (not upcoming cases) to support
anticipatory guidance.

Reference: Julie M. Mhlaba, Laura W. Christianson, Stuart J. Davidson, et al. Field research in the
operating room. Ergonomics Design. 2016;24(4):10-9

169 | P a g e
S06-2: The most common and financially impactful result of poor communication regarding tools and
tasks in the operating room is:

A. tool damage and repair/replacement costs (RRC)

B. over-provisioning of surgical supplies

C. lapses in sterility from improper draping

D. reductions in mean turnover time (TTm)

E. over-staffing of surgical cases

Answer: B. over-provisioning of surgical supplies

Explanation: Poor communication around tools and tasks in the operating room has many effects,
including slower turnover times, staff member exits out of the room to retrieve missing but necessary
items, and surgical delays. One of the most common and impactful problems is over-provisioning of
surgical supplies, which results in wasted money on open-but-unused items and cognitive overload on
staff. While over-provisioning and opening of unnecessary trays can result in more frequent
reprocessing and tool breakdown, the cost of this is much smaller than the other costs associate with
over-provisioning. Improper draping sufficient to cause a lapse in sterility can occur but this is not a
frequent occurrence.

References: Stockert EW, Langerman A. Assessing the magnitude and costs of intraoperative
inefficiencies attributable to surgical instrument trays. J Am Coll Sur. 2014;219(4)646-55

Zygorakis CC, Yoon S, Valencia V, et al. Operating room waste: disposable supply utilization in
neurosurgical procedures. J Neurosurg. 2017;126(2):620-5

EHR Needs of the Surgical Subspecialist: Insights from System Selection

--Saira Haque

S06-3: What functionality should the surgical specialist consider when selecting an EHR?

A. Reconciling long term meds from outside vendors

B. Presenting longitudinal data for chronic disease

C. Targeting information pertinent to surgical problem

D. Graphing of BMI, blood pressure and pulse over time

Answer: C. Targeting information pertinent to surgical problem

170 | P a g e
Explanation: The relationship between the surgical subspecialist and the patient is characterized by
episodic care that is targeted. The answers A, B and D are more appropriate for primary care providers
or specialists whose relationship with the patient is characterized by long-term care management.

Reference: https://www.facs.org/

S06-4: When an EHR vendor demonstrates the system’s functionality, which points should be clarified?

A. the programming and coding employed in the programming software

B. the ability to adapt the system to clinical and administrative workflows
C. the ability of the system to withstand cyber-attacks from external sources
D. workflow of enterprise systems beyond the scope of the current project

Answer: B. the ability to adapt the system to clinical and administrative workflows

Explanation: Clinical and administrative workflows of the system being reviewed are an important piece
of the demonstration; the other items are not relevant to a demonstration of system functionality.

References: Sanders DS, Read-Brown S, Tu DC, et al. Impact of an Electronic Health Record Operating
Room Management System in Ophthalmology on Documentation Time, Surgical Volume, and Staffing.
JAMA Ophthalmol. 2014;132(5):586-592.
https://jamanetwork.com/journals/jamaophthalmology/fullarticle/1851727

Duwayri Y et al. The vascular surgeon's roadmap to success in the Quality Payment Program. Journal of
Vascular Surgery, Volume 65 , Issue 5 , 1536. http://www.jvascsurg.org/article/S0741-5214(17)30332-
4/fulltext

Challenges of Electronic Health Record Transition for Surgical Services: A Qualitative Observational Study
of “Go Live”

--Megan Shroder

S06-5: In our study of surgical practices during an electronic health record transition, one of the most
common sources of problems and confusion was related to:

A. Admission orders
B. Phases of care
C. Operative notes
D. Inpatient documentation

Answer: B. Phases of care

171 | P a g e
Explanation: Phases of Care used in the Epic System were a common point of confusion across surgical
faculty, residents, and staff. There were numerous help tickets regarding lost orders with several
potential patient safety issues. Activities done in the different Phases of Care in surgery are highly
specialty-specific. Workflows varied across surgical specialties and were not amenable to instruction
with simple examples. Significant errors related to Phases of Care occurred when the timing of surgery
changed, and such changes are the norm rather than the exception in organizations that provide trauma
care and emergency surgery services. Innovative solutions to various problems that arose were
developed and applied by providers until more formalized solutions could be implemented, as has been
seen in some previous studies on electronic health record transitions.

Reference: J Tim Scott, Thomas G Rundall, Thomas M Vogt, John Hsu. "Kaiser Permanente's experience
of implementing an electronic medical record: a qualitative study". BMJ. 2005; 331:1313-1316.

S06-6: An important recommendation from our study of surgical practices during an electronic health
record transition at an academic medical center is dedicated training for:
A. Work shared across teams of residents, staff, and faculty
B. Emergency department to operating room admissions
C. Physician scheduling of procedures from the outpatient setting
D. Documentation for trauma resuscitation

Answer: A. Work shared across teams of residents, staff, and faculty

Explanation: In our study, the surgical teams were not well prepared for functioning as group and
academic practices (i.e., with support of trainees). Provider training focused on individual charting
responsibilities, with it largely being assumed that the physician provider would perform all tasks and be
consistently available to enter and sign orders. In the emergent setting, surgeons are busy doing
procedures or resuscitating patients. Training for faculty was done separately from trainees and staff.
The faculty training did not address how work would be shared among team members.

Reference: J Tim Scott, Thomas G Rundall, Thomas M Vogt, John Hsu. "Kaiser Permanente's experience
of implementing an electronic medical record: a qualitative study". BMJ. 2005; 331:1313-1316.

172 | P a g e
S07: Panel - LEANing into the Quality Payment Program: Using Agile Strategies for Real Practice
Improvement

--Julia Skapik, et al

S07-1: Which is NOT one of the four reporting categories in the Quality Payment Program (QPP) Merit-
based Incentive Payment Program for individual providers and groups?
A. Advancing Care Information
B. Improvement Activities
C. Clinical Decision Support
D. Quality Measures

Answer: C. Clinical Decision Support

Explanation: The Quality Payment Program is part of the Medicare Access and CHIP Reauthorization Act
of 2015 (MACRA), which eliminated the Sustainable Growth Rate (SGR) and created a broad-based pay
for performance program for clinicians (physicians and beyond) and accountable care organizations.
Clinicians have two tracks to choose from in the Quality Payment Program based on their practice size,
specialty, location, or patient population: Merit-based Incentive Payment System (MIPS) or Advanced
Alternative Payment Models. The MIPS arm of the program is targeted at fee-for-service based
individual and group physicians and has four components: Advancing Care Information, Improvement
Activities, Quality Measures, and Cost. Clinical Decision Support is not included in the reporting
requirements for this program.
References: https://qpp.cms.gov/
https://www.federalregister.gov/documents/2017/11/16/2017-24067/medicare-program-cy-2018-
updates-to-the-quality-payment-program-and-quality-payment-program-extreme

S07-2: Lean is a tool that has been around the manufacturing industry for a long time and only more
recently has it been applied to healthcare organizations. This is partially due to the healthcare
reimbursement system changing from fee for service to pay for value incentivizing more and more
organizations to undergo Lean transformations. Taking on a Lean transformation can provide an
organization significant gains in productivity, quality, and cost reduction but requires a cultural shift.
This type of a shift is driven by leadership and needs to have the right principles in place to transform
the culture. Which of the following are the most appropriate management principles to create a Lean
culture?

A) Value to the customer, open communication, participate and contribute, pursue excellence, share
knowledge, keep it simple.
B) Be a leader, exceed customer expectations, continuously improve, keep it simple, seek perfection,
respect for every individual.
C) Attitude of continuous improvement, value creation, unity of purpose, respect for front-line
workers, visual tracking, flexible regimentation.

173 | P a g e
D) Create value at every step, quantity over quality, keep it simple, seek relentless perfection, share
knowledge, listen and communicate.

Answer: C) Attitude of continuous improvement, value creation, unity of purpose, respect for front-line
workers, visual tracking, flexible regimentation.

Explanation: To achieve a successful Lean transformation and sustain the gains you must change the
culture of the organization. These 6 principles: attitude of continuous improvement, value creation,
unity of purpose, respect for front-line workers, visual tracking, and flexible regimentation have been
found across multiple successful Lean organizations and provide a starting framework for leaders to
instill as they take on their own lean journeys.

References: John S. Toussaint, MD, and Leonard L. Berry, PhD.The Promise of Lean in Healthcare. Mayo
Clinic Proceedings [Internet]
http://www.mayoclinicproceedings.org/article/S0025-6196(12)00938-X/fulltext

S07-3: Capturing and extracting valid and reliable data from certified EHRs requires iterative
improvements to the certified EHRs, workflow, data entry screens and strategies to capture valid data
based on the measures requirements and specifications. Which of the following is most likely the case in
terms of iterative effort?
A. 80% of the effort is by the vendor to provide modifications within the certified EHR/20% is with the
clinical side to modify workflow to accommodate eCQM data capture
B. 60% of the effort is by the vendor to provide modifications within the certified EHR/40% is with the
clinical side to modify workflow to accommodate eCQM data capture
C. 50% of the effort is by the vendor to provide modifications within the certified EHR/50% is with the
clinical side to modify workflow to accommodate eCQM data capture
D. 20% of the effort is by the vendor to provide modifications within the certified EHR/80% is with
the clinical side to modify workflow to accommodate eCQM data capture
E.10% of the effort is by the vendor to provide modifications within the certified EHR/90% is with the
clinical side to modify workflow to accommodate eCQM data capture

Answer: D. 20% of the effort is by the vendor to provide modifications within the certified EHR/80% is
with the clinical side to modify workflow to accommodate eCQM data capture

Explanation: Eisenberg et al., (ND) in a study done for American Hospital Association on the impact of
meaningful use clinical quality indicators found that early adopters of electronic clinical quality
measures (eCQMs) had significant challenges getting to valid and reliable data for quality improvement.
This study found that iterative modifications were needed to fix the problems with data, with a majority
of the effort required by the clinical team. This has continued to be the case with extracting reliable data
from certified EHRs. There findings indicate 20% of the effort is by the vendor to provide modifications
within the certified EHR/80% is with the clinical side to modify workflow to accommodate valid and
reliable eCQM data capture.

174 | P a g e
References: Eisenberg, F., Lasome, C., Advani, A., Martins, R., Craig, P. and Sprenger, S. (N.D.) A study of
the impact of Meaningful Us electronic clinical quality
measures www.aha.org/content/13/13ehrchallenges-report.pdf

S07-4: Which of the following strategies is most likely to improve problems with electronic quality
measure data capture and reporting?
A. Using only data from a clinical registry
B. Manual abstraction
C. Natural language processing in writing clinical notes
D. Use of a clinical data model associated with a standardized data warehouse

Answer: D. Use of a clinical data model associated with a standardized data warehouse

Explanation: While the use of any of these techniques may be used in the capture or validation of
quality measure data, the most direct and immediately impactful is the use of a data model, such as the
Quality Data Model (QDM). The data model would need to be mapped to data as it is captured into the
system, with that data model information being tagged to it as metadata as it moves into the data
warehouse. For maximum effectiveness, the quality measures targeted should be mapped to the data
model at the beginning of the period of capture and tested throughout the period of performance.

References: Taxiarchis Botsis, Gunnar Hartvigsen, Fei Chen, Chunhua Weng. Secondary Use of EHR: Data
Quality Issues and Informatics Opportunities. AMIA Summit on Translational Bioinformatics. 2010

Johnson III RJ. A Comprehensive Review of an Electronic Health Record System Soon to Assume Market
Ascendancy. J Healthc Commun. 2016, 1:4. DOI: 10.4172/2472-1654.100036

S08: Presentations - Bridging Analytics

POKE-R: Reducing Patient Harm

--James McGlothlin

S08-1: After the analytics have been developed, one of the most important elements to a successful
project is a good deployment strategy for user adoption. When attempting to implement POKE-R in a
specific unit, what is of the following is a key element which will help us succeed?

A. Utilizing specially trained nurses

B. Having one source of truth for all types of orders
C. Performing structured rounding
D. Meeting regularly with informatics professionals
E. Preventing duplicate orders in the EMR

175 | P a g e
Answer: C. Performing structured rounding

Explanation: Having a process for all of a patient's providers to meet and discuss the day’s orders and
events together is key to reducing POKE-R events. The POKE-R project can be implemented without
requiring any special training for nurses and multiple EMRs can be leveraged. It is important to make
sure that clinicians have the information to make decisions on their own on a daily basis without relying
on IT. It is important, as well, to allow clinicians to write any orders in the EMR which they may deem
necessary.

Reference: https://hbr.org/2013/11/a-remedy-for-fragmented-hospital-care

S08-2: To develop a a project such as POKE-R, it is important to establish specific and attainable goals.
Which of the following is a reasonable and significant goal for a POKE-R project.

A. Reduce surgeries

B. Reduce lab draws

C. Reduce medication administrations

D. Change nutritional orders

E. Reduce mortality

Answer: B. Reduce lab draws

Explanation: All lab draws are POKE-R events and it is possible to significantly reduce the number of lab
draws and the cost of lab draws chosen.

While surgeries are POKE-R events, it is not realistic to reduce surgeries. Instead of reducing
medications, the POKE-R project aims to alter the route used for medications.

Nutritional orders are not POKE-R events. Mortality may be reduced but this is not an achievable goal
to make as a primary goal of a project.

Reference: http://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0006174603620369

176 | P a g e
The Clinical Informatics Fellowship Experience: A Qualitative Analysis (qACIF)

--Benjamin Orwoll

S08-3: You are a first-year resident and you find yourself very interested in how the information and
computer systems at your institution can be leveraged to improve care. A faculty member mentioned
the clinical informatics fellowship, but you are not sure what kinds of qualifications you might need to
apply successfully. Of the following, which profile most closely approximates the experiences prior to
fellowship from the cohort of fellows surveyed in the qACIF study?

a. Quality improvement scientists with extensive research experience who want to apply informatics
principles in their work
b. Physicians with bachelors or graduate degrees in computer science intending to apply machine
learning algorithms in health care.
c. Young physicians with interests in technology who are frustrated with the current state of health IT
and desire to be leaders toward future system change.
d. Physicians with IT leadership background who need dedicated training in clinical informatics to obtain
board certification
e. Entrepreneurial physicians who have previous business experience and need to gain a foothold for
their devices in the health information ecosystem.

Answer: c. Young physicians with interests in technology who are frustrated with the current state of
health IT and desire to be leaders toward future system change.

Explanation: This question really addresses information from the qACIF study related to the
demographics and background, as well as motivations, of the first few groups of CI fellows to enter the
fellowship. In general, responders were relatively young (low 30's) and did not tend to have a computer
science background. They tended to express frustration with the current system as a significant
motivator for further work in the area of clinical informatics and expressed the desire to be actors for
future change. In general, CI fellows did not have significant prior independent practice experience.
Though there were aspirations in some of the fellows toward academic research and entrepreneurship,
the most representative answer is C.

S08-4: Clinical informatics fellowships, as they are currently structured, do not specify training in a
specific medical specialty or specialties as a prerequisite to application. Thus, clinical informatics can be
a subspecialty of any medical specialty. Based on the results of the qACIF survey, which of the following
medical specialties is most significantly underrepresented among clinical informatics fellows?

a. Pediatrics
b. Emergency Medicine
c. Pathology
d. General Surgery
e. Internal medicine

177 | P a g e
Answer: d. General Surgery

Explanation: The respondents from the qACIF survey included over 70% of the known active CI fellows
at the time. Though internal medicine, pediatrics, emergency medicine, and pathology were well-
represented, self-reported primary medical specialties in the survey included only one proceduralist: an
interventional radiologist. 2.6% (1/38) is a much lower representation of procedural specialties than
among active physicians. The reasons for the underrepresentation of procedural/surgical specialties
were not explored in this survey. New recruitment strategies and/or curriculum changes may be needed
in order to develop a diverse clinical informatics work force.

References: AAMC 2016 Physician Specialty Data Report

(https://www.aamc.org/data/workforce/reports/457712/2016-specialty-databook.html)

Interdisciplinary Development of Palliative Care EHR Decision Support

--Ruth Bush

S08-5: Palliative care is associated with more efficient resource use and decreased medical costs. The
targeted technology available in the EHR, including clinical decision support could facilitate screening for
earlier identification of patients in or about to be in distress; to provide more appropriate referrals to
specialized care; and to facilitate more rapid communication among patients, caregivers, and clinicians.

Which of the following is a good working definition of palliative care:

a. A short-term approach to providing pain control.

b. Targeted patient education designed to prevent individual chronic disease development.

c. Medical care, which prevents or treats symptoms and side effects of disease and treatment and is
designed to be patient and family-centered.

d. Engagement of the patient in their health decision making through technology such as patient portals
and tracking devices.

Answer: C. Medical care, which prevents or treats symptoms and side effects of disease and treatment
and is designed to be patient and family-centered.

Explanation: The World Health Organization defines palliative care (PC) as, “an approach that improves
the quality of life of patients and their families facing the problem associated with life-threatening
illness, through the prevention and relief of suffering by means of early identification and impeccable
assessment and treatment of pain and other problems, physical, psychosocial and spiritual.” A
reasonable working definition of palliative care is medical care, which prevents or treats symptoms and
side effects of disease and treatment and is designed to be patient and family-centered. While it
contains elements of pain control, patient education, and engagement these choices are too limited and
178 | P a g e
do not recognize the multi-faceted, interdisciplinary care of patient and family, which is expressed in
choice “c”.

References: WHO | WHO Definition of Palliative Care. WHO [Internet]. World Health Organization; 2012
[cited 2017 Apr 6]; Available from: http://www.who.int/cancer/palliative/definition/en/

Vorvick LJ. What is palliative care [Internet]. 2016. Available from:

https://www.nln.nih.gov/medlineplus/ency/patientinstructions/000536.htm

Wysham NG, Hua M, Hough CL, Gundel S, Docherty SL, Jones DM, et al. Improving ICU-Based Palliative
Care Delivery: A Multicenter, Multidisciplinary Survey of Critical Care Clinician Attitudes and Beliefs. Crit
Care Med. 2016/09/13. 2016;

S08-6: Which of the following statements most accurately reflects the results of the palliative care
trigger list feasibility study:

a. The final algorithm version had several hundred variables to ensure it captured every possible
individual needing palliative care.

b. The final algorithm needed to be validated with manual chart review.

c. The trigger list had poor sensitivity; patients retrospectively identified were not likely to have been
seen prospectively by the palliative care team.

d. The algorithm relied primarily on laboratory testing as the basis for the clinical decision support.

Answer: B. The final algorithm needed to be validated with manual chart review.

Explanation: The final algorithm was reduced to 49 variable and retrospectively identified almost 98% of
patients who were visited prospectively by the palliative care team. The algorithm relied on a number
of variables including: race/ethnicity, language, being unresponsive, feeding tube insertions, specialty
bed use in patients with difficulty moving, and two-person movement assistance. Accuracy of the
electronic data verified using manual chart review to provide verification of electronic clinical decision
support algorithm, which is expressed in choice “b”.

Reference: Rhodes RL, Kazi S, Xuan L, Amarasingham R, Halm EA. Initial Development of a Computer
Algorithm to Identify Patients With Breast and Lung Cancer Having Poor Prognosis in a Safety Net
Hospital. Am J Hosp Palliat Care. 2015/07/05. 2016;33(7):678–83.

179 | P a g e
S09: Presentations - Clinical Decision Support - Trends in Optimization

Optimize Up: Using Analytics for Clinical Decision Support Optimization

--Anwar Mohammad Sirajuddin

S09-1: Which one of the following is NOT a potential outcome of Clinical Decision Support (CDS) alerts?

A. Creation of a continuity of care document

B. Prevention of adverse drug events

C. Improvement of reimbursement and regulatory compliance

D. Provide treatment recommendations

Answer: A. Creation of a continuity of care document

Explanation: A key component of the Electronic Health Record are Clinical Decision Support (CDS) alerts
and/or systems that aid the user with clinical decisions. Key outcomes of CDS alerts could be to provide
treatment recommendations, prevent adverse drug events, cost containment and improvement of
reimbursement and regulatory compliance

CDS alerts are not designed to create a continuity of care document (CCD). Continuity of care documents
are structured documents created within the EMR but not within the CDS component.

Reference: https://www.healthit.gov/providers-professionals/faqs/what-clinical-decision-support

S09-2: In order to optimize Clinical Decision Support (CDS) alerts, Memorial Hermann identified
potential areas of inefficient alerting to the clinical users. The main two criteria they reviewed were:

A. Low frequency alerts and high alert override rates

B. Low frequency alerts and low alerts override rates

C. High frequency alerts and low override rates

D. High frequency alerts and high alert override rates

Answer: D. High frequency alerts and high alert override rates

Explanation: Data extraction tools and data visualization tools identify potential areas of inefficient
alerting to the clinical users. Knowing that CDS alerts need to be actionable and measurable, Memorial
Hermann identified these opportunities by looking at two key criteria:

• High frequency alerts (resulting in “alert fatigue”)

• High override rate (suggesting low clinical utility for the alert)
180 | P a g e
Reference: https://www.healthit.gov/sites/.../strategies_for_optimizing_an_ehr_system_0.docx

An Evaluation of Clinical Order Patterns Machine-Learned From Clinician Cohorts Stratified by Observed
Versus Expected 30-Day Mortality Rates

--Jason Ku Wang

S09-3: With the emergence of data-driven clinical decision support tools, in particular those that
aggregate the collective expertise of clinicians and make recommendations based on patterns learned
from historical electronic health record data, defining a robust external reference standard is necessary.
For the case of promoting compliance with clinical practice guidelines, which of the following proves to
be the largest barrier to defining a reference standard, even for common admission diagnoses?

A. There are too few clinical practice guidelines in the literature to assemble a robust reference standard

B. In translating clinical practice guidelines for a given admission diagnosis into a finite reference
standard, subjectivity and interpretation are required

C. There is no systematic way for multiple clinicians with different opinions to reach a consensus on a
single reference standard

D. Clinical practice guidelines derived from more reputable literature source should be assigned more
weight than those derived from less reputable sources

Answer: B. In translating clinical practice guidelines for a given admission diagnosis into a finite
reference standard, subjectivity and interpretation are required

Explanation: With inherent variability in medical practice, clinical practice guidelines for a given
admission diagnosis may vary from guideline to guideline. Furthermore, since the guideline may be
written in a manner that emphasizes certain procedures, treatments, etc., more than others, it is left up
to the clinician reading the guideline to determine whether or not a clinical order should be included in
the reference standard. This subjectivity proves to be the largest challenge. Clearly, for rare admission
diagnoses, the number of published clinical practice guidelines diminishes substantially, precluding the
creation of a robust reference standard. However, the question prompt specifically states “even for
common admission diagnoses,” implying that the availability of clinical practice guidelines is not the
main issue here. Note also that the process of achieving consensus between multiple clinicians with
different opinions can be achieved via adjudication.

Reference: Wang JK, Schuler A, Shah NH, Baiocchi MTM, Chen JH. Inpatient Clinical Order Patterns
Machine-Learned from Teaching Versus Attending-Only Medial Services. AMIA Informatics Summit.
2018.

181 | P a g e
S09-4: Clinical order patterns derived from data-mining electronic health records can be a valuable
source of decision support content. However, learned patterns can be compromised by underlying
provider experience as clinical orders of less experienced providers (trainees) can exhibit greater
variability and worsened patient outcomes than those of experienced providers (attendings). Based on
the conclusions of this podium abstract, which choice of training data will invariably yield the greatest
compliance with clinical practice guidelines?

A. Training indiscriminately on all available data

B. Always training on a cherry-picked subset of the most expert clinicians

C. Always weeding out data generated by less experienced clinicians (e.g. trainees) to improve model
accuracy

D. No single approach will invariably yield the best outcomes

Answer: D. No single approach will invariably yield the best outcomes

Explanation: In alignment with clinical practice guidelines, we find that the low-mortality clinician model
outperforms the crowd clinician model across 6 common admission diagnoses. However, we cannot
conclude that abiding by the same approach (e.g. always training exclusively on low-mortality “experts”
or always training on all available data) will invariably result in the greatest compliance with clinical
practice guidelines, across all diagnoses.

Reference: ang JK, Schuler A, Shah NH, Baiocchi MTM, Chen JH. Inpatient Clinical Order Patterns
Machine-Learned from Teaching Versus Attending-Only Medial Services. AMIA Informatics Summit.
2018.

Developing a Learning Electronic Medical Record System

--Shyam Visweswaran

S09-5: A learning EMR identifies and highlights EMR data that is relevant for a given patient and clinical
context. Which of the following can be enabled by such a learning EMR?
A. Faster access to patient data
B. Increased responsiveness of EMR to mouse clicks
C. Reduced time needed to assess the patient’s condition
D. More coherent display of patient data

Answer: C. Reduced time needed to assess the patient’s condition

Explanation: The learning EMR focuses the physician’s attention on relevant patient data. This could
help reduce the time needed to assess the patient’s condition and improve the quality of the resulting
judgments, enabling improved decision making, reduced medical errors, and greater efficiency.

182 | P a g e
References: King, AJ, Cooper, GF, Hochheiser, H, Clermont, G, Visweswaran, S. Development and
preliminary evaluation of a prototype of a learning electronic medical record system. In: AMIA Annual
Symposium Proceedings. 2015 Nov 17; 2015:1967-75.

S09-6: A learning EMR discovers what type of patterns from the EMR database?
A. Medical appointment patterns
B. Patient data access patterns
C. Patterns of comorbidities
D. Medication prescription patterns

Answer: B. patient data access patterns

Explanation: The learning EMR uses a data-driven approach to predict and highlight data that are most
relevant to a patient case. It does so by analyzing patterns of patient data access in the EMR. From these
patterns it learns a model of which EMR data are likely to be relevant to highlight for a given patient and
clinical context.

References: King, AJ, Cooper, GF, Hochheiser, H, Clermont, G, Visweswaran, S. Development and
preliminary evaluation of a prototype of a learning electronic medical record system. In: AMIA Annual
Symposium Proceedings. 2015 Nov 17; 2015:1967-75.

S10: Presentations - Collaboration, Pathways and Training in EHR Implementations

Using a Cloud-based Collaboration and Dissemination Platform to Support the Implementation of an

Evidence-based Clinical Pathway Program Across a Multi-Entity Academic Healthcare System

--Emilia Flores

S10-1: Which of the following best describes a clinical pathway?

A. A graphical overview of a process

B. A document that contains criteria for guiding diagnosis and treatment of diseases or conditions
C. A method for translating evidence into local work structures and processes
D. A document that aims to standardize care for a specific clinical problem in a specific population
E. Structured and multidisciplinary plan of care, used to incorporate evidence into local practice, that
has criteria-based progression, and aims to standardize care for a specific clinical problem in a specific
population.

183 | P a g e
Answer: E. Structured and multidisciplinary plan of care, used to incorporate evidence into local
practice, that has criteria-based progression, and aims to standardize care for a specific clinical problem
in a specific population.

Explanation: An analysis of the seminal articles on clinical pathways by Kinsman et at. (2010) found that
clinical pathways were characterized by the following descriptions: the intervention is a structured
multidisciplinary plan of care and meets three of the following four criteria: (1) the intervention is used
to incorporate guidelines or evidence into local practice; (2) the intervention contains detailed steps for
the plan of care; (3) the intervention has timeframes or criteria-based progression; and (4) the
intervention aims to standardize care for a specific clinical problem, procedure or episode of healthcare
in a specific population.

Options A, B, C, and D describe parts of the definition of a clinical pathway, but do not encompass all
characteristics.

References:

Kinsman L, Rotter T, James E, Snow P, Willis J. What is a clinical pathway? Development of a definition to
inform the debate. BMC Medicine. 2010;8(1):31.

Rotter T, Kinsman L, James E, Machotta A, Willis J, Snow P, Kugler J. The effects of clinical pathways on
professional practice, patient outcomes, length of stay, and hospital costs: Cochrane SystematicRreview
and Meta-analysis. Evaluation & the Health Professions. 2012;35(1):3-27.

S10-2: Research findings have shown that patients do not always receive evidence-based care and
treatment, despite the availability of evidence and published guidelines (Agency for Healthcare Research
and Quality (2016)). Clinical pathways are one method for integrating evidence into practice; however,
the development and implementation approach can have a significant impact on provider adoption.
Which of the following approaches for clinical pathway development and implementation is most likely
to have a positive impact on provider uptake?

A. Using a rigorous process and framework for clinical pathway development and dissemination,
including identification of a clinical owner, basing the pathway on a synthesis of current guidelines, use
of a multidisciplinary stakeholder group to review the evidence and adapt based on the local setting,
and identification of barriers and facilitators to dissemination and tailoring the plan accordingly
B. Use of technology to provide centralized and easy access to clinical pathways across all practice
locations
C. Incorporating access to clinical pathways in provider workflow
D. Use of smart alerts to identify relevant clinical pathways for a given patient
E. Using a rigorous process and framework for pathway development and dissemination, having a
centralized and easy to access resource for storing pathways, and use of technology to provide access
to relevant pathways in the provider workflow

184 | P a g e
Answer: E. Using a rigorous process and framework for pathway development and dissemination,
having a centralized and easy to access resource for storing pathways, and use of technology to provide
access to relevant pathways in the provider workflow

Explanation: Utilizing a systematic approach and framework to identify a clinical owner and recruit a
representative stakeholder group, conducting rapid reviews of the evidence, adapting the evidence to
the local setting, converting evidence-based guidelines into clinical pathways, and then utilizing
technology to facilitate dissemination and thoughtful integration into provider workflow can increase
the likelihood of clinical pathway adoption (Davies, Walker, Grimshaw, 2010).

Options A, B, C, and D are components of a systematic approach for pathway development,

dissemination, and implementation.

References:

Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge
translation: time for a map?. Journal of Continuing Education in the Health Professions. 2006;26(1):13-
24.

ADAPTE Collaboration (2010) Manual for guideline adaptation – version 2.0. Available at ADAPTE
website: http://www.g-i-n.net/document-store/working-groups-documents/adaptation/adapte-
resource-toolkit-guideline-adaptation-2-0.pdf (last accessed 20 March 2018).

Built-in Care Pathways and Optimization of Electronic Medical Record System for Efficient Throughput at
a High-Volume Academic Medical Institution

--Rika Ohkuma

S10-3: A surgical service line A in an academic medical institution investigated their current workflow to
find opportunities to reduce clinical variance postoperatively. They found that postoperative orders
varied by attending surgeons and ordering residents. A current clinical pathway was developed four
years ago, but rotating residents were not aware of the care pathway. Which is the most appropriate
next step to reduce variance of clinical treatment by ordering physicians?

A. More education sessions for ordering physicians to think and order more carefully

B. Develop order sets aligned to the care pathway and have them embedded in EMR

C. Develop a more detailed care pathway so that they can follow each process

D. More process alerts in EMR to raise awareness of the care pathway

Answer: B. Develop order sets aligned to the care pathway and have them embedded in EMR

Explanation: The goal of the service line is to standardize postoperative care among all physicians.
Integrating expected orders for each procedure into EMR as order sets makes it easy for ordering

185 | P a g e
physicians to follow the care pathways without building them from the scratch. A brief education
session for rotating residents about the order sets would help to be compliant to a new workflow (A). A
detailed care pathway might be difficult to follow without a systematic support (C). The more often
alerts goes off, the less physicians pay attention to them.

S10-4: A surgical service line B investigated their current workflow to find opportunities to reduce
postoperative length of stay. They found that front line care givers are not aware of benchmark and
expected length of stay. Inefficient timely preparation caused a delay in hospital discharge or a transfer.
Which is the most appropriate next step to reduce unnecessary delay to discharge patients?

A. Start a discharge plan in EMR when patients become clinically ready

B. Start a discharge plan on the day of surgery and update an estimated discharge date daily in EMR

C. Set a target length of stay for each procedure and show an expected discharge date in a patient
chart to set a same expectation in a multidisciplinary team

D. Track overall service line length of stay monthly

Answer: C. Set a target length of stay for each procedure and show an expected discharge date in a
patient chart to set a same expectation in a multidisciplinary team

Explanation: The goal of the service line is to set a same expectation among front line care givers. Visual
cue of target length of stay and discharge dates help them to advance the level of care to meet the
discharge goals. It must be too late if you start a discharge plan when patients are clinically ready (A).
Early discharge planning and daily updates are critical (B), however, the daily duplicated process should
be avoidable by step-wise process guidance for a smooth discharge. Outcome report (D) would help to
understand an overall baseline and a longitudinal trend, and procedure specific (patient customized)
goal should be considered for more accurate prediction.

Evaluation of a Peer Training Model for Electronic Health Record System Implementation

--William Lancaster

S10-5: How does the peer training model (PTM) differ from the traditional training model?

a. With the PTM, providers train other providers in their same or similar specialty as to how to
use the new technology.
b. With the PTM, certified trainers train all of the providers in how to use the new technology.
c. The PTM is less costly than the traditional model.
d. The traditional model allows providers to become more engaged earlier on in the
implementation process.
e. The PTM does not require additional time away from clinical activities.

186 | P a g e
Answer: a. With the PTM, providers train other providers in their same or similar specialty as to how to
use the new technology.

Explanation: The traditional training model uses certified trainers experienced in an EHR to teach
providers how to use a new system. The peer training model coaches providers, who are well known
and respected at their institution, to train their peers in the use of the EHR. A purported benefit of this
approach learning the system by end-users in a manner that better conforms to their workflow as
opposed to learning the various system functions.

Reference: R. A. Orfaly, J. C. Frances, P. Campbell, B. Whittemore, B. Joly, and H. Koh, “Train-the-trainer

as an educational model in public health preparedness,” J. Public Health Manag. Pract. JPHMP, vol.
Suppl, pp. S123-127, Nov. 2005.

S10-6: The majority of the peer trainer survey respondents identified with which of the following
statements:

a. I do not feel that my peers learned the system better because it was taught by a peer

b. I felt more prepared for Go-Live after having served as a peer trainer

c. Fewer problems were identified earlier as a result of using the peer training model

d. I sacrificed too much clinical time in order to serve as a peer trainer.

Answer: b. I felt more prepared for Go-Live after having served as a peer trainer

Explanation: Those that participated as peer trainers overwhelmingly felt more prepared for Go-Live
(78.9%). This likely reflects a greater familiarity with the system that was achieved by spending hours
teaching the system to their colleagues and answering their questions. Because of this, peer trainers
were able to serve as an additional layer of support for their colleagues during Go-Live which may have
eased the transition.

Incorrect Answer A: On average, participants felt that their peers learned the system better because of
the peer training model (68.0%) This agreement is likely due to the fact that the peer trainers had
greater familiarity with their peers’ workflow than a certified trainer.

Incorrect Answer C: The majority of survey respondents (61.6%) also felt that the peer training process
allowed for the earlier identification of problems prior to Go-Live. Problems noted during training and
described in the comments were largely related to built note templates and order sets that did not
match the users’ workflows correctly. Many of these were revised or optimized prior to Go Live.

Incorrect Answer D: 50.1% of trainers did not feel that they sacrificed. too much clinical time in order to
be a peer trainer.

187 | P a g e
S11: Panel - Quality Measurement: Authoring and Implementing with Clinical Quality Language and
FHIR®

--Ben Hamlin, Anne Smith, Bryn Rhodes, Carol Macumber

S11-1: Clinical Quality Language (CQL) is an HL7 standard authoring language that:

a. Defines relationships between patients and clinical concepts

b. Uses data from electronic health records or health information technology systems to measure health
care quality
c. Allows logic to be shared with both quality measures and decision support
d. Contains quality data for one patient

Answer: c. Allows logic to be shared with both quality measures and decision support
Explanation: CQL is an expression standard that is used for both clinical decision support applications
and quality measures and allows logic to be shared with both quality measures and decision support.

References: https://ecqi.healthit.gov/cql

S11-2: Data elements specified for clinical quality measures reference the Quality Data Model (QDM)
which defines:
a. The source of the acquired data
b. The relationships between patients and clinical concepts in a standardized format
c. The codes that define each data element concept
d. SNOMED CT categories

Answer: b. The relationships between patients and clinical concepts in a standardized format

Explanation: The QDM is a reference model for the data elements in a clinical quality measure that
defines the relationship between patients and clinical concepts in a standardized format.

References: https://ecqi.healthit.gov/qdm

188 | P a g e
S11-3: Clinical quality measures rely on standard terminologies for accuracy and reliability of the process
being measured. For patient reported measures and clinical observations, the preferred terminology is:

a. CPT

b. SNOMED CT

c. LOINC

d. ICD

e. HCPCS

Answer: c. LOINC

Explanation: LOINC is the standard for identifying health measurements, observations, and documents.

References: https://loinc.org/

S11-4: FHIR® (Fast Health Interoperability Resources) is a standard for exchanging healthcare
information electronically. FHIR® is useful for measurement because:

a. FHIR® is a relatively simple exchange model that maintains data integrity

b. FHIR® can be used as a standalone standard or with other existing standards

c. It is organized into modules.

d. It has patient resources.

Answer: a. FHIR® is a relatively simple exchange model that maintains data integrity.

Explanation: While all of the answers are true for FHIR®, the most useful aspect of FHIR® is that it is a
simple model for exchanging quality measurement information that ensures consistency of the data
being exchanged.

Reference: FHIR Overview. https://www.hl7.org/fhir/overview.html

189 | P a g e
S12: Presentations - mHealth Solutions for Screening, Communication, and Disease Management

Digital Drawing Platform for Early Dementia Screening

--Kelvin Tsoi

S12-1: Assuming that digitization of cognitive screening questionnaires is possible, which of the
following is the main benefit of digitized screening tests that can supply the dramatically expanding
elderly population?

A. Enhancement of screening accuracy

B. Easy collection of digital data of participants
C. Enabling self-administered assessment
D. Prevention of subjective judgments on the drawing images

Answer: C. Enabling self-administered assessment

Explanation: A is incorrect, as we do not have a comprehensive digital screening platform for other
cognitive functions, including memory tests. The descriptions on B and D are true, but not the main
purpose of using digital screening.

See reference on the presentation.

S12-2: Which of the following best describes the difference between scoring methods for drawing tests
in paper-and-pencil form and scoring based on a digital drawing test?

A. Scoring methods for drawing tests in paper-and-pencil form are well-studied, but scoring based on
digital drawing test is not.

B. Scoring based on a digital drawing test considers participants’ drawing behaviors, but scoring
methods for drawing tests in paper-and-pencil form can only consider the final drawing image.

C. Scoring based on digital drawing test embraces advanced computing techniques is more
comprehensive than the traditional screening tests in paper-and-pencil form.

D. Scoring methods for drawing tests in paper-and-pencil form are more objective than the scoring
method on digital drawing test.

Answer: B. Scoring based on a digital drawing test considers participants’ drawing behaviors, but scoring
methods for drawing tests in paper-and-pencil form can only consider the final drawing image.

Explanation: The key advantage of digital drawing test is the capability of recording the whole drawing
process in human-unreachable precision. On the other hand, it is impossible to capture the intermediate

190 | P a g e
steps of the drawing tests in paper-and-pencil tests with a comparable precision. The description in A is
incorrect, as there are also other studies on digital clock drawing test. The description in C is vague since
the traditional screening tests also include memory tests and other cognitive function, so they are more
comprehensive than the digital drawing test. The description in D is reversed. Scoring method on digital
drawing methods are objective that human scoring is not involved.

Reference: Souillard-Mandar W, Davis R, Rudin C, Au R, Libon DJ, Swenson R, Price CC, Lamar M, Penney
DL. Learning Classification Models of Cognitive Conditions from Subtle Behaviors in the Digital Clock
Drawing Test. Mach Learn. 2016;102(3):393-441.

Increasing Efficiency and Enabling Innovation by Implementing Secure Text Messaging Application for
Communication and Coordinating Patient Care

--Neha Patel

S12-3: Which of the following communication tools are Health Insurance Portability and Accountability
Act [HIPAA]-complaint?

a. Pagers

b. Hospital-issued Smartphones for text

c. Hospital-issued Smartphone for voice calls

d. Hospital-issued Tablets

Answer: c. Hospital-issued Smartphone for voice calls

Explanation: The HIPAA privacy rule provides an individual with the right to access and amend protected
health information (PHI) about the individual that is maintained in a designated record set. Accordingly,
if text messages are used to make decisions about patient care, then they may be subject to the rights of
access and amendment. Text messaging merely represents a set of risks that, like other communication
technologies, needs to be managed appropriately to ensure both privacy and security of the information
exchanged. Text messages may reside on a mobile device indefinitely, where the information can be
exposed to unauthorized third parties due to theft, loss, or recycling of the device. Text messages often
can be accessed without any level of authentication, meaning that anyone who has access to the mobile
phone may have access to all text messages on the device without the need to enter a password. Texts
also are generally not subject to central monitoring by the IT department. Although text messages
communicated wirelessly are usually encrypted by the carrier, interception and decryption of such
messages can be done with inexpensive equipment and freely available software (although a substantial
level of sophistication is needed).

191 | P a g e
Reference: HIPAA Compliance for Clinician Texting. Adam Greene;
http://library.ahima.org/doc?oid=105342#.Wqk1PMuWyUk

S12-4: CMS recently announced that texting patient orders would be allowed but if a hospital ensured
that the process is through a secure text messaging platform and complies with which of the following:

a. Secure sign-on process(s)

b. Encrypted messaging

c. Allowed use of text messaging (for example, short message service (SMS) text messaging)

d. Customized message retention time frames

Answer: c. Allowed use of text messaging (for example, short message service (SMS) text messaging)

Explanation: On December 28th, 2017, the Center for Clinical Standards and Quality/Survey &
Certification Group within the Department of Health and Human Services issued a memorandum that
reaffirmed CMS’s stance on texting patient information, specifying that messages sent among clinicians
are permissible so long as healthcare teams use a secure platform. Texting patient orders, however, is
still prohibited across all platforms.

Reference: https://www.cms.gov/Medicare/Provider-Enrollment-and-
Certification/SurveyCertificationGenInfo/Downloads/Survey-and-Cert-Letter-18-10.pdf

Practical Apps: A Framework to Assess the Quality of Mobile Health Apps for Chronic Disease
Management

--Xinjie Yin

S12-5: Which of the following is one of the dimensions of the evaluation framework for mobile health
apps for chronic disease management?
A. Accessibility
B. Cost-benefit analysis
C. Sustainability
D. Patient networking

Answer: A. Accessibility
Explanation: Accessibility is part of the evaluation framework.
References:
Monkman H, Griffith J, Kushniruk AW. Evidence-based heuristics for evaluating demands on ehealth
literacy and usability in a mobile consumer health application. InMedInfo 2015 Aug 12 (pp. 358-362).
192 | P a g e
Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new
tool for assessing the quality of health mobile apps. JMIR mHealth and uHealth. 2015 Jan;3(1).

S12-6: Which poses the greatest challenge for evaluation development of patient-facing mobile
applications?
A. Prediction of user commitment
B. Extreme expense involved
C. Rapid change in mobile platforms
D. Medical society endorsement

Answer. A. Prediction of user commitment

Explanation: Current estimates are that patients abandon up to 80% of apps after two weeks of use.
Predicting user commitment is difficult when developing evaluation criteria of patient-facing mobile
apps. Mobile health app evaluation framework development does not necessarily need to be a highly
expensive endeavor. Change in mobile platforms is not rapid. Medical society endorsement has not
been found to be an important evaluation criterion of a patient-facing mobile app.
References: practicalapps.ca
Baldwin JL, Singh H, Sittig DF, Giardina TD. Patient portals and health apps: Pitfalls, promises, and what
one might learn from the other. Healthcare. Volume 5, Issue 3, September 2017, Pages 81-85.

S13: Presentations - Healthcare Big Data Analytics

Governing Predictive and Prescriptive Models: Adding the Right People and Processes to Technology

--Farah Ahmed

S13-1: The use of predictive models in healthcare organizations has become more common in the last
decade. According to Becker’s Hospital Review, one of the key takeaways from the 2018 Healthcare
Information Management Systems Society Conference (HIMSS) was that predictive analytics seems to be
driving both operational improvements and clinical decisions (1). However, with the rise of these
modeling methods, new challenges have arisen with interoperability and transparency within healthcare
systems. While it is clear that predictive analytics has become significant in healthcare, what is the
biggest challenge in the deployment of such models?

a. Predictive modeling is not effective when used for small healthcare organizations due to the
complexity of the algorithm(s)

b. Predictive model implementation leads to complete replacement of current clinician workflows

c. Lack of oversight in implementation and minimal stakeholder involvement hinders acceptance and
use

d. Minimal validation techniques and guidelines are available to ensure scientific validity of the models
193 | P a g e
Answer: c. Lack of oversight in implementation and minimal stakeholder involvement hinders
acceptance and use

Explanation: The best response to this question is “c” since it identifies the practical challenges when
implementing predictive models in healthcare organizations. When implementing a model for the first
time in clinical practice, oversight, stakeholder support, and consensus of model development based on
data/assumptions must occur for it to be successful.Stakeholder engagement is vital for the
implementation of a predictive model to ensure the appropriate staff has full knowledge of the model
deployment and its outcomes (3). The model would not replace the clinical workflow, as indicated in
“b”, but rather be integrated into current processes without being a hindrance. Obtaining oversight
requires resources and attaining stakeholder support can be time-consuming. Furthermore, the belief
that predictive models are more effective in large healthcare organizations is a myth, as indicated in
answer “a” (2). The effectiveness of a predictive model is based on the volume of historical data
collected by a hospital, not the size of the institution. Answer “d” is also incorrect since validation and
guidelines for predictive models have been published in literature and advised by different analytical
bodies (4).

References:

1. Michelson D. Becker’s Hospital Review: The Health IT & CIO Report [Internet]. Las Vegas: Dan
Michelson. 2018 March -[cited 2018 Mar 25]. Available from:
https://www.beckershospitalreview.com/healthcare-information-technology/the-no-1-takeaway-from-
himss-2018-amazon-and-apple-and-uber-oh-my.html

2. Amarasingham R, Patzer, RE, Huesch M, Nguyen Q, Xie B, Implementing Electronic Health Care
Predictive Analytics: Considerations and Challenges. Health Affairs. 2014; 33(7):1148-1154

3. Bradley P. Becker’s Hospital Review: Health Information Technology [Internet]. ZirMed: Paul Bradley.
2018 January -[cited 2018 Mar 25]. Available from: https://www.beckershospitalreview.com/healthcare-
information-technology/4-misconceptions-about-predictive-modeling-in-healthcare.html

4. Collins GS, Reitsma, JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction
model for individual prognosis or diagnosis (TRIPOD): The TRIPOD Statement. 2015; 162(1):55-64

S13-2: The term “black-box medicine” is one that describes a shortcut to answering complex questions
without fully understanding the problems or solutions (1). This shortcut most often occurs to prevent
the dissemination of proprietary algorithms in predictive models. With more hospitals utilizing
internally and externally developed predictive models that follow “black box medicine,” what guidelines
should end-users follow when using these types of prediction models?

a. End-users should not endeavor to understand the algorithm due to clinical liability and patient privacy
issues

b. End-users should override or appeal recommendations made by predictive models when they have
sound reason

194 | P a g e
c. End-users should adapt and accept all outcomes made by predictive models that are scientifically
valid.

d. End-users do not need to be educated on the possible outcomes presented by the predictive model
since they are the subject matter experts.

Answer: b. End-users should override or appeal recommendations made by predictive models when
they have sound reason

Explanation: The best answer is “b” since end-users who are often physicians should feel comfortable in
overriding decisions recommended by the model. No model is perfect. Even the best models make
inaccurate predictions due to data entry errors, never seen before data patterns, or missing important
information not captured in the data (3). Response “a” negates the important concept that
comprehension of a model can only come with transparency(2). So although patient data are kept
hidden in the “black box” end-users like physicians will need to understand the model to make informed
decisions on patient care. Response “c” is untrue since it asks end-users to blindly follow predictive
model recommendations, which is unreasonable. While these models are formulated with immense
amount of patient data, the number of use cases to test validity are often minimal (4). HealthCatalyst
warns healthcare organizations from buying into the “hype” and vendors that oversell their product (4).
Response “d” is also incorrect as it is important to equip physicians with the proper training to
understand the predictive model so that they may act appropriate when they are faced with incorrect
solutions, assumptions, and other issues (4).

References:

1. Ford RA, Price II WN, Privacy and Accountability in Black-Box Medicine. Michigan Telecommunications
and Technology Law Review. 2016; 23(1):1-43

2. PR Newswire [Internet]. Tampa: PR Newswire Association, LLC; c2018 [cited 2018 Mar 20]. Providers
to Adopt IT Outsourcing Solutions in 2016 as More Hospitals and Physician Practices Slide Deeper Into
Financial Uncertainty, Black Book Survey; [1 screen]. Available from:
https://www.prnewswire.com/news-releases/providers-to-adopt-it-outsourcing-solutions-in-2016-as-
more-hospitals-and-physician-practices-slide-deeper-into-financial-uncertainty-black-book-survey-
300183692.html

3. Cohen, IG, Amarasingham R, Shah A, Xie B, Lo B. The Legal and Ethical Concerns that Arise from Using
Complex Predictive Analytics in Healthcare. Health Affairs. 2014; 33(7):1139-1147

4. Amarasingham R, Patzer, RE, Huesch M, Nguyen Q, Xie B, Implementing Electronic Health Care
Predictive Analytics: Considerations and Challenges. Health Affairs. 2014; 33(7):1148-1154

195 | P a g e
Using a Validated Predictive Analytics Model for Early Detection of Patients with Sepsis

--Gordon Moyer

S13-3: You've created a predictive analytics model for sepsis. In your validation of the model, you
calculate a C-statistic of 0.5. How do you interpret this result?

A. The model is no better at predicting which patients will and will not develop sepsis, compared with
chance alone.
B. The model is good at predicting which patients will and will not develop sepsis.
C. The model is strong at predicting which patients will and will not develop sepsis.
D. The model always predicts which patients will and will not develop sepsis.
E. Based on this predictive model, 50% of patients will develop sepsis and 50% will not.

Answer: A. The model is no better at predicting which patients will and will not develop sepsis,
compared with chance alone.

Explanation: C-statistic is a measure of "goodness of fit" for a logistic regression model. It measures the
predictive accuracy of such models.
C-statistic is measured on a scale from 0 to 1. A score of 0.5 means that the model is no better at
predicting an outcome, compared to random chance. A score of 1 means that the model perfectly
predicts patients who will attain a certain outcome from those who will not.
A C-statistic of 0.7 generally indicates a good predictive model. A C-statistic greater than 0.8 generally
indicates a strong predictive model.
(A C-statistic less than 0.5 indicates a very poor predictive model, worse than chance)
Reference: http://www.statisticshowto.com/c-statistic/

S13-4: You've created a predictive analytics model for sepsis. In your validation of the model, you
calculate a C-statistic of 0.85. How do you interpret this result?

A. The model is no better at predicting which patients will and will not develop sepsis, compared with
chance alone.
B. The model is good at predicting which patients will and will not develop sepsis.
C. The model is strong at predicting which patients will and will not develop sepsis.
D. The model always predicts which patients will and will not develop sepsis.
E. Based on this predictive model, 85% of patients will develop sepsis.

196 | P a g e
Answer: C. The model is strong at predicting which patients will and will not develop sepsis..

Explanation: C-statistic is a measure of "goodness of fit" for a logistic regression model. It measures the
predictive accuracy of such models.
C-statistic is measured on a scale from 0 to 1. A score of 0.5 means that the model is no better at
predicting an outcome, compared to random chance. A score of 1 means that the model perfectly
predicts patients who will attain a certain outcome from those who will not.
A C-statistic of 0.7 generally indicates a good predictive model. A C-statistic greater than 0.8 generally
indicates a strong predictive model.
(A C-statistic less than 0.5 indicates a very poor predictive model, worse than chance)
References: http://www.statisticshowto.com/c-statistic/

Progression of a Quality Scorecard Leveraging Informatics Tools

--Mark Connolly

S13-5: Which of the following is a key quality reporting best practice?

A. Excessive consultation with subject-matter experts

B. Develop a standard validation process to ensure quality and consistency

C. Provide access to underlying data to all report users to promote transparency

D. Do not release measure definitions to minimize the risk of the Hawthorne effect

Answer: B. Develop a standard validation process to ensure quality and consistency

Explanation: Consultation with subject-matter experts, having a standard validation process, and
releasing measure definitions are best practices that provide trust in the quality of data reported.
Considerations of HIPAA and patient privacy should be made prior to providing complete access.
Providing access to all report users when reporting on healthcare quality and other related data
introduces risk.

References: Denney MJ, Long DM, Armistead MG , et al. Validating the extract, transform, load process
used to populate a large clinical research database. Int J Med Inform. 2016 Oct;94:271-4.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5556907/pdf/nihms808944.pdf

https://hbr.org/2017/03/when-clinicians-know-theyre-being-watched-patients-fare-better

https://www.hhs.gov/hipaa/for-professionals/privacy/guidance/minimum-necessary-
requirement/index.html

197 | P a g e
S13-6: What is the role of subject-matter experts during metric development?

A. Oversee data capture and availability

B. Ensure value of metric to warrant resources for development

B. Taking all validation responsibilities

C. Develop plan that aligns with organizational goals

D. Secure necessary resources for developing metrics

Answer: B. Ensure value of metric to warrant resources for development

Explanation: One key role of subject-matter experts (SMEs) is to evaluate the value of metrics and
whether resources should be devoted to their developemnt. SMEs should not be used or relied on for
complete validation purposes but can be used in addition to other validation processes. They also
ensure that the resources required to develop a metric will return value to the organization, that there
will be utilization of the metric to drive improvement efforts and can act as a resource to navigate
nuances during development. Data custodians oversee data capture and availability, and business
sponsors are responsible for developing the plan and securing the necessary resources.

References: Denney MJ, Long DM, Armistead MG , et al. Validating the extract, transform, load process
used to populate a large clinical research database. Int J Med Inform. 2016 Oct;94:271-4.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5556907/pdf/nihms808944.pdf

https://hbr.org/2017/03/when-clinicians-know-theyre-being-watched-patients-fare-better

https://www.hhs.gov/hipaa/for-professionals/privacy/guidance/minimum-necessary-
requirement/index.html

S14: Presentations - Infrastructure and Frameworks for Successful Implementations

Application of an Evidence-Based Adoption Framework: Practical Implementation Tools for Clinical

Operations

--Blake Lesselroth

S14-1: You have been asked by your leadership to improve and standardize medication reconciliation in
an ambulatory clinic. In response, your team has developed electronic health record features to
improve medication history collection, documentation of reconciliation, and patient counseling. When
rolling out the new technology, which of the following approaches is most important about improving
staff adoption?

198 | P a g e
A. A robust message campaign (e.g., educational materials, posters) is a critical ingredient in an
implementation bundle.

B. Software should include patient-facing affordances that improve patient reliability including
medication images, plain language, and simple response controls.

C. The implementation team should pay attention to the socio-technical fit of the product, including
workflow compatibility and implementation climate.

D. Executive sponsorship and clinical champions are necessary and sufficient for promoting adoption in
large practices, managed care organizations, and public systems.

Answer: C. The implementation team should pay attention to the socio-technical fit of the product,
including workflow compatibility and implementation climate.

Explanation: The present work and background literature suggest workflow and organizational climate
should be addressed during an implementation to improve technology adoption. Although educational
materials can be helpful, they are relatively perishable and have a less durable effect on behavior. Other
strategies, including at-the-elbow support, tend to have a greater impact. This work did not show that
patient-facing technologies either improved the quality of data or influenced staff adoption. Certainly,
executive sponsorship and clinical champions are important to an initiative. However, sponsorship
alone is rarely sufficient to ensure adoption. This work showed that a practice facilitation strategy can
address lack of leadership messaging.

Reference: Holahan PJ, Lesselroth BJ, Adams K, Wang K, Church V. Beyond technology acceptance to
effective technology use: a parsimonious and actionable model. Journal of the American Medical
Informatics Association. 2015;22(3):718-729.

S14-2: You have recently implemented new electronic health record features designed to improve the
reliability of medication reconciliation in clinic. Nurses are expected to use new software to collect and
document patient history. Physicians are expected to respond to an automated reminder confirming
medications have been reconciled. Upon review of the monthly performance dashboards reports, you
realize that nurse charting has improved but physician documentation remains unchanged. What would
be the most appropriate next step to improve physician documentation?

A. Add a new hard-stop or modal alert that requires physicians to complete documentation before
proceeding.
B. Gather observational or time-motion data to better understand physician workflow and identify
potential barriers to reminder completion.
C. Consolidate the history collection and reconciliation tasks so that nurses can complete all
documentation concurrently.
D. Meet with department leadership and script out physician performance plans that include
reconciliation as a measure of effectiveness.

199 | P a g e
Answer: B. Gather observational or time-motion data to better understand physician workflow and
identify potential barriers to reminder completion.

Explanation: It is critical to understand clinician workflow and health record use strategies to
understand barriers to adoption and use. Clinics are complex adaptive systems characterized by non-
linear workflows, process interruptions, and unanticipated or emergent properties. It is plausible that
the interface affordances are either hidden or inaccessible to clinicians at the point-of-care. Adding a
hard-stop – particularly if the affordances are problematic – could create an unmanageable bottleneck
in throughput. Delegating the tasks and documentation to nurses is not an effective workload leveling
strategy and may generate resentment. Aligning quality improvement objectives with clinician
performance can be an effective change management strategy, but not before attempting to
understand the barriers to technology use.

Reference: Holden RJ, Karsh BT. The Technology Acceptance Model: its past and its future in health care.
Journal of Biomedical Informatics. 2010;43(1):159-172.

Creating the Data Infrastructure to Support Mobile Health and Telehealth Initiatives

--Christopher Hollweg

S14-3: Which of the following best describes an error induced by automation bias?
A) Dr. Smith is alerted by an automatic EHR message that his patient may be developing severe sepsis,
along with a button to order intravenous (IV) fluids. However, this patient also has heart failure,
which may worsen with IV fluids. Dr. Smith assumes that the alert is appropriate and orders the
fluids. The patient later develops fluid overload and is transferred to the intensive care unit.
B) Dr. Smith is about to order antibiotics for his patient with a urinary tract infection. He chooses
ciprofloxacin because that is what he is used to using for his patients, but did not notice that this
patient's previous urine culture grew organisms resistant to ciprofloxacin
C) Dr. Smith needs to discontinue the medication metoprolol for a patient in the EHR. However, he
mistakenly discontinues the medication "metoclopramide," which was located right next to metoprolol
in the EHR and looked very similar.
D) Dr. Smith is about to admit his 10th patient on a busy call day. When placing his admission orders, he
mistakenly signs them in an open chart in the EHR for another patient whom he had seen earlier.

Answer: The correct answer is A.

Explanation: Automation bias refers to the propensity for humans to be over-reliant on suggestions
from automated decision support systems and ignore contradictory evidence. In this case, the patient,
who has severe congestive heart failure, should not have been given 30cc/kg of fluid, but Dr. Smith
relied on the automated decision support recommendation and gave it anyway, which resulted in an
adverse event.

200 | P a g e
Incorrect answers:
B) This error is an example of the effects of availability heuristic.
C) This error resulted from a difficult to read EHR interface.
D) This error is an example of a wrong patient order that can occur when multiple patient charts are
open in the EHR.

References: Reference: Goddard, K., Roudsari, A. & Wyatt, J. C. Automation bias: a systematic review of
frequency, effect mediators, and mitigators. J. Am. Med. Informatics Assoc. 19, 121–127 (2012).

S14-4: Which of the following is one of the largest challenges commonly addressed in clinical risk
prediction model studies involving EHR data?
A. Evaluation metrics
B. Informative presence
C. Lead time bias
D. Missing data

Answer: D. Missing data

Explanation: One of largest challenges commonly addressed in EHR based studies is missing data.
Informative presence, which many studies don’t address, is based off the principle that EHR data is
made up of sicker people. Evaluation metrics are useful for comparing risk prediction models across
different diseases, i.e. c-statistic. Lead time bias refers to the length of time between the detection of a
disease and its clinical presentation and diagnosis.

References: Benjamin A Goldstein, Ann Marie Navar, Michael J Pencina, John P A Ioannidis;
Opportunities and challenges in developing risk prediction models with electronic health records data: a
systematic review. J Am Med Inform Assoc 2017; 24 (1): 198-208. doi: 10.1093/jamia/ocw042

S14-5: Which of the following is a recommended goal when developing and implementing an electronic
decision support tool for medication reconciliation?
A. Anticipating the users’ needs
B. Conducting usability testing in different settings
C. Focusing on thoroughness rather than efficiency
D. Redesigning the existing clinical workflow to match the tool

Answer: B. Conducting usability testing

Explanation: Recommendations concerning the implementation of electronic tools to support

medication reconciliation (eMedRec) include conducting usability testing and testing the software
201 | P a g e
(reliability + interoperability) in selected and disparate settings, focusing on efficiency in addition to
clinical improvements, and adapting the tool workflow to users’ habits. It is recommended to not
anticipate the users’ needs during the development of the electronic tools to support medication
reconciliation.

References: Sophie Marien, Bruno Krug, Anne Spinewine; Electronic tools to support medication
reconciliation: a systematic review. J Am Med Inform Assoc 2017; 24 (1): 227-240. doi:
10.1093/jamia/ocw068

Use Cases for a WiFi Connected Internet of Things Button in an Academic Medical Center

--Haipeng Zhang

S14-6: The BWH Dash button is an example of using Internet of Things (IoT) devices. IoT is best defined
as:

A. A group of related ontologies within a larger database housed on a public facing network.

B. A network of physical objects that contain embedded technology to communicate and sense or
interact with their internal states or the external environment.

C. Wearable technologies that have the capability to sense biometric data from the user and
communicate with one another and across a larger network.

D. Miniaturized single function computers that communicate via physical connections to one another.

E. Low cost, single board computers that can be programmed to perform various functions.

Answer: B. A network of physical objects that contain embedded technology to communicate and
sense or interact with their internal states or the external environment.

Explanation: Answer B best encapsulates the broad definition of IoT devices. Choices C and E are
subsegments of IoT technology but not the best definition. Choices A and D are incorrect.

Reference: Gartner Internet of Things, defined: https://www.gartner.com/it-glossary/internet-of-

things/

202 | P a g e
S14-7: The BWH Dash button uses AWS IoT 1-Click technology with a custom dashboard that allows for
easy provisioning and programming of BWH Dash buttons. How many functions can BWH Dash buttons
be programmed for?

A. 1 function - single press

B. 2 functions - single press, long press
C. 3 functions - single press, double press, long press
D. 4 functions - single press, double press, triple press, long press
E. 5 functions - single press, double press, triple press, long press, long press and hold

Answer: C. 3 functions - single press, double press, long press

Explanation: The AWS IoT Button technology can be programmed to react to three different triggers: a
single press, a double press, and a long press.

Reference: https://aws.amazon.com/iotbutton/

S15: Panel - New Approaches for Improving CPOE Safety: Indications-based Prescribing and CancelRx

--Gordon Schiff, Neri Pamela, Christine Cheng, Samantha Pitts

S15-1: Physicians have long been urged to place the indication for the medication on the prescription for
drugs they are ordering. With paper prescriptions, physician acceptance has been poor, but with near
universal adoption of electronic ordering (CPOE) there are renewed opportunities to promote utilization
of this feature. Integrating indications into prescribing could pave the way for a safer, more complete
continuum of care for the patient and save time during prescribing and related tasks. Which of the
following is another significant benefit of indications-based prescribing?

A. The indication will always be put on the medication label

B. Drug costs will decrease when indications are included

C. Patients will know the reason why they are taking each of their medications

D. Brand name drugs will be selected more often

Answer: C. Patients will know the reason why they are taking each of their medications

Explanation: One of the goals of an indications-based prescribing system is to increase patient

adherence and education. With an indications-driven process, patients will be able to discuss their
indications with their providers and have the indication included on the medication label if they choose.
Prescribing by indication will allow patients to experience a safer continuum of care as different

203 | P a g e
members of their care team will understand the reason behind each of their medications and it will
allow patient to better discuss their medications.

While including indications on the medication label is important, patients should also have the option to
opt-out of including the indication if they would prefer the reason kept private, such as in cases of
mental health or STD medications. The goal would not be for a universal inclusion of indications on the
label, but instead for a system where indications would be standard with the option for patients to
suppress the indication so they have flexibility (A). However, by redesigning the workflow to include
indications and making it trivial to include indications on labels, most patients would be able to
reference their medications to understand why they are taking each one.

Finally, indications-based prescribing is designed around the idea of safer and more efficient prescribing,
not around drug costs and brands. The goal is that a drug would be selected because it treats the
indication while incorporating patient information (like allergies), guidelines, insurance and other
factors. While insurance and formulary issues are part of the process, the goal is not focused on
lowering drug costs or choosing the cheapest drug (B). In addition, the system should not have
preference for name brands, but instead choose based on whatever drug is the most appropriate for the
patient (D).

Reference: Schiff GD, Seoane-Vazquez E, Wright A. Incorporating indications into medication ordering —
time to enter the age of reason. N Engl J Med. 2016 Jul28;375(4):306–9.

S15-2: While including the indication on the prescription is not a new idea, many providers do not use
this feature on a consistent basis. It has not become standard practice partially due to poor CPOE design.
To successfully incorporate indication into the prescribing workflow, efforts need to focus on reducing
mouse clicks, providing help at the point of care, and minimizing alert fatigue while supporting
physicians’ autonomy and flexibility. Which of the following design choice best explains how a prototype
successfully incorporates indications by allowing for physician autonomy while providing useful
guidance?

A. Allow providers to search for a drug only and do not require an indication

B. Allow providers to start with a drug or a problem/indication and see drug suggestions

C. Allow providers to search for a drug only, enter an indication and see alternative drug suggestions

D. Allow providers to search for a problem only and see drug suggestions

Answer: B. Allow providers to start with a drug or a problem/indication and see drug suggestions

Explanation: For recommendations to be most useful, the indication should be required first which
allows the drug choices to be tailored to the patient and indication chosen. However, given that many
physicians order certain drugs very frequently, allowing the physician to also search for a drug provides
them the clinical autonomy and flexibility they might desire. By allowing physicians to only start with a
drug and not a problem, we are missing an opportunity to increase provider efficiency and minimize

204 | P a g e
cognitive load during their clinical workflow. If we allow providers to only start with a problem, we have
limited the systems flexibility of use and removed some of their clinical autonomy.

Reference: Improving HIT Prescribing Safety by Incorporating Indications Into Computerized Prescriber
Order Entry. Content last reviewed March 2017. Agency for Healthcare Research and Quality, Rockville,
MD. http://www.ahrq.gov/chain/research-tools/featured-certs/improving-hit-prescribing-safety.html

S15-3: Associating indications with a drug order may help prevent medication errors associated with
look-alike-sound-alike (LASA) drugs. Which of the following best describes the utility of associating high-
level indications concepts with drug pairs on the Institute for Safe Medication Practices’ list of
commonly confused drug names?

a. The majority of drug pairs that were comprised of antineoplastic drugs could be differentiated using
high-level indication concepts.

b. The majority of drug pairs that were comprised of antibacterial drugs could be differentiated using
high-level indication concepts.

c. More than half of all the evaluated drug pairs could be differentiated using high-level indication
concepts.

d. Only 20% of all evaluated drug pairs could be differentiated using high-level indication concepts.

Answer: c. More than half of all the evaluated drug pairs could be differentiated using high-level
indication concepts.

Explanation: The presented study used high-level indications concepts from a proprietary knowledge
base and associated them with drugs on the Institute for Safe Medication Practices’ list of commonly
confused drugs.Of the evaluated drug pairs, 60% had no overlap in high-level indications. Twenty
percent of the evaluated drug pairs had a complete overlap in high-level indications, and most of these
were drug pairs that were comprised of antineoplastic drugs, anti-infective drugs, anti-diabetic drugs or
opioid analgesics.

Reference: Seone-Vazquez E, Rodriguez-Monguio R, Alqahtani S, Schiff G. Exploring the potential for

using drug indications to prevent look-alike and sound-alike drug errors. Expert Opinion on Drug Safety.
2017;16:10:1103-1009.

S15-4: The National Council for Prescription Drug Programs’ SCRIPT standard supports electronic
functionality for sending prescription cancellations from electronic health records (EHRs) to pharmacies,
known as CancelRx. However, adoption of this functionality has been limited. Which of the following
presented the greatest challenge to pilot, rather than enterprise-wide,implementation?

A. Impact on pharmacy workflow

B. Impact on prescriber workflow
C. Restricted functionality of prescription cancellation in pilot implementation
D. Buy-in from key stakeholders
205 | P a g e
Answer: C. Restricted functionality of prescription cancellation in pilot implementation

Explanation: Pilot implementation at Johns Hopkins was planned to systematically understand the
expected performance of the system, the implications for workflows of key stakeholders, and the impact
on existing medication safety risks and risks introduced by implementation. However, configuration in
the Johns Hopkins EHR for use by a cohort of pilot users restricts the functionality of prescription
cancellation, as only prescriptions written by pilot users can be electronically cancelled.

Reference: Fischer S, Rose A. Responsible e-Prescribing Needs e-Discontinuation. JAMA.

2017;317(5):469–470. doi:10.1001/jama.2016.19908

S16: Presentations - Care Coordination and Communication

Improving care coordination for children with autism spectrum disorder: defining care team members
information needs

--Julia O’Rourke

S16-1: Autism spectrum disorder (ASD) is a neurodevelopmental disorder diagnosed in childhood that
affects individuals throughout their lifespan. (1). As many as one in 50 children in the U.S. between the
ages of six and 17 years have a diagnosis of ASD (2), which reflects a 200-fold increase compared to the
estimates of four cases per 10,000 in 1966 (3). Individuals with ASD experience lifelong difficulties in
social interaction, communication skills, behavior, and learning.

Care for children with ASD is complex, involving on average seven different types of interventions,
programs, and therapies (4), including state organizations, local schools, medical providers, in-home
behavioral therapy, and rehabilitation services, such as physical and occupational therapies. Children
with ASD also have increased medical needs due to comorbid mental and medical conditions including
sleep problems, seizures, gastrointestinal problems, and maladaptive behaviors (5, 6). Current
treatment delivery model for children with ASD is fragmented with treatment team members delivering
care at different entities without any systematic collaboration or communication.

To better demonstrate the roles or different care team members working with a child with ASD, we
describe treatment of a hypothetical school age boy John, who attends a public elementary school. In
school John has several care team members assisting him. His teacher and school aid are working with
him daily in the classroom; his occupational therapist focuses on sensory difficulties during a weekly 45-
minute session. His physical therapist helps with building strength and endurance during another weekly
45-minute session. His speech therapist sees him three times per week and helps with articulation
difficulties. Finally, the school’s behavioral therapist sees John two times per week to teach pragmatic
and social skills. At school, John has an Individualized Educational Plan (IEP) plan.

John’s parents also bring him to a private speech therapist and occupational therapist for weekly
sessions. In addition, John sees a psychologist for weekly sessions outside of school, and another
behavioral therapist who helps with some of the troubling behaviors at home. John is also followed by a
psychiatrist, who monitors his psychotropic medications. John sees a neurologist for sleep and seizure
206 | P a g e
problems and gastroenterologist for constipation. In this example, all care team members work with
John to help him and his family to achieve a better quality of life and help him learn skills to live as an
independent and productive adult. John has caregivers at school, at home, in private offices, and in
several medical facilities, but they work independently and do not communicate across their practices
and locations.

These treatments are fragmented and it is not clear whether the therapies being delivered are the
optimal choice for John and his family.

What is the most likely care coordination scenario?

A. John has a Medical Home at his pediatrician’s office, where a case manager (social worker or nurse)
coordinates his medical care and services with other care team members
B. John has an Integrated Medical home at his psychiatrist’s office, where a case manager coordinates
his mental health care and services with other care team members
C. John has no care coordination activities
D. John’s mother performs all care coordination activities. To do this, she has had to reduce her hours
and now works part-time.

Answer: D. John’s mother performs all care coordination activities. To do this, she has had to reduce
her hours and now works part-time.

Explanation: As described in the example of John’s fragmented care, in the current care delivery model
for ASD, there is no central person with expertise in ASD who could provide oversight of John’s care and
tailor the most appropriate treatment strategy for him. A half century ago, the American Academy of
Pediatrics developed a medical home care model (7), designed to connect children with special
healthcare needs and their families with appropriate resources to improve communication and
coordination between care providers, schools and the community for pediatric patients (8).

Although medical homes improve many aspects of pediatric clinical care (8), only 27% of children with
ASD have a medical home (9). This model does not fit care needs for children with ASD, because
pediatricians lack expertise in ASD and the time to coordinate care. Recent research shows that
pediatricians lack knowledge about current interventions and community resources for ASD (10).
Furthermore, pediatricians report a need for expert guidance in care provision for children with ASD. “I
feel like I don’t know if they’re doing the right thing because they’re all so different, and I feel like I’m
driving a team of horses and one’s going this way and the others going this way.”

Pediatricians found case managers to be useful for care coordination, but because so few ASD patients
were in each practice, they suggested moving towards community-based or insurance based case
managers (11). We believe that Specialty / Integrated Medical Home is a better model for children with
ASD (12, 13). In our work, we interviewed 10 parents and every parent said that they perform duties of
care coordinators, and all 8 mothers interviewed either worked part-time or did not work.

This is consistent with the national statistics where mothers of children with ASD earn 56% ($14 755)
less than the mothers of children with no health limitation. They are 6% less likely to be employed and
work 7 hours less per week, on average, than mothers of children with no health limitation (14, 15)

207 | P a g e
Answer A is incorrect because only 27% of children with ASD have a medical home (9). Answer B is
incorrect because Integrated Medical Home is a more advanced model of Medical Home which
integrates mental health and primary care (16).

References: 1. Simonoff E, Pickles A, Charman T, Chandler S, Loucas T, Baird G. Psychiatric disorders in

children with autism spectrum disorders: prevalence, comorbidity, and associated factors in a
population-derived sample. J Am Acad Child Adolesc Psychiatry 2008;47:921-9.
2. Center for Disease Control. Changes in prevalence of parent-reported autism spectrum disorder in
school-aged U.S. children: 2007 to 2011–2012. 2013.
3. Lotter V. Epidimiology of autistic conditions in young children. Social Psychiatry 1966;1:124-35.
4. Green VA, Pituch KA, Itchon J, Choi A, O'Reilly M, Sigafoos J. Internet survey of treatments used by
parents of children with autism. Res Dev Disabil 2006;27:70-84.
5. Gurney JG, McPheeters ML, Davis MM. Parental report of health conditions and health care use
among children with and without autism: National Survey of Children's Health. Archives of pediatrics &
adolescent medicine 2006;160:825-30.
6. Brachlow AE, Ness KK, McPheeters ML, Gurney JG. Comparison of indicators for a primary care
medical home between children with autism or asthma and other special health care needs: National
Survey of Children's Health. Archives of pediatrics & adolescent medicine 2007;161:399-405.
7. Sia C, Tonniges TF, Osterhus E, Taba S. History of the medical home concept. Pediatrics
2004;113:1473-8.
8. Homer CJ, Klatka K, Romm D, et al. A review of the evidence for the medical home for children with
special health care needs. Pediatrics 2008;122:922-37.
9. Kogan MD, Strickland BB, Blumberg SJ, Singh GK, Perrin JM, van Dyck PC. A national profile of the
health care experiences and family impact of autism spectrum disorder among children in the United
States, 2005-2006. Pediatrics 2008;122:e1149-58.
10. Nolan R, Walker T, Hanson JL, Friedman S. Developmental behavioral pediatrician support of the
medical home for children with autism spectrum disorders. J Dev Behav Pediatr 2016;37:687-93.
11. CMS expands Medicare payment for behavioral health services. 2017. (Accessed 3/7/2017, at
https://mcdonaldhopkins.com/Insights/Alerts/2016/11/09/CMS-expands-Medicare-payment-for-
behavioral-health-services.)
12. Fueyo M, Caldwell T, Mattern SB, Zahid J, Foley T. The Health Home: A Service Delivery Model for
Autism and Intellectual Disability. Psychiatr Serv 2015;66:1135-7.
13. Asarnow JR, Rozenman M, Wiblin J, Zeltzer L. Integrated Medical-Behavioral Care Compared With
Usual Primary Care for Child and Adolescent Behavioral Health: A Meta-analysis. JAMA Pediatr
2015;169:929-37.
14. Cidav Z, Marcus SC, Mandell DS. Implications of childhood autism for parental employment and
earnings. Pediatrics 2012;129:617-23.
15. Hodgetts S, McConnell D, Zwaigenbaum L, Nicholas D. The impact of autism services on mothers'
occupational balance and participation. OTJR (Thorofare N J) 2014;34:81-92.
16. Tyler ET, Hulkower RL, Kaminski JW. Behavioral Health Integration in Pediatric Primary Care:
Considerations and Opportunities for Policymakers, Planners, and Providers: Milbank Memorial Fund;
2017.

208 | P a g e
S16-2: Care coordination integrates services across settings and supports interfacing among multiple
care systems and care providers, including the following: medical, social and behavioral professionals;
the educational system; home care agencies; allied health professionals; and families (1). In a recent
opinion piece titled “Health Information Technology and Care Coordination: The Next Big Opportunity
for Informatics?”, Dr. David Bates described the care coordination needs for complex patients who are
cared for by a group of medical providers: “This entire group needs to be able to communicate better
than they do today – to see the care plan, to message each other, and to have synchronous and
asynchronous conversations about goals, and progress and lack thereof” (2).

Dr. Halamka further highlighted the need for care coordination infrastructure in 2017 NEJM: “If we want
patients to be engaged to help reduce the burden of care coordination, care plan tracking, and
communication, we need modern tools that enable patients to interact with their providers using
devices and workflows that are already part of their daily lives. Many companies are now offering such
tools” (3).

However, care coordination tools will be useful within a collaborative care model that supports care
coordination. The collaborative care model relies on information systems and case managers to link
primary care providers, patients, and mental health specialists (4). The collaborative care model has
shown to be effective in over 80 studies for in adult population for the treatment of depressive and
anxiety disorders (4, 5). Collaborative care also improves outcomes for children and adolescents with
mental health problems (6).

The collaborative care model is based on the chronic care model developed by Wagner and colleagues
nearly two decades ago to guide health services towards better management of chronic diseases (7).
According to this model, improved patient outcomes result from the interactions between an activated
patient and a proactive healthcare team.

Factors that activate patients include support for disease self-management and enabling community
resources. Factors that activate healthcare providers include adequate information systems, decision
support, and delivery system design to support longitudinal care as opposed to episodic care.

What is the greatest barrier in implementing a collaborative care model for children with ASD?

A. Severe shortage of child psychiatrists in the United States

B. No adequate payment mechanism for supporting care coordination
C. Privacy laws (The Health Insurance Portability and Accountability Act of 1996 (HIPAA) and The Family
Educational Rights and Privacy Act(FERPA))
D. Difficulties integrating with EHR systems

Answer: B. No adequate payment mechanism for supporting care coordination

209 | P a g e
Explanation: Answer A: Although there is severe shortage of child psychiatrists in the United States (8),
it is likely that increasing number of psychiatrists will not help with implementing a collaborative care
model.
Answer B: Currently, there is no payment mechanism (using CPT codes) to support case managers.
Although, CPT codes 99487-99489 have recently been published, in our experience very few payers
support these codes and the ones that do, require medical care coordination by a nurse. Children with
ASD need care coordination across medical, educational and community providers, which is typically
provided by a social worker.

Answer C: With adequate consent forms, signed by parents and legal guardians, these laws can be
followed.

Answer D: Although difficulties integrating with EHR systems exist, a qualified case manager can
manually extract and integrate information across the care team.

References: 1. Turchi RM, Antonelli RC. Patient- and family-centered care coordination: a framework for
integrating care for children and youth across multiple systems. Pediatrics 2014;133:e1451-60.
2. Bates DW. Health Information Technology and Care Coordination: The Next Big Opportunity for
Informatics? Yearbook of Medical Informatics 2015;10:11-4.
3. Halamka JD, Tripathi M. The HITECH Era in Retrospect. N Engl J Med 2017;377:907-9.
4. Thota AB, Sipe TA, Byard GJ, et al. Collaborative care to improve the management of depressive
disorders: a community guide systematic review and meta-analysis. American journal of preventive
medicine 2012;42:525-38.
5. Archer J, Bower P, Gilbody S, et al. Collaborative care for depression and anxiety problems. Cochrane
Database Syst Rev 2012;10:CD006525.
6. Asarnow JR, Rozenman M, Wiblin J, Zeltzer L. Integrated Medical-Behavioral Care Compared With
Usual Primary Care for Child and Adolescent Behavioral Health: A Meta-analysis. JAMA Pediatr
2015;169:929-37.
7. Wagner EH. Chronic disease management: what will it take to improve care for chronic illness?
Effective clinical practice : ECP 1998;1:2-4.
8. Practicing Child and Adolescent Psychiatrists by State 2015. (Accessed 03/26/2018, at
https://www.aacap.org/aacap/Advocacy/Federal_and_State_Initiatives/Workforce_Maps/Home.aspx.)

Challenges to Care Coordination for Children with Special Health Care Needs: Potential Solutions

--Pooja Paode

S16-3: Current efforts to improve care for children with special health care needs (CSHCN) are highly
medicalized and designed through the lens of providers, despite the fact that parents are responsible for
the vast majority of the care coordination for CSHCN. In response to this issue, we created a multi-
sector, parent-centered Stakeholder Coalition dedicated to identifying solutions. The Coalition identified
communication, community collaboration, family engagement/advocacy, and systems issues/technology
as key areas for improvement for the care coordination of CSHCN.

How should we consider the interaction between systems issues/technology and the other three areas?
We should…

210 | P a g e
A. come up with solutions for the four areas separately, then evaluate how they interact.

B. prioritize the development of technological solutions, such as applications, in each of the three other
areas.

C. consider solutions that will address gaps in each of the three areas, then identify how technology
and systems redesign can enable those solutions.

D. prioritize development on technology and systems changes that simultaneously bring together all
three areas: communication, community collaboration, and family engagement.

Answer: C. consider solutions that will address gaps in each of the three areas, then identify how
technology and systems redesign can enable those solutions.

Explanation: While technology can serve to enhance solutions to care coordination for CSHCN, we
should be cautious of excessively fixating on or prioritizing technological solutions to care coordination.
It is important to first identify solutions important to children, parents, and providers, and then consider
how technology can enhance or enable those solutions to make them more effective or feasible. It
would be ineffective to come up with technological solutions or enact systems redesign independently
of the other three priority areas. Despite this, many technological solutions to health systems are
created in silos and fail to adequately include parent preferences related to communication,
collaboration, and engagement.

Reference: Turchi RM, Antonelli RC, Norwood KW, Adams RC, Brei TJ, Burke RT, Davis BE, Friedman SL,
Houtrow AJ, Kuo DZ, Levy SE. Patient-and family-centered care coordination: A framework for
integrating care for children and youth across multiple systems. Pediatrics. 2014 May 1;133(5):e1451-60.

S16-4: Parents of children with special health care needs (CSHCN) often simultaneously operate as care
coordinators, caretakers, and key decision-makers for their children. They experience a lack of
meaningful information exchange surrounding situation management, planning, monitoring, navigating,
and resource usage.

When does this problem cause the most adverse impact for parents caring for CSHCN? During…

A. provider-parent encounters
B. a variety of “tension points” throughout the child’s lifetime
C. the time that passes between clinic visits
D. searches for external resources to aid in the care of their CSHCN

Answer: B. a variety of “tension points” throughout the child’s lifetime

211 | P a g e
Explanation: Parents of CSHCN have reported feeling vulnerable during specific “tension points” in their
children’s lives. Examples of tension points include when parents are first told that their child is likely to
have significant developmental problems, when an important person in their child’s care – like a
specialist or a case manager leaves, or when a child is transferred from one service or program to
another. Parents currently do not have the information or support they need to work through these
times of tension. Consequently, parents have reported thinking “I wish someone had told me about
this” or “how come no one prepared me for this?” Solutions that promote meaningful information
exchange during these key tension points are high-impact target for systems improvement.

Reference: Ranade-Kharkar P, Weir C, Norlin C, Collins SA, Scarton LA, Baker GB, Borbolla D, Taliercio V,
Del Fiol G. Information needs of physicians, care coordinators, and families to support care coordination
of children and youth with special health care needs (CYSHCN). Journal of the American Medical
Informatics Association. 2017 Mar 24;24(5):933-41.

Accuracy of Patient Self-Report of 90-Day Post-Discharge Healthcare Utilization and Complications:

Relevance to Value Based Care

--Benjamin Rosner

S16-5: University Hospital is a regional center of excellence for total joint arthroplasty (TJA), and as such,
it draws patients from a large geographic area. However, post-discharge, most of these patients return
to their home area. When complications or readmissions occur, events for which University Hospital
bears financial risk under value-based reimbursement models, University Hospital is often unaware. In
reading the literature, those at University Hospital leading the post-discharge process improvement
efforts have learned that up to 65% of complications and readmissions may occur at other institutions.
To get a more complete picture of when and where readmissions are occurring among their patients,
the hospital is considering surveying patients directly about readmissions occurring within 90 days of
discharge. Assuming an unambiguous readmission question worded in lay language, University Hospital
might expect patients to have what degree of accuracy in self-reporting readmission events?

A. Excellent (kappa≥0.8)

B. Very good (kappa 0.60-0.79)

C. Moderate (kappa 0.40-0.59)

D. Fair (kappa 0.20-0.39)

E. Poor (kappa <0.20)

Answer: A. Excellent (kappa≥0.8)

Explanation: The study found that patients were accurate reporters of 90-day hospital readmissions,
with a kappa of 0.80, suggesting excellent accuracy.

Reference: Harrold L, Pascal S, Lewis C, O'Keefe R, Pellegrini V, Allison J, Ayers D, Franklin P. Patient
Report Improves Posthospital Discharge Event Capture in Total Joint Replacement: A Novel Approach to
Capturing All Posthospital Event Data. EGEMS; 2014 Oct 22; 2(1). PMID:25848596
212 | P a g e
S16-6: James is an analyst working for a large healthcare organization trying to understand the incidence
of 90-day post-discharge complications. He has done a literature review of papers quantifying the
accuracy of patient self-report for the presence of complications using metrics such as agreement and
kappa. He believes that he has sufficient information to bring back to leadership about overall accuracy
of patient self-report to make an informed decision about the organization's planned outreach to
patients on the topic. What should he have done differently to avoid omitting an important element in
his assessment?

A. He should have looked for papers using "concordance" as a metric rather than "agreement" and
"kappa."

B. He should have limited his search to papers surveying patients at 1 year post-encounter or longer,
because recall accuracy improves with time.

C. He failed to consider the importance of also measuring accuracy of self-reporting for the absence of
events

D. He should have limited his search to papers in which analysis was limited to patients responding to
surveys, and not including analysis of non-responders (non-responder bias).

Answer: C. He failed to consider the importance of also measuring accuracy of self-reporting for the
absence of events

Explanation: A is incorrect because agreement and kappa are appropriate, particularly when prevalence
event rates are low because kappa does not inflate with low prevalence as does agreement or
concordance. B. Is incorrect because shorter times appear to be correlated with more accurate recall. C.
is correct because accuracy of self-report should include both the accuracy of the self-report of the
presence as well as absence of events. In fact, for low prevalence events, the cohort without an event is
much larger than the cohort with an event, and neglecting to capture accuracy of the absence of events
neglects the larger cohort of patients. Methodologically, it can be more challenging to determine
accuracy around the absence of events, which is why many studies have failed to include it. D. In
incorrect. Although non-responder bias analysis can be challenging, it is critical in understanding
whether the analysis of the responders can be generalized to the full population receiving surveys.

Reference: Alazzawi S, Bardakos NV, Hadfield SG, Butt U, Beer ZH, Field RE. Patient-reported
complications after elective joint replacement surgery: Are they correct? Bone Jt J; 2012 Aug 1;94-
B(8):1120-1125. PMID:22844056

213 | P a g e
Thursday, May 10th

S17: Presentations - Decision Support for Clinical Screening

The Power of Two: Rapid Screening and Intervening Combining clinical decision support (CDS) tools and
nurse-driven protocols to effectively evaluate patients for high risk conditions and potential interventions
at hospital admission

-- John McGreevey

S17-1: All of the following topics would be a good fit for the type of universal screening assessment
described in this presentation EXCEPT:

A. Pregnancy status

B. Capturing detailed oncologic history

C. MRSA status

D. Chlorhexidine bathing indications

Answer: B. Capturing detailed oncologic history

Explanation: The criteria for a good candidate for universal screening include applicability to a large
population of patients, high impact workflows without a current ideal screening solution, and limited
nursing assessment needed. Answers A, C and D meet this threshold while option B does not and would
require a provider to gather this history.

S17-2: Effective screening of patients for MRSA status, pregnancy status and to identify indicators for
chlorhexidine can be potentiated when clinical decision support is combined with which one of the
following?

A. Problem lists

B. Medication reconciliation

C. Pre-selected orders in order sets

D. Nurse-driven protocols and human assessment

Answer: D. Nurse-driven protocols and human assessment

Explanation: By combining clinical decision support tools with nurse-driven protocols and human
assessment patients have the potential to be more effectively screened for MRSA status, pregnancy
status, and indications for chlorhexidine bathing as described in this presentation.

214 | P a g e
Answer C is incorrect because the pre-selection of orders was described as an ineffective solution in this
presentation.

Answers A and B, while important concepts, were not components of the universal screening approach
described in this presentation.

Cancer Patient Trial Matching Assisted by An Efficient Automated Pre-Screening Application

--Hung Le

S17-3: Cancer claims the lives of half a million Americans every year. One of the major bottlenecks in
the development of anti-cancer drugs is the lack of sufficient patient enrollment in clinical trials. Which
of the following will likely happen if the enrollment goal is not met?

A. Delay or even termination of clinical trials, which, in turn, amounts to substantial loss to the
pharmaceutical companies

B. The sponsor will ask the principal investigator to refund the investment to recruit patients

C. The sponsor can extend the time frame for enrollment indefinitely until patient enrollment goal has
met

D. The investigator will be able to use synthetic patients to add to the number of actual patients to
reach the number needed for enrollment

Answer: A. Delay or even termination of clinical trials, which, in turn, amounts to substantial loss to the
pharmaceutical companies

Explanation: It is a poorly kept secret in the world of clinical trials that issues with patient recruitment
and enrollment are the primary causes for missing clinical trial timelines. For each day a company goes
beyond the planned deadline for a clinical trial, it could be lose as much as $600,000 in foregone sales of
smaller products and as much as $8 million on blockbuster drugs. Sponsors and CROs are eager to use
strategies involving technology to increase the rate of patient enrollment.

The costs associated with patient recruitment delays are not only high, but difficult to anticipate and
control. Consistently updated regulations, newly developed media by which to attract patients and
investigators, and selecting appropriate service providers for recruitment all play a role in patient
recruitment dynamics.

Reference: Industry Standard Research. (2014). The Expanding Web of Clinical Trial Patient Recruitment.
Retrieved from http://www.isrreports.com/wp-content/uploads/2014/04/ISR-The-Expanding-Web-of-
Clinical-Trial-Patient-Recruitment-Whitepaper.pdf

S17-4: At the department of Investigational Cancer Therapeutics (Clinical Phase I/II program), UT MD
Anderson Cancer Center, we have developed an automated patient pre-screening application to

215 | P a g e
significant reduce manual effort to 92% and increase enrollment rate. What eligibility criteria were used
in this automated application? (select all that apply)

A. Patient age

B. Tumor type or disease

C. Number of prior therapies

D. Molecular target

E. Performance status

Answer: B. Tumor type or disease D. Molecular target

Explanation: To develop this pre-screening application, we first identified two of the most discerning
eligibility criteria in oncology trials: tumor type and molecular target (mutation). Mutation analysis plays
an indispensable role in identifying actionable molecular events, predicting responses to targeted
therapies and ensuring optimal treatment

Reference: H. Le, J. Zeng, C. Beer, S. Montez, D. Sanders, J. Gong et al ,Cancer Patient Trial Matching
Assisted by An Efficient Automated Pre-Screening Application. AMIA 2018 Clinical Informatics
Conference, Phoenix, AZ, May 8-10.

StartSmartTM: Decision Support Technology for Prenatal Screening

--Bonnie Gance Cleveland

S17-5: Which of the following best describes the iterative IT development process used in the creation
of StartSmart, prenatal screening application, guided by the Davis’s Technology Acceptance model?

a) Review of the literature, focus groups with pregnant women, technology development using Davis
Technology Acceptance Model, patient and provider end-user testing, refinement

b) Focus groups with providers & patients guided by the SBIRT Model, prototype development, alpha
testing, beta testing

c) Prototype development using guidelines, focus groups, and expert consultation; alpha testing with
patients and providers; revisions; beta testing with patient and providers; refinement

Answer: c) Prototype development using guidelines, focus groups, and expert consultation; alpha
testing with patients and providers; revisions; beta testing with patient and providers; refinement

Explanation: The iterative development guided by the Davis’s Technology Acceptance Model used in
this project included foundational work by our group including reviewing the guidelines, conducting
focus groups with providers and patients, interviewing expert clinicians and patients, developing a

216 | P a g e
prototype, alpha testing with patients & provider, refining, beta testing with patient and providers, and
refining.

Reference: Belanger E, Bartlett G, Dawes M, Rodriguez C, Hasson-Gidoni I. Examining the evidence of

the impact of health information technology in primary care: an argument for participatory research
with health professionals and patients. Int J Med Inform. 2012;81(10):654-661.

S17-6: Which one of the following best describes the process for applying the Davis’s Technology
Acceptance Model to the assessment of the StartSmart Prenatal Screening Technology?

a) An iterative development of the decision support/patient education application with end users’ input
at each level

b) Assessment of usability and ease of use was included in each step of the development

c) Assessment included user input at each phase/ beta testing with patient interviews that addressed
acceptability, ease of use, and intention to change; provider surveys regarding acceptability and ease
of use.

Answer: c) Assessment included user input at each phase/ beta testing with patient interviews that
addressed acceptability, ease of use, and intention to change; provider surveys regarding acceptability
and ease of use.

Explanation: The iterative development and assessment guided by the Davis’ Technology Acceptance
Model used in this project included expert clinicians and patients, developing a prototype, alpha testing
with patients & provider with questions specific to navigation, usability, and ease of use; refining; beta
testing with patient and providers regarding usability and ease of use; and further refinement.

Reference: Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information
technology. MIS Quarterly. 1989;13(3):319-340.

S18: Presentations - Supporting Best Practices

Tracking and Managing Abnormal Mammograms with SMART

--Caitlin Cusack

S18-1: At the Veteran’s Health Administration, the SMART project is being implemented to improve the
ability to track clinical management against a single index abnormal mammogram via the use of a
“Breast Care Cascade.” Which of the following “opens” a cascade?

217 | P a g e
a. The text of a received mammogram report is entered into the EHR.

b. An order for a mammogram is placed.

c. A woman who is due for a mammogram presents for a preventive care visit.

d. The pathology report for a breast biopsy is entered into the EHR.

Answer: b. An order for a mammogram is placed.

Explanation: A Breast Care Cascade is opened under three circumstances: The BIRAD code of the
mammogram is entered into the BIRAD field in the record; the provider enters an outside result into the
SMART template; or an order for a mammogram is placed. (a) is incorrect since free-text does not open
the cascade. (b) is incorrect since merely being due for a mammogram does not necessarily result in the
woman having a mammogram ordered. (d) is incorrect since the Breast Care Cascade tracks care against
a single index mammogram. Data regarding a breast biopsy done in the management of an abnormal
mammogram would be included in the cascade, but in and of itself would not start a cascade.

Reference: Callen JL, Westbrook JI, Georgiou A, et al. Failure to follow-up test results for ambulatory
patients: a systematic review. Journal of general internal medicine. 2012 Oct 1;27(10):1334-48.

S18-2: Ms. Jane Doe presents for a routine preventive care visit to her primary care physician (PCP). She
is due for a mammogram that is ordered. When completed the mammogram is recorded as a BIRAD 0:
Incomplete. The radiologist discusses the findings with the physician covering for Ms. Doe’s PCP, who
orders additional views. The followup views are done the following day and reported as BIRAD 4,
suspicious abnormality. Both reports generate SMART alerts, one to the PCP, the second to the covering
physician. When the PCP views the initial BIRAD 0 alert, which SMART template is shown?

a. BIRAD 0 SMART Template.

b. BIRAD 2 SMART Template.

c. BIRAD 4 SMART Template.

d. Multiple Tests Please Summarize SMART Template.

Answer: c. BIRAD 4 SMART Template.

Explanation: The linked SMART template defaults to the highest BIRAD result across all alerts and
providers. If the follow up images were not completed when the provider opens the alert than (a) the
BIRAD 0 SMART Template would be linked. (b) is incorrect since none of the images were coded as a
BIRAD 2. If there had been a breast image, such as an ultrasound, without a BIRAD code then (d)
Multiple Tests Please Summarize SMART Template would be linked.

218 | P a g e
Murphy DR, Meyer AN, Vaghani V, et.al. Electronic triggers to identify delays in follow-up of
mammography: harnessing the power of big data in health care. Journal of the American College of
Radiology. 2017 Nov 1.

Integrating Evidence to Inform Lab Test Selection into a Knowledge Management Framework

--Spencer DesAutels

S18-3: An academic medical center makes evidence-based local practice decisions across multiple
content domains on a frequent, recurring basis. New external knowledge, e.g., a new guideline or meta-
analysis, is shared through various institutional channels and eventually leads to each institutional
decision being made or revised. Documenting this underlying knowledge has the potential to promote
adherence reuse, while simultaneously establishing a basis for crucial ongoing decision maintenance. To
maximize efficiency of documentation, the institution should consider which of the following?

a. Creating a shareable document that provides a running tally of all decisions by a decision-making
body.

b. Creating a single record of the underlying knowledge and linking it to a unique record of each
decision.

c. Selecting a standardized citation format for each publication type.

d. Requiring all committees to use an approved communication channel for official communications.

Answer: b. Creating a single record of the underlying knowledge and linking it to a unique record of each
decision.

Explanation: (b) is the best option because this allows the institution to create and maintain the
minimum number of records of the knowledge disseminated across the institution and formally links
each decision that draws on the same evidence. While (a) shares many of these benefits, it does not
allow the institution to clearly draw links and ensure consistency between the decisions from multiple
committees/working groups that draw on the same knowledge. (c) and (d) are unrelated to efficiency of
knowledge documentation.

Reference: Shellum JL, Freimuth RR, Peters SG, Nishimura RA, Chaudhry R, Demuth SJ, Knopp AL, Miksch
TA, Milliner DS. Knowledge as a Service at the Point of Care. AMIA Annu Symp Proc. 2017 Feb
10;2016:1139-1148. PubMed PMID: 28269911.

S18-4:There are many sources that provide relevant information on lab testing for patient care decisions
and analytics. These sources are internal and external, and refer to testing at different scales and for
different purposes. Furthermore, sources are constantly emerging, changing, or being deprecated. To
make a fully informed decision, e.g., selection of lab tests to perform or lab tests to restrict to certain
clinical scenarios, it is often necessary to consult multiple resources concurrently. How can an institution
maximize use of these heterogeneous data sources?
219 | P a g e
a. Publish a list of approved resources.

b. Negotiate contracts with vendors to present data in institutionally-approved formats.

c. Invest in mechanisms to bring informational resources to the point of care.

d. Bring all applicable data together in a consistent internal format that references each data source.

Answer: d. Bring all applicable data together in a consistent internal format that references each data
source.

Explanation: (d) is the best option because it allows flexibility in integrating new sources as they become
available, provides a single access point, and creates a standard representation users can gain familiarity
with. (a) allows a user to have confidence in selected sources, but still places a burden on the user to
consult each individually. Among other challenges, (b) precludes the existence of emergent knowledge
sources, while (c) is not relevant to acknowledging heterogeneity of data sources.

Reference: Lee VS, Kawamoto K, Hess R, Park C, Young J, Hunter C, Johnson S, Gulbransen S, Pelt CE,
Horton DJ, Graves KK, Greene TH, Anzai Y, Pendleton RC. Implementation of a Value-Driven Outcomes
Program to Identify High Variability in Clinical Costs and Outcomes and Association With Reduced Cost
and Improved Quality. JAMA. 2016 Sep 13;316(10):1061-72. doi: 10.1001/jama.2016.12226. PubMed
PMID: 27623461.

Cleaning House: EHR Order Set Innovation and Transformation at Penn Medicine

--John McGreevey

S18-5: All of the following contributed to the success of the order set transformation process described
in this presentation except:

A) Specialty Lead engagement

B) Legacy Order Set analysis

C) Creation of a modular order set framework

D) Development of a standard order set layout and naming convention

E) Replication of all legacy order sets into the new EHR

Answer: E) Replication of all legacy order sets into the new EHR

Explanation: In the order set transformation process presented, there was a conscious decision to not
replicate each and every legacy order set. Rather, there was a decision to carefully assess the need for
each legacy order set, in the interest in consolidating order sets and reducing the total number of order
sets in circulation. All of the other responses were described as success factors in the order set
transformation.

220 | P a g e
S18-6: Based on this presentation, each of the following order set sections would be a suitable core
module except:

A) Admission

B) Labs

C) Vital Signs

D) Notify Physician

E) Activity

Answer: B) Labs

Explanation: Based on the experience of the presenters, the optimal core modules were ones where
there was broad commonality in the content across order sets. These modules could be built once,
include all likely orders within that category, and then be re-used in future order sets. Commonalities
were observed in admission orders, vital signs, notify physician, and activity sections of legacy order
sets. In contrast, lab orders tended to be variable from one order set to another and each order set
tended to have a unique combination of lab orders, thus labs would be unsuitable as a core module.

References:

1. A general reference that discusses broad themes around implementation of HIT:

Cresswell KM, Bates DW, Sheikh A. Ten key considerations for the successful implementation and
adoption of large-scale health information technology. JAMIA. 2013;20(e1):e9-e13. doi:10.1136/amiajnl-
2013-001684.

2. A work describing lessons learned during another institution’s EHR migration:

Rizer MK, Kaufman B, Sieck CJ, Hefner JL, McAlearney AS. Top 10 lessons learned from electronic
medical record implementation in a large academic medical center. Perspectives in Health Information
Management. 2015;12(Summer):1g.

3. A reference to ONC’s “How to implement EHRs” guide:

How to implement EHRs. Office of the National Coordinator for Health Information Technology.
Available from: https://www.healthit.gov/providers-professionals/ehr-implementation-steps

4. A trade press article describing Vanderbilt’s preparations for their recent EHR conversion:

Barkholz D. Vanderbilt is a case study for the dreaded EHR conversion. Modern Healthcare. May 1, 2017.
Available from: http://www.modernhealthcare.com/article/20170501/NEWS/170509989

221 | P a g e
S19: Panel - Implementing and Spreading Patient-centered, Evidence-Based Technologies: The Fall
TIPS Story

Patricia Dykes, Denise Goldsmith, Ann O’Brien

S19-1: The Fall TIPS (Tailoring Interventions for Patient Safety) program is based on over a decade of
research supporting its effectiveness with reducing falls and related injuries in hospitals. Which of the
following are most likely to prevent a fall in patients with physiological risks for falling?
A. Universal fall precautions
B. A bed alarm
C. A fall prevention plan tailored to patient-specific risk factors
D. A fall prevention bundle

Answer C. A fall prevention plan tailored to patient-specific risk factors

Explanation: Patients at risk for physiological falls require interventions that are tailored to their
personal risk factors. Patients at risk due to a gait disturbance require different interventions than
patients at risk due to a cognitive deficit or confusion.

References: Dykes PC, Carroll DL, Hurley A, Lipsitz S, Benoit A, Chang F, Meltzer S, Tsurikova R, Zuyov L,
Middleton B. Fall prevention in acute care hospitals: a randomized trial. JAMA. 2010 Nov 03;
304(17):1912-8. PMID: 21045097; PMCID: PMC3107709.

Dykes PC, Duckworth M, Cunningham S, Dubois S, Driscoll M, Feliciano Z, Ferrazzi M, Fevrin FE, Lyons S,
Lindros ME, Monahan A, Paley MM, Jean-Pierre S, Scanlan M. Pilot Testing Fall TIPS (Tailoring
Interventions for Patient Safety): a Patient-Centered Fall Prevention Toolkit. Jt Comm J Qual Patient Saf.
2017 Aug; 43(8):403-413. PMID: 28738986.

S19-2: Which of the following are most likely to support evidence-based fall prevention interventions for
all patients?
A. Universal fall precautions
B. Clinical decision support
C. A fall prevention bundle
D. Infobuttons in the EHR

Answer B. Clinical decision support

Explanation: There are many factors that place hospitalized patients at risk for falls. Clinical decision
support can link patient-specific risk factors to interventions most likely to prevent a fall based on each
individual patient’s fall risk profile.

222 | P a g e
References: Dykes PC, Carroll DL, Hurley A, Lipsitz S, Benoit A, Chang F, Meltzer S, Tsurikova R, Zuyov L,
Middleton B. Fall prevention in acute care hospitals: a randomized trial. JAMA. 2010 Nov 03;
304(17):1912-8. PMID: 21045097; PMCID: PMC3107709.

Dykes PC, Duckworth M, Cunningham S, Dubois S, Driscoll M, Feliciano Z, Ferrazzi M, Fevrin FE, Lyons S,
Lindros ME, Monahan A, Paley MM, Jean-Pierre S, Scanlan M. Pilot Testing Fall TIPS (Tailoring
Interventions for Patient Safety): a Patient-Centered Fall Prevention Toolkit. Jt Comm J Qual Patient Saf.
2017 Aug; 43(8):403-413. PMID: 28738986.

S19-3: Which of the following is the strategy most likely to achieve successful implementation of Fall
TIPS into an EHR?

A. Hospital leadership support

B. Integration during the design/pre-implementation phase
C. Fall prevention workflow analysis
D. Front line staff involvement

Answer B. Integration during the design/pre-implementation phase

Explanation: There are many factors that lead to a successful implementation of the Fall TIPS program
into a vendor EHR product. Integration of Fall TIPS into the nurse’s workflow in advance of “go live” will
most likely reduce delays, obstacles and barriers demonstrated when integration occurs during
“optimization” phase.

References: Dykes PC, Carroll DL, Hurley A, Lipsitz S, Benoit A, Chang F, Meltzer S, Tsurikova R, Zuyov L,
Middleton B. Fall prevention in acute care hospitals: a randomized trial. JAMA. 2010 Nov 03;
304(17):1912-8. PMID: 21045097; PMCID: PMC3107709.

Dykes PC, Duckworth M, Cunningham S, Dubois S, Driscoll M, Feliciano Z, Ferrazzi M, Fevrin FE, Lyons S,
Lindros ME, Monahan A, Paley MM, Jean-Pierre S, Scanlan M. Pilot Testing Fall TIPS (Tailoring
Interventions for Patient Safety): a Patient-Centered Fall Prevention Toolkit. Jt Comm J Qual Patient Saf.
2017 Aug; 43(8):403-413. PMID: 28738986.

S19-4: While the Fall TIPS program is currently used in over 100 hospitals in the US, and around the
world, there are challenges to spread that can be addressed by organization leadership. What strategy
has been used to improve long term effectiveness of the Fall TIPS program.

A. Regular fall prevention education

B. Regular leadership participation in safety rounds
C. Regular leadership seminars on fall prevention
D. Regular monitoring/feedback related to patient engagement in fall prevention

223 | P a g e
Answer D. Regular monitoring/feedback related to patient engagement in fall prevention

Explanation: Patient engagement in all three steps of the fall prevention process (risk assessment,
developing a tailored plan, and consistently implementing the plan) improves the “dose” of the Fall TIPS
intervention and has been shown to decrease both falls and fall-related injuries.

References: Dykes PC, Carroll DL, Hurley A, Lipsitz S, Benoit A, Chang F, Meltzer S, Tsurikova R, Zuyov L,
Middleton B. Fall prevention in acute care hospitals: a randomized trial. JAMA. 2010 Nov 03;
304(17):1912-8. PMID: 21045097; PMCID: PMC3107709.

Dykes PC, Duckworth M, Cunningham S, Dubois S, Driscoll M, Feliciano Z, Ferrazzi M, Fevrin FE, Lyons S,
Lindros ME, Monahan A, Paley MM, Jean-Pierre S, Scanlan M. Pilot Testing Fall TIPS (Tailoring
Interventions for Patient Safety): a Patient-Centered Fall Prevention Toolkit. Jt Comm J Qual Patient Saf.
2017 Aug; 43(8):403-413. PMID: 28738986.

S20: Presentations - Safety: Root Cause Analysis, Virtual Care, and Medical Reconciliation

Verbal Orders: Root Cause Analysis of Co-signature Authentication Timeframes

--Thomas Powell

S20-1: During a visit from the Joint Commission, the inspection team asks for additional information
regarding verbal orders (VO) usage for the respiratory therapy department. Aside from the number and
percentage of verbal orders, which additional analysis can be performed to present a more detailed
overview of order procedures?
A. Perform a dual domain analysis of time/day for threshold based similarity
B. Parse the data to find patterns in VO receiver(s) and/or author(s)
C. Compare the respiratory therapy VO statistics against other departments in the health system
D. Compare the VO records against the health care system policy, describing the data points and
timeline to authenticate the validity/authenticity/adherence of the order.
E. Identify grouping/bundling of orders to differentiate clusters from single purpose VO

Answer: D. Compare the VO records against the health care system policy, describing the data points
and timeline to authenticate the validity/authenticity/adherence of the order.

Explanation: Rationale: Given the current guidance and legislation, the onus regarding VO timelines,
rate of issuance and monitoring falls directly on the institution. While pattern analysis of
time/frequency, receiver/author, department comparisons and overall granularity analysis might yield
vital information regarding VO for health care administrators, the regulatory elements focus more on
policy adherence than the constituent data used to create and inform the health care system rules and
procedures.

224 | P a g e
References: A review of verbal order policies in acute care hospitals (2012), The Joint Commission
Journal on Quality and Patient Safety, Volume 38, Issue 1, Pages 24-AP4.

S20-2: When analyzing Verbal Order (VO) records within the EHR, which data element is often
overlooked?
A. Staff scheduling
B. Discharge timelines
C. Clinic type
D. Physician inbox time
E. Mode of VO delivery

Answer: D. Physician inbox time

Explanation: Rationale: Scheduling, discharge, clinic type and mode (verbal or telephone) have all been
addressed in published studies regarding VO analysis within the last twenty years. Given the role of the
author is the primary means of VO authentication, the timeline of review is vital towards establishing a
root cause analysis.

References: Gellert GA, Catzoela L, Patel L, Bruner K, Friedman F, Ramirez R, Saucedo L, Webster SL,
Gillean JA. The Impact of Order Source Misattribution on Computerized Provider Order Entry (CPOE)
Performance Metrics. Perspectives in health information management. 2017;14(Spring).

Quality Standards for Virtual Care: The Roles and Responsibilities of Stakeholders for Advancing Care
Quality in an Online Environment

--Kevin Smith

S20-3: A virtual care (telemedicine) provider group has convened a clinical quality council to provide
guidance for ongoing clinical quality monitoring and management.

Which of the options below is most appropriate for establishing quality outcome metrics?

A. Start out by creating a new virtual care framework for measuring clinical quality.

B. Partner with a statistician to analyze virtual care claims data.

C. Establish a disciplinary action and remediation program for underperforming doctors.

D. Establish outcome measure targets for the most common virtual care visit types.

225 | P a g e
Answer: D. Establish outcome measure targets for the most common virtual care visit types.

Explanation: Establishing outcome measure targets provides a standardized framework for evaluating
quality performance. These outcomes also relate to clinical effectiveness, or the impact of telemedicine
on outcomes, such as adhering to best practice guidelines.

Incorrect options: the standards or framework for virtual care clinical should not be different from care
provided in other settings, a statistical analysis is not helpful until the outcome measures have been
established, disciplinary action for underperforming doctors does not relate to establishing clinical
quality criteria.

Reference: Creating a Framework to Support Measure Development for Telehealth (pp. 9-14, Rep.).
(2017). Washington, D.C.: National Quality Forum.

S20-4: A health care system wishes to improve appropriate antibiotic prescribing practices for the
management of acute upper respiratory infections in the virtual care (e-visit) setting by improving
clinician adherence to an evidence-based clinical guideline.

Given the above information, which of the following may be the most appropriate intervention to
improve adherence to the evidence-based clinical guideline?

A. Add a link to the Infectious Disease Society of America web site on the clinician screen.

B. Incorporate contextual clinical decision support which highlights pertinent patient history of the
current illness, a brief summary of the guideline criteria with actionable guidance, all within the
clinician workflow.

C. Provide a variety of pop-up reminders and alerts reminding clinicians of the importance of antibiotic
stewardship.

D. Disable e-prescribing functionality for antibiotic medications to prevent clinician over-prescribing.

Answer: B. Incorporate contextual clinical decision support which highlights pertinent patient history of
the current illness, a brief summary of the guideline criteria with actionable guidance, all within the
clinician workflow.

Explanation: CDS strategies are most effective when the users are knowledgeable about the clinical
guideline and are provided with CDS that is incorporated into their typical workflow - such as
highlighting key data elements and including a brief summary of the guideline within the workflow along
with actionable guidance.

Individual interventions outside of the clinical workflow, such as provider education, may or may not
result in changes in clinician adherence to clinical guidelines.

Incorrect options:

226 | P a g e
A. “Add a link to the Infectious Disease Society of America web site on the clinician screen.” This item is
incorrect as it requires the clinician to browse to an external web resource. Although the IDSA web site
has excellent evidence based guideline information, the clinician may have to search for the guideline
articles making this workflow inconvenient and less effective.

C. “Provide a variety of pop-up reminders and alerts reminding clinicians of the importance of antibiotic
stewardship.” This item is incorrect since reminders about antibiotic stewardship are less specific or
contextual compared to item b. Pop-up alerts and reminders may lead to "alert fatigue" which can
reduce the CDS effectiveness.

D. “Disable e-prescribing functionality for antibiotic medications to prevent clinician over-prescribing.”

This item is incorrect because it is extremely impractical. Disabling antibiotic prescribing would certainly
prevent over-prescribing of antibiotics but would also prevent the appropriate prescribing of antibiotics.

Reference: Castillo RS, Kelemen A. Considerations for a successful clinical decision support system.
Comput Inform Nurs. 2013;31(7):319-26.

S20-5:Which of the following is most true regarding the current state of clinical quality monitoring and
management among virtual care (telemedicine) providers and vendors in the United States?

A. The CDC has been adopted as the source of truth for virtual care quality standards.

B. Any virtual care provider or vendor who can demonstrate that they have a medical reference
available for the clinicians, such as UpToDate, is considered to be compliant with quality standards.

C. There are no established standards for measuring clinical quality outcomes in virtual care in the
United States.

D. The World Health Organization, in partnership with AHRQ, is considered as the source of truth for
virtual care quality

Answer: C. There are no established standards for measuring clinical quality outcomes in virtual care in
the United States.

Explanation: Although many providers, vendors, and organizations (such as the American Telemedicine
Association) have developed and published clinical guideline documents for virtual care, there currently
are no agreed-upon outcome measurement standards.

Incorrect options: The CDC has excellent resources for clinical guidance but has not been adopted as the
standard, creating guideline documents or making a clinical reference available does not ensure that
guidelines are followed. The WHO and AHRQ are also excellent resources related to clinical quality but
have not been adopted as the standard.

Reference: Practice Guidelines for Live, On-Demand Primary and Urgent Care [PDF]. (2014, November).
Washington, D.C.: American Telemedicine Association.

227 | P a g e
S20-6: The quality committee has a debate about how to approach clinical quality as it relates to the
relatively new and rapidly growing model of virtual care.

What is the greatest challenge for establishing clinical quality targets for virtual care?

A. The conditions treated via virtual care are vastly different than those treated in a clinic setting.

B. The challenges for establishing clinical quality targets are essentially the same no matter the
method of care delivery.

C. The potential limitations of telemedicine equipment and internet connectivity could make it more
difficult to deliver quality care.

D. The lack of research in this area makes it impossible to establish quality targets.

Answer: B. The challenges for establishing clinical quality targets are essentially the same no matter the
method of care delivery.

Rationale: Establishing standards for the quality of virtual care based on evidence-based, nationally
recognized best practices is an effective way to support quality in the performance of a virtual care
program while avoiding unnecessary and redundant measure development.

Incorrect options: Conditions treated via virtual care are no different than those treated in other
settings, equipment and internet connectivity should not affect a clinician's ability to provide
appropriate care as other options can be provided, and the availability of research in this area has no
impact on setting quality standards.

Reference: Creating a Framework to Support Measure Development for Telehealth (pp. 9-14, Rep.).
(2017). Washington, D.C.: National Quality Forum.

Using Qualitative Techniques to Classify Medication Reconciliation Errors

--Rod Tarrago

S20-7: According to the Institute for Healthcare Improvement (IHI), the medication reconciliation
process involves three steps. Which of these is NOT one of the steps:

A. Verification

B. Clarification

C. Communication

D. Reconciliation

Answer: C. Communication
228 | P a g e
Explanation: According to the IHI, the process involves verification (collection of the medication history),
clarification (ensuring that the medications and doses are appropriate) and reconciliation
(documentation of the changes in orders). Communication is involved in all three but is not a specific
step.

Reference: Institute for Healthcare Improvement. 5 million lives getting started kit: preventing adverse
drug events (medication reconciliation), how- to guide. Available at:
http://www.ihi.org/IHI/Programs/Campaign/ ADEsMedReconciliation.htm. Published Oct. 1, 2008.
Accessed September 2010.

S20-8: The Joint Commission states that the intention of medication reconciliation is to address all of the
following except:

A. Duplications

B. Expirations

C. interactions

D. Omissions

E. Continuations

Answer: B. Expirations

Explanation: According to the Joint Commission, there is evidence that medication discrepancies can
affect patient outcomes. Medication reconciliation is intended to identify and resolve discrepancies—it
is a process of comparing the medications a patient is taking (and should be taking) with newly ordered
medications. The comparison addresses duplications, omissions, and interactions, and the need to
continue current medications. The types of information that clinicians use to reconcile medications
include (among others) medication name, dose, frequency, route, and purpose. Organizations should
identify the information that needs to be collected to reconcile current and newly ordered medications
and to safely prescribe medications in the future.

Reference: https://www.jointcommission.org/assets/1/6/NPSG_Chapter_OME_Jan2017.pdf

229 | P a g e
S21: Presentations - Patient Generated Data, Interventions, and Telehealth

Patient Generated Safety Data in the Hospital Setting: Trust, Transparency and Learning

--Sarah Collins

S21-1. Which of the following related to safety reporting in the hospital is true?

A. Most recognized safety concerns and incidents are reported

B. Patients and families are not considered key stakeholders in safety reporting

C. Most recognized safety concerns and incidents are not reported

D. Safety data reported by patients and families provides little insights

Answer: C. Most recognized safety concerns and incidents are not reported

Explanation: Only a fraction of safety concerns and safety risks that may lead to adverse events are
reported. In addition, hospital safety incidents are underreported and this limits our understanding of
safety and ability to learn from unsafe system processes. Patients and families are important
stakeholders in hospital safety reporting, and their concerns can provide valuable insights.

References:

Levinson DR. Hospital incident reporting systems do not capture most patient harm. Washington DC:
Office of the Inspector General. 2012 Jan.

Weingart SN, Pagovich O, Sands DZ, et al. What can hospitalized patients tell us about adverse events?
Learning from patient‐reported incidents. J Gen Intern Med. 2005 Sep 1;20(9):830-6.

Couture, B, Lilley, E, Chang, F, et al. User-centered design methods for development of an mHealth
application for use in the hospital setting by patients and care partners. Appl Clin Inform. In press 2018.

S21-2: Which of the following is an identified sociotechnical requirement for patient safety reporting
most related to trust from patients?

A. Availability of a real-time clinician dashboard

B. Protected anonymity when submitting

C. Comparing data to other safety data sets

D. Clear responsibility for follow up

Answer: B. Protected anonymity when submitting

Explanation: The ability to submit anonymously should be an option for patients and families using a
safety reporting application in the hospital setting. Users should also feel confident that this anonymity
230 | P a g e
will be protected and preserved. This is important for maintaining trust of patients and families to
foster a culture of safety.

Reference: Couture, B, Lilley, E, Chang, F, et al. User-centered design methods for development of an
mHealth application for use in the hospital setting by patients and care partners. Appl Clin Inform. In
press 2018.

Building and Evaluating a Patient-Facing Intervention to Deliver Implantable Cardioverter-Defibrillator

Remote Monitoring Data to Patients via MyChart: Patient and Provider Perceptions

--Romisa Rohani Ghahari

S21-3: A healthcare system wishes to send personalized information to patients based on implantable
cardiac defibrillator (ICD) remote monitoring data. Survey data from a recent study showed that most of
the participants (86%) wanted to receive more information than they currently receive from the clinic
about their remote monitoring data through the standard of care letter. Which of the following is most
appropriate based on the findings from this study?

A. Patients would be most satisfied with a standard of care letter when compared to the ICD data
summary.

B. Patients found the standard of care letter easier to access and use compared to ICD data summary.

C. Participants found the mailed letter with the ICD data summary easier to access compared to the
electronic format.

D. Either method (mailed letter vs. electronic format) of providing the ICD data summary works well
for the patients.

Answer: D. Either method (mailed letter vs. electronic format) of providing the ICD data summary works
well for the patients.

Explanation: The goal of a healthcare system is to personalize device information based on patients’
preferences. The results of this study showed that although electronic format was easier to access,
either mailed letter or electronic format worked for the patients. Patients preferred to receive the ICD
data summary over the standard of care letter.

Reference: Ghahari RR, Holden RJ, Flanagan ME, Wagner S, Martin E, Ahmed R, Daley CN, Tambe R,
Chen E, Allmandinger T, Mirro MJ. Using cardiac implantable electronic device data to facilitate health
decision making: A design study. International Journal of Industrial Ergonomics. 2018 Mar 31;64:143-54.

S21-4: A healthcare system wishes to provide patients with their device information without increasing
clinical workload and having a negative effect on providers. Prior research reported that providers were

231 | P a g e
concerned that messaging patients with their device data could potentially increase the clinical
workload. We evaluated the effect of providing patients with their device information on clinical
workload and clinician attitude towards ICD data summary. Which of the following is most appropriate
based on the findings from this study?

A. Providing patients with their device information did increase healthcare utilization.

B. Providing patients with their device information did not increase the workload of clinicians from
increased patient questions (calls or PHR messaging).

C. Clinicians believed the ICD data summary was NOT easy for patients to understand and use for
managing their health.

D. Providing patients with their device information did increase the device-specific workload for
clinicians.

Answer: B. Providing patients with their device information did not increase the workload of clinicians
from increased patient questions (calls or PHR messaging).

Explanation: The results of this study showed that providing patients with their device information did
not increase the device-specific workload for clinicians or healthcare utilization in patients. Further,
clinicians who participated in this study agreed that patients would be able to use the ICD data summary
for self-management.

Reference: Ghahari RR, Holden RJ, Flanagan ME, Wagner S, Martin E, Ahmed R, Daley CN, Tambe R,
Chen E, Allmandinger T, Mirro MJ. Using cardiac implantable electronic device data to facilitate health
decision making: A design study. International Journal of Industrial Ergonomics. 2018 Mar 31;64:143-54.

Predictors of average adherence to the use of home telehealth devices in Veterans with heart failure

--Jenice Ria Guzman

S21-5: Adherence to the use of home telehealth devices in Veterans with heart failure enrolled in the VA
Home Telehealth Program ________ over time.
a. decreased
b. increased
c. reversed
d. stayed the same

Answer: a. decreased

Explanation: Current study that will be presented found that adherence to the use of home telehealth
devices decreases over time in Veterans with heart failure.

232 | P a g e
References: Guzman-Clark J, van Servellen G, Chang B, Mentes J, & Hahn T (2013). Predictors and
outcomes of early adherence to the use of a home telehealth device by older Veterans with heart
failure. Telemedicine & e-Health, 19(3):217-223.

Ong M, Romano PS, Edgington S, et al. Effectiveness of remote patient monitoring after discharge of
hospitalized patients with heart failure: The Better Effectiveness After Transition-Heart Failure (BEAT-
HF) Randomized Clinical Trial. JAMA Intern Med 2016;176(3):310-318.

S21-6: It is important to understand adherence to the use of home telehealth devices because:
a. Adherence is complicated and needs to be monitored.
b. Home telehealth devices are expensive and should only be given to patients who will use them.
c. Poor adherence to home telehealth devices over time may be a potential reason for negative
results in prospective randomized controlled trials.
d. There needs to be a conversation with patients regarding expected adherence to the use of home
telehealth devices for outcome effectiveness.

Answer: c. Poor adherence to home telehealth devices over time may be a potential reason for negative
results in prospective randomized controlled trials.

Explanation: Although all the answers have some truth in them, c is the best answer and has been
previously mentioned in prior articles on the topic.

References: Chaudhry SI, Mattera JA, Curtis JP, et al. Telemonitoring in patients with heart failure. N
Engl J Med 2010; 363: 2301–09. Guzman-Clark J, van Servellen G, Chang B, Mentes J, & Hahn T (2013).
Predictors and outcomes of early adherence to the use of a home telehealth device by older Veterans
with heart failure. Telemedicine & e-Health, 19(3):217-223.

Ong M, Romano PS, Edgington S, et al. Effectiveness of remote patient monitoring after discharge of
hospitalized patients with heart failure: The Better Effectiveness After Transition-Heart Failure (BEAT-
HF) Randomized Clinical Trial. JAMA Intern Med 2016;176(3):310-318.

233 | P a g e
S22: Presentations - Identifying Medication Risk

Building a pharmacist prioritization tool using pharmacy-specific data and readmission risk

--Samuel Kabue

S22-1: When building a universal readmission risk prediction model for use in pharmacy, it is important
to base the modeling strategy on:

A. Commercially available 'black box' prediction models

B. A single predictor such as age or sex

C. Predictors that are readily available in electronic medical records within many health
systems/pharmacy departments

D. Pharmacists have time to intervene on every patient and therefore have little need for risk prediction
models

Answer: C. Predictors that are readily available in electronic medical records within many health
systems/pharmacy departments

Explanation: One challenge in universal use of pharmacy-specific prediction models is ensuring that all
predictors and outcomes used in the model are readily available within a particular health system. When
a health system/pharmacy department does not capture a predictor or outcome, the model cannot be
used effectively.

Robust prediction models will typically require more than one predictor.

Budgets restrict the purchase of commercial products and even when purchased, coefficients and some
predictors remain proprietary limiting interpretation.

Pharmacist-capacity is generally overwhelmed by the low pharmacist-patient ratio.

References: Escobar, G. J., et al. (2015). "Nonelective Readmissions and Postdischarge Mortality:
Predictive Models Suitable for Use in Real Time." Med Care 53(11): 916-923.

Steyerberg, E. W., et al. (2010). "Assessing the performance of prediction models: a framework for
traditional and novel measures." Epidemiology 21(1): 128-138.

Kansagara, D., et al. (2011). "Risk prediction models for hospital readmission: a systematic review."
JAMA 306(15): 1688-1698.

Escobar G, Greene J, Scheirer P, et al. Risk adjusting hospital inpatient mortality using automated
inpatient, outpatient, and laboratory databases. Med Care. 2008;46:232–239.

S22-2: A pharmacy department or health system wishes to use our readmission risk prediction model
but does not have a COPS2 score, LAPS2 score or RxDxCG score readily available. Which if the following

234 | P a g e
predictors currently available to them can be used to in place of these predictors for the best
performing model?

A. Diagnosis groupings

B. Drug subgroups

C. Admission category indicators (i.e. emergency department versus medical)

D. Number of refills on each medicine

Answer: B. Drug subgroups

Explanation: Our study shows that addition of COPS2, LAPS2 and RxDxCG scores to the reference model
of age, sex, LOS and admission category makes a good performing model. Individually COPS2 is
associated with the greatest increase in performance over the reference model. RxDxCG is associated
with new identical increase in performance as the drug subgroups, therefore implying that they are
interchangeable in our models.

Diagnosis groupings were associated with the smallest increase in model performance.

Admission category was part of the base reference model.

There is no evidence yet linking number of refills to readmission risk.

References:

Escobar, G. J., et al. (2015). "Nonelective Readmissions and Postdischarge Mortality: Predictive Models
Suitable for Use in Real Time." Med Care 53(11): 916-923.

Steyerberg, E. W., et al. (2010). "Assessing the performance of prediction models: a framework for
traditional and novel measures." Epidemiology 21(1): 128-138.

Kansagara, D., et al. (2011). "Risk prediction models for hospital readmission: a systematic review."
JAMA 306(15): 1688-1698.

Escobar G, Greene J, Scheirer P, et al. Risk adjusting hospital inpatient mortality using automated
inpatient, outpatient, and laboratory databases. Med Care. 2008;46:232–239.

A data mining approach to detect medication order near misses associated with electronic order sets

--Ron Li

S22-3: Which of the following best describes an error induced by automation bias?

A. Dr. Smith is alerted by an automatic EHR pop-up alert notifying that his patient may be developing
severe sepsis and includes a button that places an order for 30cc/kg of normal saline. However, this
patient also has congestive heart failure with an ejection fraction of 20%. Dr. Smith, assuming that
the alert is appropriate, does not consider this patient's heart failure diagnosis and proceeds to order

235 | P a g e
the 30cc/kg of normal saline. The patient later develops acute pulmonary edema and has to be
transferred to the intensive care unit.

B. Dr. Smith is about to order antibiotics for his patient who has urinary tract infection. He chooses
ciprofloxacin because that is what he is used to using for his patients, but did not notice that this
patient's previous urine culture grew organisms resistant to ciprofloxacin

C. Dr. Smith needs to discontinue the medication metoprolol for a patient in the EHR. However, he
mistakenly discontinues the medication "metoclopramide," which was located right next to metoprolol
in the EHR and looked very similar.

D. Dr. Smith is about to admit his 10th patient on a busy call day. When placing his admission orders, he
mistakenly signs them in an open chart in the EHR for another patient whom he had seen earlier.

Answer: A. Dr. Smith is alerted by an automatic EHR pop-up alert notifying that his patient may be
developing severe sepsis and includes a button that places an order for 30cc/kg of normal saline.
However, this patient also has congestive heart failure with an ejection fraction of 20%. Dr. Smith,
assuming that the alert is appropriate, does not consider this patient's heart failure diagnosis and
proceeds to order the 30cc/kg of normal saline. The patient later develops acute pulmonary edema and
has to be transferred to the intensive care unit.

Explanation: Automation bias refers to the propensity for humans to be over-reliant on suggestions
from automated decision support systems and ignore contradictory evidence. In this case, the patient,
who has severe congestive heart failure, should not have been given 30cc/kg of fluid, but Dr. Smith
relied on the automated decision support recommendation and gave it anyway, which resulted in an
adverse event.

The other options are incorrect because:

B. This error is an example of the effects of availability heuristic.

C. This error resulted from a difficult to read EHR interface

D. This error is an example of a wrong patient order that can occur when multiple patient charts are
open in the EHR

Reference: Goddard, K., Roudsari, A. & Wyatt, J. C. Automation bias: a systematic review of frequency,
effect mediators, and mitigators. J. Am. Med. Informatics Assoc. 19, 121–127 (2012).

236 | P a g e
S22-4: Which of the following functions would you typically fail to achieve in your design of electronic
order sets?

A. Save time for clinicians for placing orders

B. Promote adherence to clinical practice guidelines

C. Provide clinical decision support

D. Increase the number of billable diagnoses for a clinical encounter

Answer: D. Increase the number of billable diagnoses for a clinical encounter

Explanation: While the challenge of adequately capturing billable diagnoses for clinical encounters may
have informatics solutions, electronic order sets only suggest orders for a given diagnosis or clinical
workflow and typically do not include any suggestions for billable diagnoses.

The other options are incorrect because:

A. order sets can save time for placing orders by having relevant orders be located in one place in the
EHR

B. order sets can be curated to include orders modeled after clinical practice guidelines

C. By including relevant orders for a diagnosis, order sets are a form of clinical decision support

Reference: McGreevey, J. D. Order sets in electronic health records: Principles of good practice. Chest
143, 228–235 (2013).

Detecting post-approval Adverse Drug Events within EHR Data: A case study correlating Lipophilic Beta-
Blockers with the subsequent diagnosis of Parkinson’s Disease.

--Cyril Spiro

S22-5: A group of researchers are studying potential adverse drug reactions (ADR) related to
prescriptions of medications and subsequent diagnosis of a specific disease. The researchers first select
all the ADRs related to the specific disease detected by an ADR detection heuristic. Then the researchers
study the ADRs by looking at their odds ratios. The goal of the researchers is to shortlist the potential
ADRs for a subsequent detailed study. A few of the potential ADRs are related to a specific class of
medications. Previous research has shown that this class of medications, let’s call it “X class,” have a
high chance of interacting with other medications and cause complications in patients. The X class of
medications have also shown conflicting results in previous research studies which were studying the
ADR in question. Assume that a patient can be on X class medication along with other medications in the
potential ADR list and the researchers want to shortlist the remaining ADRs in the list.

237 | P a g e
Which step can the researchers take to most efficiently shortlist the remaining and offset the potential
effects of X class medications on other potential ADRs that were detected?

A. Recalculate the odds ratios of only the remaining potential ADRs for the particular disease in
question by excluding patients that were on X class of medications.

B. Stop the study until a consensus is reached by clinical trials on the ADR related to X class of
medications and the specific disease in question.

C. Continue shortlisting the potential ADRs without any change in the analysis strategy.

D. Start the entire analysis from the beginning by first removing the patients who were on X class of
medications from the data and then running the ADR detection heuristic again on entire available data.

Answer: A. Recalculate the odds ratios of only the remaining potential ADRs for the particular disease in
question by excluding patients that were on X class of medications.

Explanation: The goal of the researchers is to find the potential ADRs for a specific disease by looking at
the data in an EHR database. Considering that the database will have multiple diseases and medications,
the ADR detection heuristic will have to scan through the entire EHR database to detect the potential
ADRs detected in the database. If the researchers want to negate or offset the potential effects of X
class medications on other potential ADRs for a particular disease, it will be inefficient to rescan through
the entire database and recalculate the odds ratios for all ADRs (not just the ones related to the disease
in question). Nor will it be a good strategy to continue without recalculating the odds ratios or to
completely halt the study because of one ADR mixed results. The best strategy here will be to reduce
the computing costs and time delays by only recalculating the odds rations for the ADRs related to the
diseases in question.

Reference: Szumilas M. Explaining Odds Ratios. Journal of the Canadian Academy of Child and
Adolescent Psychiatry. 2010;19(3):227-229.

S22-6: A group of researchers are studying a particular potential adverse drug reaction (ADRs) related to
a particular disease (disease Dx) and a particular medication (medication Rx) found by using data
obtained from an EHR system. The design of the study is that of a retrospective cohort study. On
investigating the data on the detected potential ADR by using statistical tests and running confounding
analysis, the researchers find that all the statistical tests are pointing in the direction of the potential
ADR being an actual ADR. What can the researchers conclude from the results of the statistical tests?

A. There is a causal effect of the prescriptions of medication Rx to patients leading to them developing
the disease Dx. No more analysis or studies are necessary to confirm the findings.

B. There is strong evidence to conclude that there is a correlation between prescription of medication
Rx to patients and the subsequent development of disease Dx in those patients. Conducting clinical
trials is necessary to establish a causation.

238 | P a g e
C. No inference can be made from the statistical tests about either causation and correlation between
drug Rx and disease Dx.

D. There is strong evidence to conclude that there is a correlation between prescription of medication
Rx to patients and the subsequent development of disease Dx in those patients. Running further
statistical tests on the EHR data (observational data) will establish causation.

Answer: B. There is strong evidence to conclude that there is a correlation between prescription of
medication Rx to patients and the subsequent development of disease Dx in those patients. Conducting
clinical trials is necessary to establish a causation.

Explanation: The researchers are conducting statistical tests on observational data using a retrospective
cohort study design. The statistical tests will not be able to establish causation between the prescription
of medication Rx and the subsequent diagnosis of disease Dx because there might be potential
confounders in the study that the researchers cannot control for using the observational data alone. To
establish causation, the ADR will need be studied in a controlled experiment were the effect of the
potential confounders are minimized. The researchers can however check for a significant correlation
between medication Rx and disease Dx using the observational data alone. The results from the
statistical tests on the observational data can then be used as a hypothesis which can be tested with
clinical trials.

Reference: Kawachi I. Injustice at work and health: causation or correlation? Occupational and
Environmental Medicine 2006;63:578-579.

S23: Panel - Closing-the-loop: Transforming Medical Information into Action

--Chancey Christenson, Kenan Katranji, Gargi Schneider

S23-1: What do experts identify as the most challenging aspect of the implementation of
pharmacogenetics into the clinic?

A. Translation of genetic information into clinical action

B. Test cost, test reimbursement or other economic issues
C. Availability of high quality genotyping test (CLIA approved)
D. Electronic medical record use, such as application of CDS
E. Clinician and/or patient resistance

Answer: A. Translation of genetic information into clinical action

Explanation: Survey of pharmacogenomic “experts” (PGRN and ASCPT) on the top 3 challenges to
implementing pharmacogenetics in the clinic, 95% of respondents selected “process required to
239 | P a g e
translate genetic information into clinical actions.” The next 2 responses were genotype test
interpretation (e.g. using genotype information to assign phenotype), and providing recommendations
for selecting the drug/gene pairs to implement.

Reference: Relling MV, Klein TE. CPIC: Clinical Pharmacogenetics Implementation Consortium of the
Pharmacogenomics Research Network. Clin Pharmacol Ther. 2011 Mar;89(3): 464-7.

S23-2: What type of genetic information does the MyCode program at Geisinger use?
A. Whole Gene
B. Proteins
C. Exons
D. Enzymes
E. Introns

Answer: C. Exons

Explanation: The exons are the areas containing information for the creation of proteins and they make
up about 1% of the DNA. With current technology, exon sequencing is more cost-effective than whole
genome sequencing, making it better suited to large scale precision health applications. As of March 1st,
2018, the MyCode program at Geisinger has sequenced 92,816 participants exomes with Regeneron
Genetics Center.

Reference: https://www.geisinger.org/mycode

S23-3: Of the following which is the most challenging aspect of Natural Language Processing?
A. Tokenization
B. Parts of Speech Tagging
C. Dealing with ambiguity
D. Parsing of a sentence

Answer: C. Dealing with ambiguity

Explanation: The challenge with NLP is to assign semantic meaning to language. NLP uses different
knowledge sources and contextual information to derive meaning; however, there is often ambiguity
within sentences that can be difficult to resolve. For example, in an emergency medicine context,
"shock" will have completely different meaning than when the term is used to describe an emotional
response within the context of a psychiatry visit.

Reference: Shortliffe EH. Biomedical Informatics. 4th ed. London: Springer; 2014. 264 p

240 | P a g e
S23-4: Which of the following is most likely to be stored as structured data?
A. Video
B. Images
C. Text
D. Audio

Answer: C. Text

Explanation: Although text can be stored as both structured and unstructured data, the other choices
are generally stored only as unstructured data

References: McCallum A. Information Extraction: Distilling Structured Data from Unstructured Text.
Queueing Syst. 2005 Nov;3(9):4:48–4:57.

S24: Presentations - Patient Deduplication, Clinical Phenotypes, and Collecting Social Determinants of
Health

Developing Methods for Standardized Patient Cohort Deduplication Across the Carolinas Collaborative, A
Regional Data Research Network

--Robert Bradford

S24-1: A Clinical Data Research Network (CDRN) has decided on a hashing algorithm that utilizes: Last
Name, Date of Birth, and Last 4 digits of the Social Security Number (SSN), where Date of Birth and Last
4 digits of the SSN are translated directly into text (with no formatting changes). Site A has the following
data for their patient:
First Name: John
Middle Name: A
Last Name: Smith
Date of Birth: 01/31/53
SSN: 000-0000-7890
State: NC

Which of the following options of data available at Site B would cause the de-duplication process to FAIL
(hashes do not align).

A. SSN: 123-456-7890
B. Middle Name: 'null'
C. First Name: "John A"
D. Date of Birth: 31-JAN-1953
E. State: FL

Answer: D. Date of Birth: 31-JAN-1953

241 | P a g e
Explanation: Option D is the correct choice. While the date values both represent the same date within
their respective databases, when converted into 'Text' the format difference (Site A: MM/DD/YY, Site
B:DD-Mon-YYYY) would produce two different hash values.

Option A shows that Site B has extra information about the patient's SSN, however the hash only
requires the last 4 to be provided. The last 4 digits of the SSN match between Site A and Site B.

Options B, C, and E are not used in the defined hashing method, therefore differences in formatting and
the values themselves will not affect the hash.

Reference: Christen P. Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution,
and Duplicate Detection. New York: Springer; 2012. doi: 10.1007/978-3-642-31164-2

S24-2: One of the methods to success in developing the Carolina's Collaborative was the ability to test
the de-duplication process internally between two different data sources. If another institution were to
join the Collaborative but did not have a legacy EMR to use as an internal gold standard in testing the
de-duplication process, what would be the best validation approach to attempt first?

A. Implement the process without additional checks, using previous validation results from other sites as
justification

B. Use only dummy data as a validation mechanism

C. Use another internal patient-centered data source, such as claims or registry data

D. Obtain an IRB approval and execute a Data Use Agreement with a collaborating site to share patient
data for purposes of testing the de-duplication process

Answer: C. Use another internal patient-centered data source available, such as claims or registry data

Explanation: Utilizing internal data already available is the easiest, safest, and most thorough course
that an institution can pursue when testing data linkage. Most institutions will have an ancillary data
source that contains the same identifiers used in the hashing algorithm. Even though it is not EMR data,
it allows a new process (such as the Collaborative de-duplication process) to be compared to a
previously established and validated linkage to data, such as claims or registry data.

This method avoids the complicated processes for sharing fully identified patient data outside of the
institution, especially when Social Security numbers are included (Option D). Options A and B would be
risky and ill-advised since some level of validation with real data must occur to identify potential
conflicts in data formats between institutions.

Reference: Kho AN, Cashy JP, Jackson KL, Pah AR, Goel S, Boehnke J, et al. Federated queries of clinical
data repositories: the sum of the parts does not equal the whole. J Am Med Inform Assoc. 2015
Sep;22(5):1072-80. doi: 10.1093/jamia/ocv038. Epub 2015 Jun 23.

242 | P a g e
Practical Clinical Phenotype Definitions with SNOMED CT

--DuWayne Willett

S24-3: A health care system wishes to construct diagnosis groupers for a large number of clinical
conditions (clinical phenotypes) to support chronic disease registries and population health initiatives.
To streamline grouper construction and maintenance, they wish to explore ways to build diagnosis
groupers besides a one-dimensional list of terms. Hierarchy-based groupers are possible when using
SNOMED CT because:

A. SNOMED has parent-child ("is a") relationships between concepts that relate specific concepts to
more general concepts

B. SNOMED has attribute relationships between concepts that specify additional defining characteristics
of one of the concepts

C. SNOMED has fewer concepts than ICD

D. Clinical terminologies in EHRs are only mapped to SNOMED, and not to ICD

Answer: A. SNOMED has parent-child ("is a") relationships between concepts that relate specific
concepts to more general concepts

Explanation: The correctAnswer is (A). "Is-a" relationships that link specific concepts to more general
ones enable succinct definition of groupers by referring to a parent and all its descendants with a single
reference to the entire hierarchy.

Attribute relationships that define a SNOMED concept more specifically by referring to other SNOMED
concepts enable more precise definitions, but do not create hierarchical relationships. SNOMED
contains substantially more concepts than the number of ICD-10 codes. Clinical terminologies (e.g. from
IMO or Health Language) used in EHRs are mapped to both ICD codes (for billing) and SNOMED concepts
(for interoperability).

References:

Chute CG. Clinical classification and terminology: some history and current observations. J Am Med
Inform Assoc. 2000;7(3):298-303.

Benson T, Grieve G. Principles of health interoperability: SNOMED CT, HL7, and FHIR. Third edition. ed.
London: Springer; 2016. xxix, 451 pages

S24-4: Compared with groupers listing all individual codes, hierarchy-based groupers:

A. require more frequent updating when new codes or concepts are added or deprecated

B. more closely match clinical thinking about disease subtypes to include or exclude from a grouper
definition

243 | P a g e
C. take more time to construct and maintain

D. are more likely to be missing relevant diagnosis clinical terms

Answer: B. more closely match clinical thinking about disease subtypes to include or exclude from a
grouper definition

Explanation: The hierarchical "is a" (type of) relationships between SNOMED concepts express type-
subtype relationships that match how clinicians think about clinical disorders and their subtypes. This
streamlines clinical vetting of groupers to achieve faithful representation of the intended clinical
condition. Groupers designed with SNOMED hierarchies are more resilient to additions or deprecations
of individual codes/concepts than list-based groupers: addition of new "descendants" within an existing
SNOMED hierarchy does not require a change to the grouper definition. SNOMED hierarchy groupers
require far fewer concepts/codes to define (median of 3 concepts in this report) than groupers using
lists of codes, and thus take less time to construct and maintain. Hierarchy-based groupers
automatically include a complete set of relevant descendant concepts.

References:

Chute CG. Clinical classification and terminology: some history and current observations. J Am Med
Inform Assoc. 2000;7(3):298-303.

Benson T, Grieve G. Principles of health interoperability: SNOMED CT, HL7, and FHIR. Third edition. ed.
London: Springer; 2016. xxix, 451 pages

Supporting Disparities Research by Increasing the Collection of Social Determinants of Health Data
through EHRs

--Lauren Hovey

S24-5: To facilitate the standardized collection of REaL data, which of the minimum data sets provides
an optimal level of granularity?

A. OMB minimum standard

B. HHS race and ethnicity standards

C. CDC Race and Ethnicity Code Set Version 1.0

D. The optimal data standards are those that are fit for purpose and the context in which the data are
collected.

Answer: D. The optimal data standards are those that are fit for purpose and the context in which the
data are collected.

244 | P a g e
Explanation: Providers must balance a number of clinical and administrative responsibilities in collecting
REaL and SOGI data. This includes managing workflow and time constraints, patient relationships, and
navigating the EHR product itself. Most EHRs and health IT products are capable of demographic data
collection; therefore, “the best fit” for granularity of demographic data capture involves a triangulation
of the providers’ priorities for data capture and use, patient needs, and available standards and
products.

Reference: FHIR US Core Implementation Guide. Race and Ethnicity - CDC.

http://hl7.org/fhir/us/core/stu1/CodeSystem-cdcrec.html

Current Measures of Sexual Orientation and Gender Identity in Federal Surveys. August 2016.
https://s3.amazonaws.com/sitesusa/wp-
content/uploads/sites/242/2014/04/WorkingGroupPaper1_CurrentMeasures_08-16.pdf

S24-6: Even when providers and/or health systems are motivated to collect and use REaL/SOGI data for
clinical purposes, their vendor products may not be optimized for the task. Conversely, vendors may
offer robust functionalities that are unintuitive or that providers are not well versed in using. For EHR
products designed to facilitate REaL/SOGI collection and use, which of the following is an optimal
solution to reducing provider burden during information capture?

A. Pre-populated fields related to gender/sex-specific history

B. Free text fields that allow providers to enter relevant information that may not be captured

C. Capture of the information in a pre-assessment (e.g., via portal or registration questionnaire)

D. Structured fields for REaL/SOGI that occur in a consistent place in the EHR

Answer: B. Free text fields that allow providers to enter relevant information that may not be captured

Explanation: (B) Ideally, vendors support providers’ in how to use structured REaL/SOGI fields and
where they appear within the patient record. (A) Products that restrict or pre-populate sex-specific
history, exam, or ordering templates based on a patient’s recorded sex at birth may limit the clinician’s
ability to record and update the patient record (e.g., a transgendered patients’ medical transition history
and current anatomy). (C) Free text fields are useful for difficult-to-code information, but are difficult to
access for future care. (D) The ideal mode and timing of demographic data collection is an open question
as far as how best to encourage patient disclosure; however, the utility of this information depends on
its electronic captured and how effectively it is incorporated into the EHR.

Reference: U.S. Department Of Health And Human Services Implementation Guidance On Data
Collection Standards For Race, Ethnicity, Sex, Primary Language, And Disability Status. Oct 2011.
http://aspe.hhs.gov/datacncl/standards/ACA/4302

245 | P a g e
S25: Presentations - Methods for Enhancing EHR Usability

Categorization of EHR Design Problems Using Usability Heuristics

--Benjamin Duncan

S25-1: The main goal of the heuristic categorization methodology introduced here is intended to
accomplish which of the following?
A) Allows for grouping of the most significant issues in usability and to prioritize them based on their
effect on workflow
B) Make suggestions to users to improve the usability of the overall system
C) Have usability experts review a chosen interface and compare it against accepted usability principles
D) Evaluators work through a series of tasks and ask a set of questions from the perspective of the user.

Answer: A) Allows for grouping of the most significant issues in usability and to prioritize them based on
their effect on workflow

Explanation: In the prior work, a range of user problems was documented and their impact on nursing
workflow characterized [1]. However, compared to heuristic evaluation where a usability expert reviews
an interface and compares usability problems against usability principles, usability categorization allows
for the grouping of usability problems and then categorizes them based off of a set of heuristics.

Reference: Duncan, B., Solomon, A.J., Doebbeling, B.N., Furniss, S.K., Grando, A., Poterack, K, Miksch, T.,
Helmers, R., &Kaufman, D.R. “A Comparative In-Situ Study of Two Medical Reconciliation Interfaces in
Pre-Operative Nurse Assessments. WISH 2017, AMIA 2017.

S25-2: Why should workflow be studied in advance of an enterprise-wide EHR conversion?

A) Make improvements to the current clinical systems being used
B) Perform log file analysis on user interactions to develop a standard workflow for EHR-mediated tasks
C) Understand variation across different clinical sites and systems in the healthcare settings
D) Determine how well clinicians can perform their specific tasks through various EHRs based off
interactive behavior measures and human-computer interaction frameworks.

Answer: C) Understand variation across different clinical sites and systems in the healthcare settings

Explanation: In order to understand workflow in advance of an enterprise-wide workflow, the variation

across not only various clinical sites but the variation across clinicians needs to be understood to
understand bottlenecks in workflow along with sources of variation. This allows for the harmonization of
workflows and the establishment of a unified workflow that considers all clinical sites and their
independent workflows.

246 | P a g e
References: Doebbeling B, Burton M, Kaufman D, Poterack K, McCullough M, Grando A. Integrated
Workflow Capture in an EHR Conversion: Standardizing on Best Practice Methods. American Medical
Informatics Association. 2016;

Disrupt and Adjust: Insights from a Contextual Inquiry of A Novel Documentation and Workflow Tool for
Inpatient Providers

--Subha Airan-Javia

S25-3: Your hospital identifies a safety event in which an immunocompromised patient with recent
history of extended-spectrum beta-lactamase resistant E. coli bacteremia was not started on sufficiently
broad antibiotics when she developed a fever overnight, and effective antibiotics were not instituted
until the error was caught 18 hours later. During root cause analysis, your team identifies that the
patient’s progress notes made clear that the patient should be started on vancomycin, imipenem, and
amikacin in the setting of a new fever, however the written handoff did not include this
information.Which of the following systems would be most likely to prevent this error?

A. Instituting a policy in which all febrile, immunocompromised patients are immediately transferred to
the intensive care unit.

B. An easily accessible documentation template for fever in immunocompromised patients.

C. Educating providers about the importance of situational awareness in the written handoff.

D. Populating the progress note & written handoff from a single source of truth

Answer: D. Populating the progress note & written handoff from a single source of truth

Explanation: In addition to writing and updating admission notes, progress notes, and discharge
summaries, front line clinicians (FLCs) must also update plans of care and problem lists in their handoffs.
Often, these documents do not communicate with each other, forcing FLCs to maintain each document
independently. As length of stay for an individual patient and team census/workload increases, progress
notes and handoffs are known to drift away from each other in content and decrease in overall quality
(1). In the absence of links from information in the electronic health record (EHR) to elements of the
written handoff, inconsistencies are known to occur for the list of problems, code status, medications,
and allergies (2-4). In this scenario, the key piece of information that was lost between documents (a
contingency plan in case this patient developed a fever) is unlikely to be captured discretely in the EHR.
Thus, a unified documentation system that populates both the progress note and the written handoff
from a single source of truth would most likely have avoided this error since the FLC would only have
had to make an update in one location instead of two.

Instituting a policy requiring transfer of febrile, immunocompromised patients to the intensive care unit
would have introduced another handoff and required substantial time to accomplish. The intensive care
unit clinicians would still have needed to have read the most recent progress note in addition to any
transfer notes or the handoff in order to have caught this mistake, which may not be part of their

247 | P a g e
workflow. Thus, this approach is not likely to prevent this error in addition to leading to many
potentially unnecessary intensive care unit admissions.

While a documentation template for fever in immunocompromised patients can provide guidance to
FLCs to include explicit fever plans, the problem in this case is not that the patient had no fever plan.
Rather this patient’s individualized, unusual fever plan was not copied over from the progress note to
the handoff. Thus, a documentation template alone would probably not have prevented this error.

While it is certainly important to educate providers about the importance of situational awareness in
the written handoff, high census, individual patient complexity, and increased length of stay make it
increasingly difficult for FLCs to keep all inpatient documents updated and high quality. Thus, education
alone is unlikely to prevent the error seen in this patient without technological solutions that streamline
workflow and decrease the documentation burden (2).

References:

1. Miller DM, Schapira MM, Visotcky AM, et al. Changes in written sign-out composition across
hospitalization: Handoffs and Written Sign-out Quality. J Hosp Med 2015;10:534–6.
doi:10.1002/jhm.2390

2. Airan-Javia SL, Kogan JR, Smith M, et al. Effects of Education on Interns’ Verbal and Electronic Handoff
Documentation Skills. J Grad Med Educ 2012;4:209–14. doi:10.4300/JGME-D-11-00017.1

3. Aylward MJ, Rogers T, Duane PG. Inaccuracy in Patient Handoffs: Discrepancies between Resident-
Generated Reports and the Medical Record. Minn Med 2011;94:38–41.

4. Derienzo C, Lenfestey R, Horvath M, Goldberg R, Ferranti J. Neonatal intensive care unit handoffs: a
pilot study on core elements and epidemiology of errors. J Perinatol. 2014;34(2):149–152.

S25-4: A healthcare organization aims to increase the number of problems added to the problem list in
the inpatient setting. They design a system that forces clinicians to enter each problem on the problem
list before being able to document the plan related to that problem. Which of the following unintended
consequences is MOST likely?

A. Clinicians will document multiple problems under a single problem heading.

B. Case-mix index will go down, leading to reduced hospital billing.

C. Clinicians will enter fewer problems on the problem list for each patient.

D. The hospital will be at higher risk of Medicare fraud from inaccurate problem lists.

Answer: A. Clinicians will document multiple problems under a single problem heading.

Explanation: In addition to simplifying administrative efforts, quality improvement, and billing, accurate
problem list documentation improves patient outcomes (1). Forcing clinicians to enter a problem on the
problem list before being able to document a plan for that problem, mandatory problem-oriented

248 | P a g e
charting, is the most effective method for encouraging problem list utilization (2). While this approach is
highly effective for this purpose, clinical informaticists should be aware that such rigid structures and
forcing functions may aggravate clinicians due to limited expressivity (3–5). This frustration often leads
clinicians to document multiple problems under a single problem heading in order to gain efficiency
from free-text data entry (6), which can reduce overall data quality.

Mandatory problem-oriented charting increases problem list completeness [Wright, Stanford], which is
unlikely to reduce the number of problems entered per patient or reduce the case-mix index. There is
no evidence that clinicians enter problems that do not exist on the problem list that could lead to
Medicare fraud.

References:

1. Hartung DM, Hunt J, Siemienczuk J, Miller H, Touchette DR. Clinical implications of an accurate
problem list on heart failure treatment. J Gen Intern Med. 2005 Feb;20(2):143–7.

2. Wright A, McCoy AB, Hickman T-TT, Hilaire DS, Borbolla D, Bowes WA, et al. Problem list
completeness in electronic health records: A multi-site study and assessment of success factors. Int J
Med Inf. 2015 Oct;84(10):784–90.

3. Rosenbloom ST, Denny JC, Xu H, Lorenzi N, Stead WW, Johnson KB. Data from clinical notes: a
perspective on the tension between structure and flexible documentation. J Am Med Inform Assoc.
2011 Mar 1;18(2):181–6.

4. Wright A, Maloney FL, Feblowitz JC. Clinician attitudes toward and use of electronic problem lists: a
thematic analysis. BMC Med Inform Decis Mak. 2011;11(1):1.

5. Hodge CM, Narus SP, Hodge C. Electronic problem lists: a thematic analysis of a systematic literature
review to identify aspects critical to success. J Am Med Inform Assoc. 2018.

6. Li RC, Garg T, Cun T, Shieh L, Krishnan G, Fang D, et al. Impact of problem-based charting on the
utilization and accuracy of the electronic problem list. J Am Med Inform Assoc. 2018.

249 | P a g e
S26: Presentations - Workflow Enhancements for the HER

Improving Point of Care Information Workflow with EHR Embedded Search

--Bhrandon Harris

S26-1: On average, what is the number of unanswered clinical questions that clinicians have after a
patient encounter?
A. 0.25 questions per encounter
B. 0.5 questions per encounter
C. 1 question per encounter
D. 2 questions per encounter

Answer: B. 0.5 questions per encounter

Explanation: According to Del Fiol, et al, 2014, clinicians typically have an average of 0.57 (95% CI, 0.38-
0.77) unanswered clinical questions per patient seen. This is based on a meta-analysis of 72 articles
investigating information seeking practices of clinicians with regards to clinical encounters.

Reference: Del Fiol G, Workman TE, Gorman PN. Clinical questions raised by clinicians at the point of
care: a systematic review. JAMA internal medicine. 2014 May 1;174(5):710-8.

S26-2: What are the two most commonly cited reasons that clinicians do not pursue answers to clinical
questions that have arisen during an encounter?

A. Lack of time and forgetting the question

B. Forgetting the question and a lack of urgency of the question
C. Lack of time and a lack of urgency of the question
D. Lack of time and doubt that an answer exists
E. Forgetting the question and viewing question as unimportant

Answer: D. Lack of time and doubt that an answer exists

Explanation: While clinicians typically have an average of 0.57 (95% CI, 0.38-0.77) unanswered clinical
questions per patient seen, about 50% of those questions are never pursued. The most common reason
cited in the literature is lack of time followed by doubt that an answer exists. Of the 50% of clinical
questions that are pursued, clinicians are able to find an answer in 78% (67%-88%). However, this means
that a majority of clinical questions that exist after patient encounters are not pursued or resolved.

Reference: Del Fiol G, Workman TE, Gorman PN. Clinical questions raised by clinicians at the point of
care: a systematic review. JAMA internal medicine. 2014 May 1;174(5):710-8.

250 | P a g e
Integration of Patient Controlled Analgesic Infusion Forms into the Electronic Medical Record (the eAIF
Project).

--MaryAnn Connor

S26-3: Opioids are a class of medication that confer great and immediate benefit but also carry
significant immediate and long-term risks. As pain increases and doses escalate, side effects ensue, and
an opioid rotation becomes necessary. What is the best method that ensures opioid conversion and
rotation are done safely?

a. Converting between opioids by hand calculation can be done in a standardized manner by clinicians
b. Electronic calculations performed using dedicated smartphone applications, or online resources
c. Use of a consistent conversion algorithm, and automated calculation embedded within the
electronic medical record.
d. Conversion first to morphine, a universal opioid, and then secondary conversion to another opioid.

Answer: c. Use of a consistent conversion algorithm, and automated calculation embedded within the
electronic medical record.

Explanation: While converting between opioids by hand calculation can be done in a standardized
manner by clinicians, it may cause errors, potentially leading to under- or over- dosing of patients.
Electronic calculations performed using dedicated smartphone applications, or online resources are
more accurate, but each clinician may use a different resource, all of which may employ formulas or
conversion ratios Linking opioid conversion calculations directly to the EMR permits greater
standardization, accuracy, and control over opioid rotations. All doses administered are accurately
counted, and the algorithm used throughout the hospital can be consistently applied.

References: Durham. M. et al (2017). Addressing safe opioid monitoring practices using an

interprofessional approach. JONA, 47(11), 537-544.

Reis-Pina, P., Lawlor, P., & Barbosa, A. (2017). Adequacy of cancer-related pain management and
predictors of undertreatment at referral to a pain clinic. Journal of Pain Research, 10, 2017-2107.

S26-4: The Rapid Application Development (RAD) model should be used in software development:
a. To address gaps by disruption to remove the status quo legacy system and build from the ground up
focusing only on fundamentals.
b. Only when you are unable to use the preferred approach of Structured Systems Analysis and Design
Method (SSADM).
c. When there is limited time and availability for users (e.g., physicians and nurses) to provide input in
the design.
d. To allow users to interact with systems analysts and develop models and prototypes that represent
all system processes, inputs, and outputs using a multidisciplinary team with the model changing as
stakeholders weigh in.
251 | P a g e
Answer: d. To allow users to interact with systems analysts and develop models and prototypes that
represent all system processes, inputs, and outputs using a multidisciplinary team with the model
changing as stakeholders weigh in.

Explanation: SSADM covers those aspects of the life-cycle of a system from the feasibility study stage to
the production of a physical design; it is generally used in conjunction with other methods, such as
PRINCE, which is concerned with the broader aspects of project management. SSADM sets out a cascade
or waterfall view of systems development, in which there are a series of steps, each of which leads to
the next step.

The Rapid Application Development (or RAD) model is based on prototyping and iterative model with no
(or less) specific planning. In general, RAD approach to software development means putting lesser
emphasis on planning tasks and more emphasis on development and coming up with a prototype. In
disparity to the waterfall model, which emphasizes meticulous specification and planning, the RAD
approach means building on continuously evolving requirements, as more and more learnings are drawn
as the development progresses.

Reference: McGonigle, D and Mastrian, K. (2012). Nursing Informatics and the Foundation of
Knowledge. Burlington, MA. Jones and Bartlett Learning Witten, J. L., & Bentley, L. D. (2007). Systems
Analysis & Design Methods. New York, NY: McGraw-Hill Companies, Inc.

Preoperative Nursing Workflow and Problem Solving

--Lu Zheng

S26-5: A health care system wishes to reduce the cognitive burden of clinicians working in preoperative
care to ensure the quality of patient-care delivery and enhance the patient safety. However, the
clinicians are not typically trained in human factors or cognitive science. They may not be able to
pinpoint the sources of burden. Which method is best suited to characterize cognitive burden in clinical
work?
A. Semi-structured interview
B. Video-screen capture of the EHR use
C. Ethnographic study of workflow
D. Think-aloud protocol
E. A multi-method approach that triangulates evidence from different sources.

Answer: E. A multi-method approach that triangulates evidence from different sources.

Explanation: Cognitive informatics looks at the mental process and decision-making method when a
clinician uses the electronic health record system. The cognitive burden appears when the workflow or
environment necessitates that clinician has to commit information to memory, perform duplicate work,
and engage in frequent multitasking. It is difficult to capture that information in a survey. The screen
capture of the EHR use can partially reflect the complexity of interacting with a difficult system, but may

252 | P a g e
not fully reflect the clinicians’ personal mental burden. The Think-Aloud protocol inspires the individual
to express what’s on their mind as they encounter particular difficulties. Ethnography of workflow
allows researchers to trace the trajectory of problems, for example, the bottleneck in communications,
difficult interfaces and unmet information needs. The combination of methods can best identify and
characterize the sources of cognitive burden and can be used to fashion solutions.

Reference: Grando A, Groat D, Furniss SK, Nowak J, Gaines R, Kaufman, DR, Poterack K, Miksch T,
Helmers R. Using Process Mining Techniques to Study Workflows in a Pre-operative Setting. AMIA
Annual Symp 2017, November 4-8, 2017.

S26-6: The PreOp nurse needs to help coordinate the activities of several surgery team members, such
as the anesthesiologist, the resident, the physician assistant or the surgeon, when they stop by and talk
to the patient during the PreOp assessment process. The nurse needs to seamlessly and efficiently
communicate to clinicians that some action is needed before the patient can be transferred to the OR.
Which strategy do you think is the best way to handle this coordination process with minimal disruption
given the constraints of the 30 minutes Preop assessment?

A. Write it on the blue sheet (checklist) and provide access to all clinicians.
B. Use a symbolic system (e.g., color coded magnetic strips on the exterior of the patient room)
that signals to team members that certain actions are needed.
C. Call, email or text the team member before they come to see the patient, so that they will know
before they enter the room.
D. Wait until they come and talk to the member directly at the patient bedside.

Answer: B. Use a symbolic system (e.g., color coded magnetic strips on the exterior of the patient room)
that signals to team members that certain actions are needed.

Explanation: Coordination in a short period of time given a multiplicity of patient needs is challenging.
Reaching the right person without extraneous interruptions and actions needs a coordinating strategy.
Incorporating the artifacts and tools in the room environment to remind people is straight-forward,
requires minimal effort and is less disruptive than the other methods.

Reference: Furniss SK, Burton MM, Larson DW, Kaufman DR. Modeling Patient-Centered Cognitive Work
for High-Value Care Goals. Proceedings International Symposium on Human Factors and Ergonomics in
Health Care 2016 Jun (Vol. 5, No. 1, pp. 112-119). Sage India: New Delhi, India: SAGE Publications.

S27: Panel - Clinical Informatics Efforts to Combat the Opioid Crisis – Experiences from the Front Line

Barry Blumenfeld, Patrick Burns, James Huizenga, Steven Labkoff, Despina-Maria Michaels

S27-1: What major event transpired in 2016 that had a material effect on how primary care doctors
think about prescribing opioid analgesics:

253 | P a g e
A. The Trump Administration published a new law on the limits on daily opioid MMEs

B. The CDC published guidelines to inform primary care and other clinicians on the most appropriate
way to prescribe opioids

C. The FDA provided new guidance to prescribe abuse deterrent opioids whenever possible

D. A new formulation of an abuse deterrent opioid was approved for sale and use in the US

Answer: B. The CDC published guidelines to inform primary care and other clinicians on the most
appropriate way to prescribe opioids

Explanation: In March 2016, the CDC announced is Guideline for Opioid use for primary care clinicians.
This provided 12 specific action points within the guideline for best practices in writing for opioid
analgesics.

A is incorrect. While the Trump Administration is attempting to deal with the opioid epidemic in a
number of ways, no new laws were passed in 2016 concerning the opioid epidemic. The Christi
Commission was empaneled in late 2017 to come up with recommendations concerning ways that the
US government can help with the opioid epidemic.

C is incorrect. The first abuse deterrent opioid was approved for use in the US in 2010. However, the
FDA has never issued guidance requiring the use of such formulations. That choice remains with the
clinician.

D is incorrect. While the first abuse deterrent opioid was approved in 2010, others have followed since.
The latest abuse deterrent opioid analgesic was approved in early 2018.

Reference: https://www.cdc.gov/drugoverdose/prescribing/guideline.html

S27-2: With regard to the opioid crisis, the use of big data analytics is currently in use to:

A. Identify clinicians who are appropriately writing opioid prescriptions in order to single them out as
good practice examples

B. Identify patients at risk for opioid overdose by use of a predictive model

C. Enhance the Prescription Drug Monitoring Programs (PDMP’s), which is a requirement in all states

D. Support integrated systems, which are now wide-spread in Health Information Exchanges across the
US to help deter inappropriate opioid prescribing

Answer: B. Identify patients at risk for opioid overdose by use of a predictive model

Explanation: A is incorrect: There are predictive models currently in use today to identify doctors (and
patients) who are inappropriately prescribing opioids in a way that could put the patient at risk. Some of
those issues are concomitant use with benzodiazepines or muscle relaxants.

254 | P a g e
B is correct: There are at least two companies who are marketing and implementing solutions that
identify patients at risk for opioid overdose during the prescribing moment inside the EHR.

C is incorrect: The advent of PDMPs started about 10-15 years ago. While they are tools in 49 of 50
states, they are not legally required to be used in all jurisdictions.

D is incorrect: Only a handful of IDNs and ACOs currently are leveraging predictive analytics for dealing
with the opioid crisis, but the number is growing.

References: www.ApprissHealth.com & www.PrincipledStrategies.com

S27-3: With regard to the Opioid Crisis, Clinical Decision Support tools are

A. Currently in wide scale use to deal with the opioid crisis

B. Currently not showing significant impact in cutting down the rate of opioid prescribing

C. Can bring together natural triggers inside the clinical encounter to present in-the-moment guidance

D. Generally easy to implement across a wide swath of the US due, in part, to the adoption of EHRs in
the last 5 years.

Answer: C. Can bring together natural triggers inside the clinical encounter to present in-the-moment
guidance on everything from prescribing guidelines to dosage recommendations

Explanation: A is incorrect: Though CDS has been shown to have a positive effect on things like opioid
prescribing, these tools are not yet deployed at large scale across the US at this time, though that is
changing.

B is incorrect: There are at least two recent studies, Univ of Penn and Purdue Pharma/Practice Fusion
that have shown good results of CDS implementations on curbing the use of opioids in Chronic and
Acute pain situations.

C is correct: CDS tools are beginning to be deployed in EHRs across the US. They are being shown to
have a significant impact on how doctors prescribe opioids

D is incorrect, though EHR adoption is now over 95% hospitals and over 85% in the outpatient setting,
implementing CDS in large health systems remains difficult because systems vary so greatly. As there is
currently limited uptake of CDS standards in EHRs, a write once/use many approach to CDS remains
elusive in most of the US. Though EHRs that rely on individual implementations can and do implement
CDS, getting the same tools uniformly deployed remains a challenge.

255 | P a g e
S27-4: The opioid crisis presents challenges for the entire healthcare ecosystem. Specifically, the use of
informatics tools with regards to the role of primary care clinicians should be to:

A. Do primary research to identify informatics tools that can be used to curb the opioid crisis
B. Provide specific data on every patient that has an encounter to ensure that registries and databases
have adequate information to draw upon when making policy decisions around opioid use
C. Devote a substantial part of their practice to work with research efforts from organizations like
SAMHSA, NIDA, and APA to help curb the opioid crisis
D. Leverage as many existing tools as is practical from their HER vendors to integrate appropriate
screening and documentation during routine clinical care

Answer: D. Leverage as many existing tools as is practical from their HER vendors to integrate
appropriate screening and documentation during routine clinical care

Explanation:

A is incorrect: While it would be wonderful if every clinician were to take part in clinical research, it is
impractical for most PCPs to work on research when seeing their patient panels. An occasional practice
might want to do some research, but it is unrealistic for most PCPs to create and research new
informatics tools.

B is incorrect: While the accumulation of specific data for research purposes is critical to successfully
fighting the opioid crisis, specifically using the concept of a registry in most practices remains enticing,
but not yet practical.

C is incorrect: While working with the likes of NIDA and SAMHSA is appealing, most primary care
clinicians simply do not have the time to devote to this research.

D is correct: From a practical perspective, leveraging a practices EHR infrastructure through the use of
order sets, guidelines, and clinical decision support, a local practice can and should leverage all the tools
that it has at its disposal – such as PDMP integration, predictive models, and EHR-created order sets in
the course of their clinical practice to ensure safe and effective use of opioids.

256 | P a g e
S28: Presentations - The Learning Health System, Optimization and Leveraging Technology - All to
Improve Care

Implementation of a Learning Healthcare System for Sickle Cell Disease

-- Erin Crowgey

S28-1: A division within a health care system wishes to improve the meaningful use of data entered into
the electronic healthcare records (EHR). The majority of EHR systems handle coding and liability issues
well, but are not robust for clinical research and data visualization at the bedside. Which of the
following options is technologically possible to enhance the downstream use and analysis of data
entered directly into the EHR?

A. Have providers enter more data via free text in note sections.

B. Encourage providers to work independently on improving data entry into EHR.

C. Create a team consisting of providers and health informaticians to implement a process that
supports structured data entry at the bedside.

D. Request that the health informatics department implement solutions per each individual clinician.

E. Enhance the use of IC-9/10 coding for data analysis techniques.

Answer: C. Create a team consisting of providers and health informaticians to implement a process that
supports structured data entry at the bedside.

Explanation: A multi-disciplinary team that bridges the gaps between clinical data entry and
downstream use of EHR data, is required to improve the meaningful use of EHR data. Clinicians and
health informaticians need to work in collaboration to improve clinical workflows. Structured data entry
at the bedside is required for robust downstream analysis.

Reference:. Bush RA, Kuelbs C, Ryu J, Jiang W, Chiang G. Structured Data Entry in the Electronic Medical
Record: Perspectives of Pediatric Specialty Physicians and Surgeons. J Med Syst. 2017 May;41(5):75. doi:
10.1007/s10916-017-0716-5. Epub 2017 Mar 21

S28-2: A common data model (CDM) enables a systematic analysis of disparate databases. The
Observational Medical Outcomes Partnership (OMOP) is one type of CDM that transforms data
contained within EHR databases into a common format, or data model, with common representation of
the data elements, terminologies, vocabularies, and coding schemes. Which of the following options is
solution for a department looking to enhance the meaningful use of EHR data?

A. Recommend providers focus on analytical techniques that can only be deployed directly in EHR
system and that are independent of any data transformation processes.

B. Create a custom CDM, or use a publicly available model, to map EHR data into a standardized
format.

257 | P a g e
C. Require health informaticians to analyze EHR independently and provide PDF reports to providers.

D. Have providers within a department create their own individual lists of important data elements, and
capture them inside excel.

E. Recommend to the clinicians within the department that this is a health informatics issue and does
not require their effort or input.

Answer: B. Create a custom CDM, or use a publicly available model, to map EHR data into a standardized
format.

Explanation: Common data models provide a mechanism of standardizing data entry at the bedside.
Expert clinicians across institutions should decide on key data elements required to improve patient
care, and work closely with informaticians for transforming their data dictionary into a CDM.

Reference: Maryam Garza, Guilherme Del Fiol, Jessica Tenenbaum, Anita Walden, Meredith Nahm
Zozus. Evaluating common data models for use with a longitudinal community registry. Journal of
Biomedical Informatics. Volume 64, December 2016, Pages 333-341.
https://doi.org/10.1016/j.jbi.2016.10.016

The Veterans Health Administration’s Reproductive Health Project

--Caitlin Cusack

S28-3: Clinical Decision Support Systems (CDS) within Electronic Health Records (EHR) to reduce
medication errors are widely used. Medication CDS systems range from basic systems that conduct
drug-allergy, drug-interaction, and duplicate drug checking. More advanced systems incorporate data
from the EHR such as conditions, laboratory results, and gender. CDS that extends to drug-pregnancy
checking have limited clinical usefulness because of the lack of specificity of these alerts. One practice
used with these alerts that does not improve their clinical relevancy is:

a. Remove from the cohort for rule-checking those women who have system evidence an inability to
conceive, such as those with coding consistent with hysterectomy or menopause.

b. Limit alerts for women using highly effective contraceptive or those who are unable to conceive with
their current partner.

c. Build a rules engine that utilizes medications classified with the former FDA categories of C, D, or X.

d. Capture as discrete data pregnancy status including “yes,” “no” or “unknown”.

Answer: c. Build a rules engine that utilizes medications classified with the former FDA categories of C,
D, or X.

Explanation: The FDA moved away from pregnancy categories due to the failure of the older lettering
system to correlate with available evidence regarding the safety of these medications in pregnancy.

258 | P a g e
With many medications classified as being less safe in pregnancy than research has shown, relying on
the old classification system results in numerous clinically irrelevant alerts. (a), (b) and (d) are incorrect
because all help to narrow the cohort of women these rules are run against. Many EHRs do not capture
pregnancy status as discrete data and thus most systems that conduct drug-pregnancy testing do so
against the cohort of women of reproductive age. By capturing pregnancy status and ability to conceive,
along with identifying those women using highly effective forms of contraceptive, the cohort is
narrowed allowing greater clinical relevance of alerts.

References: Schwarz EB, Parisi SM, Handler SM, et al. Clinical decision support to promote safe
prescribing to women of reproductive age: a cluster-randomized trial. Journal of general internal
medicine. 2012 Jul 1;27(7):831-8.

Kuperman, Gilad J., et al. "Medication-related clinical decision support in computerized provider order
entry systems: a review." Journal of the American Medical Informatics Association 14.1 (2007): 29-40.

S28-4: At the Veteran’s Health Administration (VHA), the Reproductive Health project uses logic on the
backend to classify women as pregnant, medically unable to conceive, low likelihood of becoming
pregnant, high likelihood of becoming pregnant, and unknown status. The due date of the VHA’s
Reproductive Health (RH) Clinical Reminder Dialog is based on these classifications. Which of the
following women is not due now to have the RH dialog completed by her physician?

a. Ms. Smith age 38 is new to the VHA, uses an IUD for contraception, and presents for her first
preventive care visit.

b. Ms. Jones has a recorded estimated due date (EDD) in the system from 4 weeks ago, but has not been
seen within the VHA during her pregnancy.

c. Ms. Noonan has a history of a hysterectomy that has been recorded in the system.

d. Ms. Johnson has a completed RH dialog from 2 months ago. She is using Depo-Provera for
contraception, but is late for her next injection.

Answer: c. Ms. Noonan has a history of a hysterectomy that has been recorded in the system

References:

Schwarz EB, Postlethwaite DA, Hung Y-Y et al. Documentation of contraception and pregnancy when
prescribing potentially teratogenic medications for reproductive-age women. Ann Intern Med.
2007;147(6):370-376.

Schwarz EB, Longo LS, Zhao X, etal. Provision of potentially teratogenic medications to female veterans
of childbearing age. Medical care. 2010 Sep 1:834-42.

259 | P a g e
Leveraging Technology to Reduce Unnecessary Post Procedure Office Visits for Chemo Port Placement

--Christine VanZandbergen

S28-5: A provider wishes to replace a post-procedure in-person follow-up visit with a virtual visit.
Assuming the technology is in place to support virtual workflow, what is the most appropriate next
step?

A. Tell all patients to register for a patient portal account and start sending their wound sites photos.

B. Engage the appropriate clinical, regulatory and information systems partners to validate the
workflow and ensure appropriate support/monitoring.

C. Tell patients to take a photo of their wound and text it to you

D. Set up the visits as billable encounters

Answer: B. Engage the appropriate clinical, regulatory and information systems partners to validate the
workflow and ensure appropriate support/monitoring.

Explanation: Post-procedure in-person follow-up visits may be able to be replaced by virtual visits
provided the right infrastructure is in place. A critical success factor is ensuring the workflow has been
fully defined and vetted with appropriate stakeholders to support these initiatives.

Reference: Watkins, M. Making Virtual Teams Work: Ten Basic Principals. Harvard Business Review,
June 27, 2013

S28-6: Which of the following is a potential benefit of replacing post-procedure follow-up visits with
virtual visits?

A. Increased revenue through new billable services

B. Decreased infection rates

C. Improved follow-up with post-procedure wound checks

D. Increased provider satisfaction

Answer: C. Improved follow-up with post-procedure wound checks

Explanation: There are several benefits of using virtual visits to evaluate patients post port insertion.
These benefits include increased compliance with follow up visits, patient time savings and increased
availability of clinic appointments for other patients.

260 | P a g e
Reference: Palen et al. Comparing virtual consults to traditional consults using an electronic health
record: an observational case-control study. BMC Medical Informatics and Decision Making, 2012, 12:65

S29: Presentations - Translating Clinical Decision Support to Practice

Translating evidence into practice: Effect of an Evidence-based Electronic Clinical Decision Support Tool
on Provider Ordering for Venous Catheters

--Emilia Flores

S29-1: The 5 Rights for clinical decision support (CDS) interventions (right information, to the right
person, in the right format, through the right channel, at the right time in the workflow) are a
framework for designing and deploying CDS interventions. Catheter-associated urinary tract infection
(CAUTI) is a common device-associated infection in the acute care setting. However, misdiagnosis of
catheter-associated asymptomatic bacteriuria as CAUTI can lead to inappropriate use of antibiotics and
related consequences, including medication adverse effects and antibiotic resistance. Appropriate urine
culturing is dependent on a clinician’s ability to recognize a specific set of symptoms. You notice an
increase in inappropriate urine culture ordering in your organization and wish to implement a CDS to
ensure that urine culturing is aligned with evidence-based recommendations. Which CDS type would be
the most appropriate for this clinical issue?

A. A predictive analytics tool to identify patients at high risk for CAUTI

B. A report containing a summary of patients that meet culturing criteria for CAUTI is available in a
reporting module or tab and is accessible to all clinicians

C. An order set that leads the provider through a series of questions based on CAUTI culturing criteria
and provides recommendations based on provider responses

D. Order set that defaults to a reflex-urine culture order if the patient has a urinary catheter. A reflex
urine culture order is one in which the lab will first conduct a urine analysis and if criteria are met (e.g.
urine analysis with >= 10-20 white blood cell count/hpf), the lab will then perform a urine culture to
assess for CAUTI.

Answer: D. Order set that defaults to a reflex-urine culture order if the patient has a urinary catheter. A
reflex urine culture order is one in which the lab will first conduct a urine analysis and if criteria are met
(e.g. urine analysis with >= 10-20 white blood cell count/hpf), the lab will then perform a urine culture to
assess for CAUTI.

Explanation: In this case, the decision support is presented to the right person (ordering provider), is
embedded within provider workflow (point of order entry) and uses objective criteria to assess whether
a urine culture should be performed. By using patient data (presence of a urinary catheter) to default to
a reflex urine culture order, this design removes the cognitive burden from the provider in having to
remember appropriate culturing criteria. By moving the culturing decision to laboratory services, this
further reduces the possibility of inappropriate culturing.

261 | P a g e
Option A is a possible option, but this information doesn’t necessarily use evidenced-based information
for culturing. Risk information for CAUTI might also be difficult for providers to interpret.

Option B is a possible option, but this report isn’t embedded in a provider workflow for assessing this
patient.

Option C is a possible option, but this relies on a provider to enter patient data into the order entry
screen.

References:

1) Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, Spurr C, Khorasani R, Tanasijevic M,
Middleton B. Ten commandments for effective clinical decision support: making the practice of
evidence-based medicine a reality. Journal of the American Medical Informatics Association.
2003;10(6):523-30.

2) Osheroff JA. Improving outcomes with clinical decision support: an implementer's guide. HIMSS.

S29-2: Zhang and Walji (2011) have proposed 14 evidence-based interface design principles that are
important to consider for any electronic health record (EHR) embedded clinical decision support (CDS),
as they can have a significant impact on user acceptance and satisfaction. (Middleton et al., 2013) Which
of the following interface features is an example of a design principle?

A. A CDS tool to promote antibiotic stewardship asks users to manually type in the patient diagnosis
associated with a medication order rather than select from a drop-down list

B. A CDS tool for vascular access selection displays relevant patient information and calculations
within the CDS interface to aid in decision making, thus avoiding having the user reference and
remember information from other screens.

C. A CDS development group organizes and displays data entry fields in all CDS tools alphabetically

D. A CDS tool for vascular access selection utilizes drop down lists rather than check boxes for the five
most common medications infused through venous access devices

Answer: B. A CDS tool for vascular access selection displays relevant patient information and
calculations within the CDS interface to aid in decision making, thus avoiding having the user reference
and remember information from other screens.

Explanation: Research has shown that short-term memory is limited (Miller, 1956). Reduce potential
errors, completion time, and improve efficiency by avoiding having users remember information from
other screens or having them perform calculations. Design Principle #5 [Memory]

Option A: this design relies on users having to remember diagnosis. Design principle #5 [Memory] states
that designs should minimize the amount of information that users have to recall from memory. A
better design would be to list potential patient diagnoses in a drop-down list.

262 | P a g e
Option C: Design principle # 3 [Match] states that the system should match user workflow in the real-
world setting and displaying information in an alphabetical order does not necessarily match the real-
world cognitive and decision-making process. A better design approach would be to understand user
workflow and the decision-making process and order fields (whether data entry or displayed patient
information) using this structure.

Option D: Design principle # 7 [Flexibility and efficiency] states that designs should use the fewest
number of steps to complete the task. In this case, since there are few predefined medications, using
checkboxes verses drop-down lists results in fewer clicks for the user.

References:

1) Middleton B, Bloomrosen M, Dente MA, Hashmat B, Koppel R, Overhage JM, Payne TH, Rosenbloom
ST, Weaver C, Zhang J. Enhancing patient safety and quality of care by improving the usability of
electronic health record systems: recommendations from AMIA. Journal of the American Medical
Informatics Association. 2013;20(e1):e2-8.

2) Zhang J, Walji MF. TURF: Toward a unified framework of EHR usability. Journal of Biomedical
Informatics. 2011;44(6):1056-67.

Clinical Pathway Regret Risk Differences Between Affluent and Underserved Populations

--Robert Aarhus

S29-3: Decision Regret is a well-studied patient-centered outcome measuring the quality of the medical
decision-making process. Using validated query instruments, researchers have determined key
characteristics of decision regret in attempts to understand, retrospectively, the origin and trajectory of
patient dissatisfaction with those decisions that have produced disappointing or unexpected healthcare
results. In particular, many randomized controlled trials have examined effects of enhanced patient
information, or conformity to the level of physician participation in the process expected by patients.
According to the literature, which of the following is a characteristic of treatment decision regret?

A. It is negatively correlated with decisional conflict

B. It often increases over time in breast cancer cases

C. It sometimes differs according to the patient's race and insurance status

D. It is uniformly decreased through the use of decision aids

E. It is seen in a majority of complex cancer cases

Answer: C. It sometimes differs according to the patient's race and insurance status

Explanation: Decision regret is a complex, post-decision cognitive process with many facets that must
be taken into consideration if a predictive model of regret is to be constructed. Decisional conflict -- a
pre-decisional state of lack of decision clarity because of too little, too much, or improperly presented

263 | P a g e
alternatives, to name a just few causes -- is positively correlated with decision regret, and is largely a
predictor of the degree to which patients are dissatisfied with outcomes. The stability of decision regret
over time has been examined and, in the case of breast cancer specifically, determined not to change
longitudinally for most patients. This stands in contrast to disease processes such as prostate cancer,
where various studies have indicated that decision regret increases over time, an important factor which
must be accounted for in the model. Decision regret has been shown to differ according to race and
insurance status, suggesting the need for a deeper look into the institutional systems to identify the
origin of these effects. Many decision aids have been evaluated in randomized controlled trials, and
while some have demonstrated a reduction in decision regret, other studies have been equivocal, or
actually show an increase in patient dissatisfaction. While decision regret can be pronounced in certain
disease processes like prostate and head and neck cancers, overall the frequency of decision regret is
fairly low, often not exceeding a quarter of all patients evaluated.

References:

1. Hu JC, Kwan L, Krupski TL, Anger JT, Maliski SL, Connor S, et al. Determinants of Treatment Regret in
Low-Income, Uninsured Men With Prostate Cancer. Urology. 2008;72(6):1274–9.

2. Collingwood SA, McBride RB, Leapman M, Hobbs AR, Kwon YS, Stensland KD, et al. Decisional regret
after robotic-assisted laparoscopic prostatectomy is higher in African American men. Urol Oncol Semin
Orig Investig. 2014;32(4):419–25.

3. Brehaut JC, O’Connor AM, Wood TJ, Hack TF, Siminoff L, Gordon E, et al. Validation of a decision
regret scale. Med Decis Mak. 2003;23(4):281–92.

4. Martinez KA, Li Y, Resnicow K, Graff JJ, Hamilton AS, Hawley ST. Decision Regret following Treatment
for Localized Breast Cancer: Is Regret Stable Over Time? Med Decis Mak. 2015;35(4):446–57.

S29-4: To prospectively assess the likelihood of future regret, a model has been developed to include
estimates of regret risk. A novel term in this context, regret risk is a statement of the likelihood of
future decision regret driven by both differences between patient expectation and outcome, as well as
the curse of the counterfactual narrative -- a seemingly certain and more positive alternative future that
might have happened had past choices been different.Key assumptions, based on decision regret
research, have driven regret risk formulation and expression. According to the presented model's
hypothesis, which of the following could explain differences in long-term cancer regret risk between
socioeconomic groups?

A. An emphasis on aleatory rather than epistemic uncertainty

B. A change in action space

C. Linear dependence on measurable decision regret

D. Cultural differences between patients and health care providers

Answer: B. A change in action space

264 | P a g e
Explanation: Keeping in mind the aphorism "all models are wrong; some models are useful," the art and
science of model building is largely an exercise in deciding what and how to represent certain
phenomena, while deliberately choosing to exclude others. The roles of aleatory (random) and
epistemic (knowledge-based) uncertainty are crucial to auguring future outcomes and anticipating
potential decision regret; the model includes both factors, but makes no claim as to which
predominates. The hypothesis that is to be explored by the model is the change in action space; when
certain treatment pathways (such as robotic surgery) are not available to populations because of cost or
geographic location, alternatives may not yield results that are as consistent, and therefore increase the
likelihood of regret. Few real-world patient utility models are linear, and no such assumption is required
(or needed) by the model. The model does not attempt to evaluate cultural factors of any stakeholder;
if such differences do exist, they would have to be ascertained by a much different approach.

References:

1. Engelhardt EG, Pieterse AH, Han PKJ, van Duijn-Bakker N, Cluitmans F, Maartense E, et al. Disclosing
the Uncertainty Associated with Prognostic Estimates in Breast Cancer: Current Practices and Patients’
Perceptions of Uncertainty. Med Decis Mak 2017;37(3):179-192.

2. Brehaut JC, O’Connor AM, Wood TJ, Hack TF, Siminoff L, Gordon E, et al. Validation of a decision
regret scale. Med Decis Mak. 2003;23(4):281–92.

S30: Panel - The Use, Usability, and Experience of Implementing a Systemwide Inpatient Portal (IPP)

--Jennifer Hefner, Timothy Huerta, Ann McAlearney, Daniel Walker

S30-1: When implementing an inpatient portal that includes secure messaging between care team
members and patients, which of the following should be included in front-line staff training?

A. Allow care team members to engage with the tool from the perspective of the patient.

B. Inform care team members that this is the most effective way to communicate with patients now and
the health system will be using it exclusively.

C. Provide them with a training manual to read during breaks.

D. Frontline staff will not be using the tool as patients do so they do not need training in this manner.

Answer: A. Allow care team members to engage with the tool from the perspective of the patient.

Explanation: Using a qualitative approach, we found evidence that successful implementation of an

inpatient portal requires training of front-line staff and nurse managers in a manner that is different
from other HIT implementations. Specifically, this finds that expanded training should allow providers to
engage with the tool from the perspective of patients, in addition to teaching both care team members
and patients about the provider-facing, components in order to fully support optimal collaborative use.
These training components are not included in the current best practices for training in EHRs and are a
new consideration for this type of collaborative tool.

265 | P a g e
Reference: McAlearney, AS, Sieck, CJ, Hefner, JL, Aldrich, AM, Walker, DM, Rize,r MK, Moffatt-Bruce, SD,
Huerta, TR. 2016. “High Touch and High Tech (HT2): Transforming Patient Engagement Throughout the
Continuum of Care by Engaging Patients with Portal Technology at the Bedside.” JMIR Research
Protocols. 2016; 5(4):e221 http://www.researchprotocols.org/2016/4/e221 doi: 10.2196/resprot.6355

S30-2: Studying patient and provider use of an inpatient portal requires analysis of log files - server-side
records of actions taken by a user - to assess how, when, and which features of the inpatient portal
were used. Which of the following is not an issue to consider when analyzing log files?

A. How to find the full-text of the electronic messages sent between patients and the care team.

B. The application may create administrative data (e.g., pushed data to ensure that the tablet does not
display stale data) that is unrelated to patient action, resulting in artifacts in the data.

C. The application itself does not offer an indicator of a Session and as such the research team must
construct an operational definition to represent a patient’s single continuous use of the tablet for the
purpose of using MCB.

D. It may take several days for a patient to get their tablet, and as such length of stay may suffer from
distortions associated with provisioning. As a result, patient length of stay (LOS) may not be an
appropriate framing for use, because of real-world demand on provisioning practices.

Answer: A. How to find the full-text of the electronic messages sent between patients and the care
team.

Explanation: The full-text of messages are recorded in the audit log files. While this is a variable within
the log files, there are IRB considerations. A full IRB application will be necessary to view the full-text as
these messages often contain PHI. Answer choices B-D however are issues that need to be considered
and will be discussed during the panel presentation.

Reference: McAlearney, AS, Sieck, CJ, Hefner, JL, Aldrich, AM, Walker, DM, Rize,r MK, Moffatt-Bruce, SD,
Huerta, TR. 2016. “High Touch and High Tech (HT2): Transforming Patient Engagement Throughout the
Continuum of Care by Engaging Patients with Portal Technology at the Bedside.” JMIR Research
Protocols. 2016; 5(4):e221 http://www.researchprotocols.org/2016/4/e221 doi: 10.2196/resprot.6355

266 | P a g e
S30-3: What approach(es) to learning do users prefer to familiarize themselves with the inpatient
portal?

A. Free exploration

B. Video tutorial

C. In-person training

D. Blended – including free exploration, video tutorial, in-person training

Answer: D. Blended – including free exploration, video tutorial, in-person training

Explanation: Based on our formal usability study, different users expressed a preference for different
approaches that would help them learn to use the inpatient portal. Some users preferred to freely
explore the application, while others watched the video tutorial. While not offered in our study, some
participants voiced a preference for in-person training. To accommodate these different preferences,
hospitals implementing this technology may offer a blend of these approaches. This blended approach
would most effectively address different user needs.

Reference:

McAlearney, AS, Sieck, CJ, Hefner, JL, Aldrich, AM, Walker, DM, Rize,r MK, Moffatt-Bruce, SD, Huerta, TR.
2016. “High Touch and High Tech (HT2): Transforming Patient Engagement Throughout the Continuum
of Care by Engaging Patients with Portal Technology at the Bedside.” JMIR Research Protocols. 2016;
5(4):e221 http://www.researchprotocols.org/2016/4/e221 doi: 10.2196/resprot.6355

S30-4: Which of the following theoretic constructs is not a valid approach to measuring use of patient
portal use.

A. Comprehensiveness

B. Coherence

C. Duration

D. Frequency

Answer: B. Coherence

Explanation: While there are problems measuring duration in the context of the OSU study, they may be
idiosyncratic to OSUs implementation. In a system that used the standard implementation, there would
be an expectation that comprehensiveness, duration and frequency would be able to be robustly
measured, coherence wasn’t a measure that was discussed.

Reference: McAlearney, AS, Sieck, CJ, Hefner, JL, Aldrich, AM, Walker, DM, Rize,r MK, Moffatt-Bruce, SD,
Huerta, TR. 2016. “High Touch and High Tech (HT2): Transforming Patient Engagement Throughout the

267 | P a g e
Continuum of Care by Engaging Patients with Portal Technology at the Bedside.” JMIR Research
Protocols. 2016; 5(4):e221 http://www.researchprotocols.org/2016/4/e221 doi: 10.2196/resprot.6355

S31: Presentations - Analysis of Medical Data and Implication for Value-based Care

A clinical informatics consult service to use aggregate patient data at the bedside

--Saurabh Gombar

S31-1: The ideal clinical decision is supported by large multicenter randomized control trials. However,
less than 20% of clinical decisions are based on RCT data. Furthermore, an estimated 20% of clinical
decisions are not based on published evidence at all. Given this limitation, physicians have long wanted
to untap the electronic medical record (EMR) to quickly generate practice-based evidence relevant to
the clinical decision they are facing. To date which of the following has NOT been a limitation to rapidly
assimilating knowledge from the EMR?

A) Tools to build a cohort of patients who faced a similar clinical scenario are not widely available.

B) The multitude of data types in the EMR (ie ICD, CPT, Notes) require physicians to be proficient with a
multitude of medical nomenclature syntaxes

C) Analyzing large datasets is too time intensive for busy physicians

D) Physicians are uncomfortable using evidence generated from EMR data.

Answer: D) Physicians are uncomfortable using evidence generated from EMR data.

Explanation: The barriers to successfully utilizing the EMR to inform clinical decisions are numerous.
First, EMR data for large cohort analysis is not available to the vast majority of physicians. Second, the
tools to generate a cohort out of an EMR is extremely difficult and requires knowledge of database
querying, esoteric medical nomenclatures, and advanced statistics in addition to clinical knowledge.
Third, the time to do such an analysis is typically in the order of months, which is outside the available
time for busy physicians and outside the length of time to be useful to a patient. However, physician
comfort with trusting evidence generated from EMRs is not a barrier to adoption. Most large
retrospective studies published in the current literature rely on database querying of the EMR for at
least one step in their methodology.

Reference: Clemens SK. Kristof C. Jones B. Mitchell E. Martinez A. Barriers to electronic health record
adoption: a systematic literature review. J Med Syst. 2016; 40(12) 252

S31-2: Stanford’s approach to enable precision medicine through aggregate patient data utilizes a
consult service. This service is offered by a multidisciplinary team comprised of an informatics physician,

268 | P a g e
an EMR data specialist to create cohorts from the EMR, and a data scientist to perform statistical
analysis. Which of the following is NOT a reason utilize a consult model to generate aggregate patient
data?

A) The pathway from clinical question to useful information requires a multitude of disparate steps best
handled by a team with complimentary skills.

B) The consult model is a model that physicians are comfortable getting subspecialist information from.

C) An asynchronous model allows for quick turnaround time and can be scaled to ever increasing
demand.

D) A consult model is already billable as long as a note is generated for the patient.

Answer: D) A consult model is already billable as long as a note is generated for the patient.

Explanation: The consult model was chosen to deliver the aggregate patient data service for a multitude
of reason. Chief among them is that the journey from clinical question to knowledge requires an array of
steps in which no single party is an expert. Other reasons to use this approach include the familiarity of
physicians with a consult model and its ability to scale infinitely as each element can be expanded as
necessary. Although billing of a service will one day be important for widespread adoption it was not a
factor when deciding which model to implement.

Reference: Longhurst, CA. Harrington RA. Shah NH. A ‘Green Button’ for using aggregate patient data at
the point of care. Health Affairs. 2014 Jul; 33(7).

269 | P a g e
The Winning Health IT Engagement Model of an ACO in a Value-based Setting

--Nathan Riggle

The presenter will provide feedback on two multiple choice questions at the conclusion of the
presentation.

270 | P a g e
S32: Presentations - Patient Portals: Challenges and Opportunities

Patient Portal Proxy Access for Adolescents: Preserving Confidential Care While Enhancing Parent Proxy
Access

--Margaret Punch

S32-1: The parent of a 13-year-old would like full access to their child’s medical information available
through an on-line portal at their local health center. What considerations should be given to this
request?

a. The parent always has full rights to access the minor’s medical record.

b. Minor patients can’t consent to their own health care under any circumstances.

c. In all 50 states in the United States there are varying statutes that allow minors to seek confidential
care and that information can legally be withheld from the parent or legal guardian.

d. In all 50 states in the United States 13-year-olds have the right to confidential care to access
contraceptive services.

Answer: c. In all 50 states in the United States there are varying statutes that allow minors to seek
confidential care and that information can legally be withheld from the parent or legal guardian.

Explanation: The legal ability of minors to access confidential care varies from state to state but all 50
states and the District of Columbia have some consent laws or statues that allow minors to seek
confidential care. This is generally related to contraceptive services, STI services, prenatal care and
abortion. In some states there are restrictions based on specific age, or other conditions. All medical
care not covered under specific laws, statues or case law does require consent of the parent or legal
guardian.

Reference: https://www.guttmacher.org/state-policy/explore/overview-minors-consent-law

S32-2: When enabling proxy access to the parent of a minor child which portal feature may need
restriction?

a. Demographics

b. Vaccine records

c. Records from birth to age 10

d. Appointment schedules

e. Allergies

Answer: d. Appointment schedules

271 | P a g e
Explanation: In no states can a minor age 11 or younger consent to confidential care. Release of
demographics, allergies or vaccine records would not disclose that a minor has sought confidential care,
but knowledge of appointment schedules could, so appointments in areas where a minor can seek
confidential care should not be displayed to a proxy-holding parent.

Reference: https://www.guttmacher.org/state-policy/explore/overview-minors-consent-law

Do Community Clinic Patients Have Internet Access for Patient Portal Use?

--Ruth Bush

S32-3: Previously identified barriers to patient portal adoption include differences in internet access,
computer literacy, and internet proficiency. This digital divide is one explanation for underserved
groups being less likely to use portals, with strong correlations to utilization by race and household
income.

In the multi-ethnic, lower income community clinic in this study, which of the following was significantly
associated with previous patient portal use?

a. Having a “smart” phone

b. Education level

c. Health status

d. Race/ethnicity

Answer: A. Having a “smart” phone

Explanation: Within this sample, the variables of race/ethnicity, gender, education level, and health
status were not statistically associated with previous patient portal use. Having a smart phone was
associated (p=.02) with previous use.

Reference: Gordon NP, Hornbrook MC. Differences in access to and preferences for using patient portals
and other eHealth technologies based on race, ethnicity, and age: a database and survey study of
seniors in a large health plan. J Med Internet Res. 2016;18(3):e50. Available at:
http://www.jmir.org/2016/3/e50/. Accessed January 16, 2018

272 | P a g e
S32-4: When reporting past use as well as likely future use of the patient portal, participants reported
using all of the following functions of the portal EXCEPT:

a. Contacting provider

b. Renewing prescription

c. Obtaining health information

d. Scheduling appointments

Answer: c. Obtaining health information

Explanation: Reported use varied, from looking at test results (31%); communicating securely with
providers (20%); scheduling appointments (18%); and requesting prescription refills (9%). They did not
report obtaining additional health information about their condition(s) using the portal. Respondents
did express that they would like to do so in the future.

Reference: Griffin A, Skinner A, Thornhill J, Weinberger M. Patient portals: who uses them? what
features do they use? and do they reduce hospital readmissions? Appl Clin Inf. 2016;7(2):489-501.

Development of a Taxonomy of Secure Messages from an Inpatient Portal

--Cynthia Sieck

S32-5: On what dimensions can secure messages be classified?

A. Sender, Type and Topic

B. Type, Inpatient and Outpatient

C. Urgent, Non-urgent and Medical

D. Dietary, Medical and Non-medical

Answer: A. Sender, Type and Topic

Explanations: These represent the most inclusive initial dimensions. The other responses all fit within
these.

Reference: McAlearney, Ann Scheck, et al. "High touch and high tech (HT2) proposal: transforming
patient engagement throughout the continuum of care by engaging patients with portal technology at
the bedside." JMIR research protocols 5.4 (2016).

273 | P a g e
S32-6: Which of the following would be applications of a secure message taxonomy?

A. To improve the functionality of the secure messaging feature of the portal.

B. To help patients view their lab results.

C. To help care team prioritize responses.

D. A and C.

Answer: D. A and C

Explanation: Development of a taxonomy of secure messages does not assist patients in using other
features of the tool such as viewing lab results.

Reference: McAlearney, Ann Scheck, et al. "High touch and high tech (HT2) proposal: transforming
patient engagement throughout the continuum of care by engaging patients with portal technology at
the bedside." JMIR research protocols 5.4 (2016).

S33: Panel - The Clinical Informatics Fellowship: A Fellow’s Perspective

--Christian Dameff, William Lancaster, Chase Parsons, Melissa Van Cain

S33-1: Clinical Informatics Fellowships are accredited by which organization?

A. The National Institute of Health (NIH)
B. The Accreditation Council for Graduate Medical Education (ACGME)
C. The Association of American Medical Colleges (AAMC)
D. The Center for Medicare and Medicaid Services (CMS)

Answer: B. The Accreditation Council for Graduate Medical Education (ACGME)

S33-2: Clinical Informatics Fellowship requires the completion of what primary residency?
A. Internal Medicine
B. Emergency Medicine
C. Pediatrics
D. Any ACGME-recognized primary specialty

Answer: D. Any ACGME-recognized primary specialty

274 | P a g e
S33-3: The responsibilities of Clinical Informatics fellows include:

A. Research in informatics, patient care, implementing Informatics solutions

B. Patient care, research in informatics, and working in a basic science lab
C. Implementing informatics solutions, research in informatics, and serving as a medical
librarian
D. Research in informatics without any dedicated time to patient care

Answer: A. Research in informatics, patient care, implementing Informatics solutions

Explanation: A fellow in an ACGME-accredited fellowship is trained to “transform[s] health care by

analyzing, designing, implementing, and evaluating information and communication systems to improve
patient care, enhance access to care, advance individual and population health outcomes, and
strengthen the clinician-patient relationship."

References: AMIA Clinical Informatics Fellows. https://www.amia.org/membership/acif

Shortliffe E.H., Detmer D.E., Munger B.S. (2016) Clinical Informatics: Emergence of a New Profession. In:
Finnell J., Dixon B. (eds) Clinical Informatics Study Guide. Springer, Cham

S33-4: The duration of the clinical informatics fellowship is:

A. 1 year
B. 2 years
C. 3 years
D. 4 years

Answer: 2 years

Explanation: The educational program in ACGME-accredited clinical informatics (CI) fellowships must be
24 months in length.

Reference: Accreditation Council for Graduate Medical Education. ACGME Program Requirements for
Graduate Medical Education in Clinical Informatics (Review Committees for Anesthesiology, Diagnostic
Radiology, Emergency Medicine, Family Medicine, Internal Medicine, Medical Genetics and Genomics,
Pathology, Pediatrics, or Preventive Medicine).
https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/381_clinical_informatics_2017-07-
01.pdf

275 | P a g e
Additional details on Program Requirements for Graduate Medical Education in Clinical Informatics are
located at:
https://www.acgme.org/Portals/0/PDFs/FAQ/381_clinical_informatics_2017-07-01.pdf
https://www.theabpm.org/become-certified/subspecialties/clinical-informatics/
http://www.abpath.org/index.php/to-become-certified/requirements-for-certification?id=40

S34: Presentations - Tool Based Analytics

Institution of a Pop Up Reminder Decreases Opioid Prescriptions for Pediatric Patients in the Emergency
Department

--Melissa Van Cain

S34-1: The opioid epidemic has had a serious effect on the US population in recent years and this is true
of adolescent patients as well as adults. What is the approximate number of adolescents who are
addicted to prescription pain relievers?

A. 1,500
B. 15,000
C. 150,000
D. 1.5 million

Answer: C. 150,000

Explanation: According to the Center for Behavioral Health Statistics and Quality, in 2014 approximately
168,000 adolescents in the US had an addiction to prescription pain relievers based on the 2014
National Survey on Drug Use and Health. That same study showed that there were more than 460,000
adolescents who were current nonmedical users of pain relievers and that most adolescents misusing
opioids receive them free from a friend or relative.

Reference: Center for Behavioral Health Statistics and Quality. Behavioral health trends in the United
States: results from the 2014 National Survey on Drug Use and Health (HHS Publication No. SMA 15-
4927, NSDUH Series H-50).
http://www.samhsa.gov/data/sites/default/files/NSDUH-FRR1-2014/NSDUH-FRR1-2014.pdf. Published
September 2015. Accessed March 25, 2018.

S34-2: Massachusetts (MA) enacted a law in 2016 that providers are required to check the MA
prescription drug monitoring program before prescribing opioids. We achieved the biggest change in
opioid prescriptions with this intervention:

276 | P a g e
A. Adding a link to the chart
B. Reminder e-mails
C. Physician training
D. Pop-up reminder
E. Hard stop

Answer: D. Pop-up reminder

Explanation: This project used a pop-up window to remind providers to check the Massachusetts
Prescription Monitoring Program web page before prescribing opioid prescriptions to a patient. There
was a significant decrease in the number of opioid prescriptions ordered after the pop-up reminder was
implemented. Adding a link to the chart did not have a significant impact because it was hard to
remember to click it. Reminder e-mails are not presented when the prescriber is actually ordering the
medication. Physician training is important, but is also not presented at the time of the order. Hard stop
was not instituted because they are very disruptive to workflow.

Reference: Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical
decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005
Apr 2;330(7494):765. Epub 2005 Mar 14. Review. PubMed PMID: 15767266; PubMed Central PMCID:
PMC555881.

Radiation Therapy Data Curation: Utilizing Interdisciplinary Expert Knowledge via an Iterative Tool-based
Process

--Thilo Schuler

S34-3: Manual data curation is currently the most common approach to retrospectively improving data
quality in clinical datasets.

Which of the following options is the greatest challenge with this approach?

A. Unsuitability for inter-institutional data exchange

B. Poor scalability

C. Poor accuracy

D. Lack of contextual knowledge

Answer: B. Poor scalability

Explanation: The biggest reason why manual curation doesn’t scale well is the need for clinical expert
involvement, which is a scarce resource. The accuracy of manual approaches is typically higher as

277 | P a g e
experts use implicit contextual knowledge. Manual curation can enable inter-institutional data exchange
if the curation strategy is designed for it (e.g. mapping to global standards).

Reference: Freitas A, Curry E. Big Data Curation. In: New Horizons for a Data-Driven Economy [Internet].
Springer, Cham; 2016 [cited 2018 Mar 25]. p. 87–118. Available from:
https://link.springer.com/chapter/10.1007/978-3-319-21569-3_6

S34-4: Using a fully automated curation approach for large clinical datasets would be ideal, as it is
scalable and requires minimal clinician input. However, in most settings this is at the moment not
achievable.

Which of the following strategies is most successful in reducing the time imposed on clinical expert
curators?

A. Using non-expert curators

B. Focus on subset of data

C. Technology support

D. More training for curators

Answer: C. Technology support

Explanation: Technology can increase the efficiency of clinical expert curators. Often a hybrid strategy is
used: A large proportion of data is curated without human intervention. Where the algorithm can’t
reach a decision with sufficient certainty, the human expert helps to make the decision. Another
technology-enabled strategy is to use the human expert to verify the automated curation by scanning
for abnormalities (e.g. via a visualisation tool) that can arise from process errors. Non-expert curators
don’t have sufficient knowledge to do the curation task. Sampling a subset of data to verify data quality
might be an option, but if manual curation is needed, this would have to be done for the full dataset.
Training can increase curation efficiency, but this is modest compared to the scale of large datasets.

Reference: The Deeply Human Core Of Roche’s $2.1 Billion Tech Acquisition -- And Why It Made It
[Internet]. [cited 2018 Mar 25]. Available from:
https://www.forbes.com/sites/davidshaywitz/2018/02/18/the-deeply-human-core-of-roches-2-1b-tech-
acquisition-and-why-they-did-it

278 | P a g e
Developing a visual annotation tool to rapidly collect expert-annotated weight errors in pediatric growth
charts

--PJ Van Camp

S34-5: The clinical importance of a child’s weights being entered correctly into the electronic health
record is to:

A) correctly classify children as obese or malnourished

B) avoid potential medication dosing errors
C) avoid useless information being stored and taking up valuable disk space
D) ensure correct reimbursement from insurance companies

Answer: B) avoid potential medication dosing errors

Explanation: Weight is often used to calculate the correct dose of medications in children. Using a
wrong weight could lead to under- or over- dosing the child and potentially result in treatment failure or
increased adverse effects, respectively. Weight alone is not enough to classify a child as obese (one
needs height as well) or malnourished (one needs intake). The storage of extra useless data is not of
clinical importance and insignificant compared to other data volume. Reimbursement by insurance is
again not of clinical importance and uses other sources of information (e.g. ICD codes)

Reference: Kaushal, R., et al., Medication errors and adverse drug events in pediatric inpatients. JAMA,
2001. 285(16): p. 2114-20.

S34-6: What is a major benefit of creating a software application to let experts annotate a large dataset
instead of manual annotation without the use of a specific tool?

A) The process of gathering, annotating and saving the data can be sped up and it allows annotators
unfamiliar with data analysis software to interact with the data in an intuitive manner.

B) Data annotated digitally is more credible than data annotated by hand because the latter is more
prone to error

C) Once a tool has been developed, it can be used for many other projects that require different dataset
annotations

D) Only experts can annotate data manually, while with a tool anyone can

Answer: A) The process of gathering, annotating and saving the data can be sped up and it allows
annotators unfamiliar with data analysis software to interact with the data in an intuitive manner.

279 | P a g e
Explanation: The point of developing a tool is to automate certain processes and facilitate interaction
with data for users. Depending on the problem, tools often reduce repetition or enhance visualization of
the task at hand.

Digitally annotated data is not per se of better quality, especially if a software error messes up the data
without the knowledge of the annotator. Vigorous testing is needed to make sure everything is as good
as if it were done manually.

Tools are developed for a specific task, meaning often it needs considerable modifications if wanting to
be used for other projects.

Finally, annotation itself depends on the expertise of the annotator, not the tool used to do the task,
though it may facilitate it.

References: https://www.usability.gov/what-and-why/user-interface-design.html
https://shiny.rstudio.com/ https://material.io/

Closing Session and Keynote Panel: Building Clinical Decision Support that Works

--Adam Wright, Joan Ash, Dean Sittig, Bill Galanter, Sarah Collins

S35-1: A healthcare system wants to reduce the time it takes to clinical decision support system
malfunctions. Because the organization participates in several quality programs, the organization’s
leadership is particularly concerned about “false negative” situations, where CDS is supposed to fire, but
doesn’t. Which of these strategies is most likely to quickly detect “false negatives”?

A. User reporting through the help desk

B. Ongoing monitoring with anomaly detection

C. Routine clinical use of the system by informatics leaders

D. Sentiment analysis of override reasons

Answer: B. Ongoing monitoring with anomaly detection

Explanation: The approach most likely to detect false negatives quickly is ongoing monitoring with
anomaly detection. This approach is suitable for detecting both false positives and false negatives, and
can work quickly. Research suggests that user reporting through the help desk is helpful for detecting
false positives (situations where alerts fire when they shouldn't), but isn't effective for detecting false
positives. Similarly, routine clinical use of the system by informatics leaders is more likely to detect false
positives - false negatives are observed less commonly with this approach, and often after significant
delay. Sentiment analysis of override reasons can only be applied when alerts fire and users comment

280 | P a g e
on them -- users can't comment on missed firings, and are very unlikely to make a note about a missed
firing if they later see an appropriate firing.

Reference: Wright A, Hickman TT, McEvoy D, Aaron S, Ai A, Andersen JM, Hussain S, Ramoni R, Fiskio J,
Sittig DF, Bates DW. Analysis of clinical decision support system malfunctions: a case series and survey.
Journal of the American Medical Informatics Association. 2016 Mar 28;23(6):1068-76.

S35-2: A few years after implementation of an Alert type of CDS, or "rule," there are a series of
complaints about how it is functioning. The rule was designed by a requesting physician and IS staff. The
requesting physician explained what they were looking to have built in a few paragraphs and then it was
approved by the CDS committee and built by an IS analyst. A new IS analyst is having a hard time
determining if the rule is doing what it was intended to do as the paragraph describing the intent is not
clear and does not provide sufficient detail.

What is the best strategy to prevent this type of difficulty?

a) Use clinical informaticists to investigate potential rule malfunctions.

b) Require rule requests to be formatted as flowcharts with sufficient details to allow IS to interpret
unambiguously and future clinicians and IS to understand the intent.

c) Require the initial requesting clinician to test all CDS and be responsible for any future queries once
the CDS receives approval.

d) Check the override rate of the rule, if greater than 85%, turn the rule off.

Answer: b) Require rule requests to be formatted as flowcharts with sufficient details to allow IS to
interpret unambiguously and future clinicians and IS to understand the intent.

Explanation: When CDS is conceived, designed and implemented, documentation needs to occur in a
manner that will most likely produce proper functioning CDS that matches the request. In addition,
studies show that despite the best intentions and efforts, CDS can malfunction in the future. The
documentation needs to support future efforts to diagnose and fix errors or outdated design
assumptions. Flowcharts are a long-used standard method for representing processes, initially in
industrial engineering almost 100 years ago and then transitioning to use in electronics and finally
computer programming. Flowcharts allow a visual representation of logic and workflow which is not
possible with prose. Thus, the use of flowcharts in rule design and implementation can help improve
initial success as well as future troubleshooting.

Reference: Osheroff JA. Improving outcomes with clinical decision support: an implementer's guide.
HIMSS.

281 | P a g e
S35-3: Leaders of a large healthcare system want to optimize their EHR by providing more effective CDS
with minimal malfunctions. Which of the following “people” strategies might help the most?

a) Set up a CDS committee of subject matter experts.

b)Involve clinical staff and encourage them to report malfunctions such as alerts not firing when they
should.

c) Assure that staff members in informatics, quality improvement, IT, and elsewhere liaise with one
another and with clinicians in multiple ways.

d)Hire additional staff to do measurement and monitoring because only they can consistently identify
malfunctions.

Answer: c) Assure that staff members in informatics, quality improvement, IT, and elsewhere liaise with
one another and with clinicians in multiple ways.

Rationale: The most useful strategy would be a multiple perspectives approach that would provide
resources for inclusion of leadership and personnel from different areas to liaise with one another and
with clinicians about CDS. The option of setting up a CDS committee of subject matter experts is
definitely a good and necessary strategy, but it does not go far enough. Involving clinical staff and
encouraging their reporting of malfunctions is also an excellent strategy, but malfunctions like alerts
failing to fire are hard for them to identify. Finally, hiring staff members to do additional monitoring
would be an excellent idea if resources were available, but this move should be part of a larger, well
planned strategy.

Reference: Wright A, Ash JS, Erickson JL, Wasserman J, Bunce A, Stanescu A, St Hilaire D, Panzenhagen
M, Gebhardt E, McMullen C, Middleton B. A qualitative study of the activities performed by people
involved in clinical decision support: recommended practices for success. Journal of the American
Medical Informatics Association. 2013 Sep 2;21(3):464-72.

S35-4: You are leading the governance committee for clinical decision support at your institution. A new
request for development of CDS that is nurse-facing was submitted. The request is to implement an
evidence-based risk assessment for peripheral nerve damage in the operating room with guidance to
support appropriate interventions. Your organization leverages the Data-Information-Knowledge-
Wisdom (DIKW) Conceptual Framework to prioritize strategic development of CDS to support nursing
practice. Which component of the framework best categorizes this CDS request?

a. Data

b. Information

c. Knowledge

d. Wisdom

282 | P a g e
Answer: c. Knowledge

Explanation: The data category pertains to facilitating data capture, such as ‘does patient smoke?’. The
nurse documents a yes/ no value. At this level the nurse just enters data to a question. There is no
nursing judgment or knowledge needed to document this data. Information category pertains to CDS
that provide nurses with information about their patient at the right time. Knowledge category refers to
any CDS that provides an alert according to the information the nurse enters into the system based on
nursing knowledge. Another example in the knowledge category is: CDS that promotes accurate
documentation of pressure ulcer stage based on nursing knowledge and recommends an appropriate
plan of care. The wisdom category includes exposing analytics for wisdom-based clinical interpretation
by the nurse, such as a sepsis alert.In a ‘wisdom’ categorized CDS, the nurse leverages his or her wisdom
when deciding if the evidence-based recommendations provided by the CDS intervention are
appropriate for that patient. To effectively distribute limited resources in meeting the highest priority
needs, the Data-Information-Knowledge-Wisdom (DIKW) Conceptual Framework can guide selection of
CDS aligned with an organization’s strategic plan for how CDS should support nurses practicing at the
top of their professional license.

References:

Whalen, K., Bavuso, K., Bouyer-Ferullo, S., Goldsmith, D., Fairbanks, A., Gesner, E., lagor, C., Collins, S.
(2016). Analysis of Nursing Clinical Decision Support Requests and Strategic Plan in a Large Academic
Health System. Applied Clinical Informatics, 7(2), 227–237.

Matney S, Brewster PJ, Sward KA, Cloyes KG, Staggers N. Philosophical approaches to the nursing
informatics data-information-knowledge-wisdom framework. Adv Nurs Sci 2011; 34(1):6–18.
doi:10.1097/ANS.0b013e3182071813

Nursing Informatics. Scope and Standards of Practice. Silver Spring, MD: American Nurses Association.
2008; 1: 177.

283 | P a g e