Journal Online
Impact of Security Awareness
Karen Quagliata, Ph.D.,
PMP, has worked in the IT
field for more than 10 years in
diverse capacities. Quagliata
currently works within the
financial services industry
as an information security
analyst, specializing in risk
management. In addition,
she is a published author in
various industry publications.
 o you have
D was conducted for the doctoral program at the
something
to say about
this article?
Visit the Journal pages
of the ISACA web site
(www.isaca.org/journal),
find the article, and
choose the Comments
tab to share your
thoughts.
Training Components on Perceived
Security Effectiveness
“Weakest link” and “unpredictable”: To
an information security practitioner, these
descriptions almost certainly identify the human
component of a layered security approach
because humans are unpredictable animals
who are susceptible to temptation, emotions
and complacency. After all, the strongest
firewall and the most sophisticated intrusion
detection/prevention software will not prevent
an individual from disclosing sensitive data via
social engineering. In fact, the recent data breach
at RSA, the security division of EMC Corp., is
proof. In this breach, the attackers were able to
gather information about the company’s SecurID
two-factor authentication products through a
phishing attack. The attackers sent a Microsoft
Excel file via e-mail to employees, and at least
one employee opened the file, thus opening the
door to the attack. Therefore, the breach was not
a technology problem, but a human problem.1 It
is no wonder that, for many years, information
security professionals have been touting the
importance of providing security awareness
training within an organization. However, no
research has been conducted to determine
the most effective components of a security
awareness training program. Professionals say to
do it, but they do not say how to do it.
To address this lack of research in the area
of security awareness components, research
University of Fairfax (Vienna, Virginia, USA)
to examine the relationship of user awareness
training components and perceived security
effectiveness. The research extended the work
of Kenneth Knapp, who, in 2005, addressed the
questions of the relationship of top management
support on perceived security effectiveness and
the constructs that mediate that relationship.
Knapp examined four mediating variables: user
training, security culture, policy relevance and
policy enforcement.2 Based on the findings of
his research, it was concluded that, of the four
mediating variables that he identified, user
training had the strongest relationship between
the independent variable of top management
support and the dependent variable of security
effectiveness. The purpose of this research, then,
was to focus on the relationship between user
training and security effectiveness.
PROJECT OBJECTIVES
The purpose of this study is to help organizations
increase their chances of implementing effective
security awareness training by identifying the best
possible set of user awareness training variables.
Using a survey, IT professionals’ perceptions of
security effectiveness within their organizations
were measured based on their attitudes toward:
1. Frequency of user awareness training
2. Method of user awareness training
3. User awareness training compliance monitoring
RESEARCH APPROACH AND METHODOLOGY
An anonymous survey was used as the data
collection tool. The survey used a five-point
Likert scale to measure the participants’ attitudes
toward components of their organizations’
security awareness trainings and the security
effectiveness within their organizations. The
research was driven by three questions:
1. What is the relationship between training
frequency and perceived security effectiveness
as measured by the survey?
2. What is the relationship between training
delivery method and perceived security
effectiveness as measured by the survey?
3. What is the relationship between training
compliance monitoring and perceived security
effectiveness as measured by the survey?
Validity of the survey was addressed by using
a panel-of-experts approach. A peer review/
ISACA JOURNAL VOLUME 4, 2011 1

• Read Creating a Culture of Security.
www.isaca.org/research
field trial was conducted during the period of 31 July 2009
and 8 September 2009, using a 10-person expert panel that
consisted of ISACA members from the St. Louis (Missouri,
USA) and Illinois (USA) chapters. Surveys were sent to
the expert panel via e-mail with a cover letter that explained the
proposed research project and that asked the
following questions:3
• Is the content of the questionnaire appropriate for
the audience?
• Are the survey items clear, and do they make sense?
• Are any of the survey items intrusive, invasive, potentially
embarrassing or of a sensitive nature?
Feedback was received via e-mail and incorporated into the
survey as deemed appropriate; for the most part, changes were
minor.
Reliability of the survey was measured using a test-retest
sequence administered to a pilot study panel of nine ISACA
members who did not participate in the expert panel. The
pilot study occurred during the period of 24 November 2009
and 12 January 2010. The survey was loaded into an online
survey web site, and members of the pilot study were randomly
assigned a number from one to nine. The panel was then sent
an e-mail that instructed the participants to complete the survey
on the online survey web site. The first phase of the test-retest
sequence occurred between
24 November 2009 and 3 December 2009. The content and
wording of the questions were not altered for the retest phase,
but the questions were randomly reordered. On 21 December
2009, another e-mail was sent to the panel, instructing
participants to complete the survey again. The final phase
of the pilot study was completed on 12 January 2010. The
results were downloaded into Statistical Package for the Social
Sciences (SPSS) software for analysis. The expected outcome
of the test-retest sequence was that there would be little or
no significant difference between the results of test and retest
data. One question did show significant difference and was,
therefore, removed from the final survey.
2 ISACA JOURNAL VOLUME 4, 2011
Upon completion of the pilot study, the final version of the
survey was created for the online survey web site. The link to
the survey was sent via e-mail to all ISACA chapter presidents
for distribution and to the ISACA headquarters for posting
on its web site. The link was also posted on ISACA social and
professional networking sites. The survey remained accessible
on the survey web site between 1 March 2010 and 31 March
2010. The data were then analyzed.
ISACA was chosen as the research site because the
international professional association is comprised of a balanced
mix of various levels of IT professionals working in multiple
“
industries and in various capacities. IT professionals, rather than
regular employees, were chosen
A debate exists within for the research because they are
the IT community more aware of IT security issues
and are a more homogeneous
regarding the
group. Presenting the survey
measurement of to the random public would
perceived security have likely resulted in more
”
inconclusive findings because
effectiveness.
of the heterogeneity of such a
large group. By the same token,
limiting the survey to one industry or organization would have
limited the scope of the research.
As Knapp points out in his research, a debate exists
within the IT community regarding the measurement of
perceived security effectiveness. The elusive nature of the term
“effectiveness,” coupled with the sensitive nature of asking
an organization to measure its security, poses a challenge to
developing a common industry definition.4 Richard A. Caralli
elaborates on the challenging aspect of defining security
effectiveness by pointing out that security is contextual and
not an isolated discipline; it depends on the organization and
its operations. Furthermore, effective security “must take into
account the dynamically changing risk environment within
which most organizations are expected to survive and thrive.”5
As such, Knapp did not attempt to establish a definition for
security effectiveness for his research. Instead, the perceived
effectiveness variable in his study was “based on the subjective
judgment of security professionals.”6 As this research extended
Knapp’s, it also based security effectiveness on the subjective
judgment of security professionals.
RESPONDENTS
A total of 133 ISACA members, representing multiple
industries, participated in the survey. The largest percentage
(26 percent) of the respondents worked in the finance,
banking and insurance industry. Government and
professional services were the two second most popular
industries. Nonprofit and industrial technology had the least
representation at 1 percent each.
The participants represented multiple countries, albeit
the majority of participants were from the United States
(73 percent). Other countries represented included: India
(17 percent), Costa Rica (5 percent), Australia (2 percent),
Belgium (2 percent) and China (1 percent).
The majority of participants (60 percent) reported that
information security is a secondary responsibility of their
jobs. This is understandable considering that the majority of
the participants (43 percent) identified themselves as audit
professionals. “Information security professional” ranked
third at 12 percent.
The majority of respondents (89 percent) held at least
one professional certification. Certified Information Systems
Auditor (CISA) and Certified Information Systems Security
Professional (CISSP) were the most common. Other popular
certifications included Certified Information Security
Manager (CISM), Certified in the Governance of Enterprise
IT (CGEIT), Certified Internal Auditor (CIA) and Certified
Public Accountant (CPA).
SURVEY RESULTS
Six key findings came out of this research, which can be
categorized as:
1. Overall security effectiveness
2. Training frequency
3. Training methodology
4. Training compliance monitoring
5. Training topics
6. Relationships
Overall Security Effectiveness
The first finding is related to security effectiveness as indicated
by how participants ranked their agreement to the survey
statement: “My organization secures its data and information
effectively.” Overall, the majority of participants agree that their
organizations secure their data and information effectively.
However, the majority of those participants do not strongly
agree. Only 22 percent strongly agreed that their organization
secures its data and information effectively. Close behind
are those respondents who were neutral on the subject (20
percent). However, the majority of respondents (42 percent) did
agree that their organization secures its data and information
effectively. While positive overall, these numbers also show that
36 percent of those surveyed believe that their organization is
either not effectively securing its data or were neutral on the
subject. Clearly, there is room for improvement.
Training Frequency
The majority of organizations (50 percent) represented in
this research deliver user security awareness training once a
year. However, only 24 percent of the organizations delivered
training more than once a year, and 17 percent delivered
the training during new employee orientation only. An even
smaller portion of the organizations either delivered it on a
voluntary basis only or never.
When training frequency was cross-tabulated with
perceived security effectiveness, the “once a year” category
had the highest rate of participants who strongly agreed that
their organization secures its data and information effectively.
The “more than once a year” category follows closely.
However, the other categories have little to no instances of
participants strongly agreeing. It would appear that the fewer
times employees are exposed to user security awareness
training, the less likely they will be to view their organizations
as effectively securing data.
Training Methodology
According to the survey, the majority of organizations
(69 percent) use some combination of methods to deliver
training, vs. the 4 percent of participants who reported that
their organizations used all of the methods listed in the
survey. Those organizations that depend solely on policies
and procedures as user security awareness training made up
the next highest majority (8 percent). Only a small portion of
the respondents answered that their organizations used only
one training method (other than policies and procedures) to
deliver user security awareness training.
When training method was cross-tabulated with perceived
security effectiveness, the “combination of methods” category
had the highest rate of participants who strongly agreed
ISACA JOURNAL VOLUME 4, 2011 3
 that their organizations secure their data and information was conducted. This raises the question: Do the organizations
effectively. The “computer-based training only” category was conduct the monitoring in such a way that employees do not
ranked second. However, the other categories have little to no know about it, or is it that the respondents were confused by the
instances of participants strongly agreeing. It would appear question on the survey?
that employees exposed to only one type of user security Those respondents whose organizations conduct training
awareness training methodology were less likely to view their compliance monitoring once a year are the largest group
organizations as effectively securing their data. to strongly agree that their organizations effectively secure
their data. Those respondents whose organizations conduct
Training Compliance Monitoring training compliance monitoring more than once a year are the
The concept of training compliance monitoring is that second largest group to strongly agree that their organizations
organizations verify that their employees are satisfying the effectively secure their data. Clearly, participants from
security training requirements, retaining that knowledge and organizations that conduct training compliance monitoring
implementing it in the workplace. The research data showed only when new access is granted to a system, or that do not
that training compliance monitoring was the second best conduct training compliance monitoring at all, are the least
predicator for perceived security effectiveness. This variable likely to strongly agree that their organizations effectively
was analyzed from two perspectives: how and how often the secure their data.
compliance monitoring was conducted.
The majority of respondents (38 percent) indicated that Training Topics
their organizations use electronic sign-off as the only means The next area of findings concerns the material that
for training compliance monitoring. The next highest group of organizations are teaching their employees regarding security
respondents (20 percent) stated that their organizations use a awareness. As seen in figure 1, the most popular security
combination of methods for training compliance monitoring. awareness training topic pertains to e-mail. Passwords and
Another 20 percent stated that their organizations use no Internet usage are close behind. However, topics such as
training compliance monitoring methods. social engineering and data encryption appear closer to the
Those respondents whose organizations use only electronic bottom of the list.
sign-off for training compliance monitoring are the largest
group to strongly agree that their organizations effectively Figure 1—User Awareness Training Topics
secure their data. Those respondents whose organizations
Security Awareness Training Topics Count
use a combination of methods for training compliance
E-mail 86
monitoring are the second largest group to strongly agree
that their organizations effectively secure their data. Clearly, Passwords 83
participants from organizations that use other methods (such Internet use 80
as verbal tests or monitoring tools) or no methods at all Locking workstations 74
are the least likely to strongly agree that their organizations Privacy 72
effectively secure their data.
Data handling/classification 68
The majority of respondents (34 percent) indicated that their
Social engineering 66
organizations conduct training compliance monitoring once a
year. The next highest group of respondents (22 percent) stated All of the topics listed 53
that their organizations conduct training compliance monitoring Network security 47
more than once a year. Another 22 percent stated that their Data encryption 35
organizations never conduct training compliance monitoring. No user awareness security training is conducted. 8
It is interesting that 16 percent of the respondents stated that
I do not know. 2
they did not know how often training compliance monitoring

4 ISACA JOURNAL VOLUME 4, 2011

 Security awareness training topics were cross-tabulated
with perceived security effectiveness. Those respondents
whose organizations covered all of the topics included in
the survey are the largest group to strongly agree that their
organizations effectively secure their data. The participants
whose organizations included some combination of the topics
listed in the survey are the second most likely to strongly
agree that their organizations effectively secure their data. The
respondents whose organizations covered only one topic in
their training were the least likely to strongly agree that their
organizations effectively secure their data.
Relationships
Finally, this research showed that there is a strong correlation
between perceived security effectiveness and the components
of training method and training compliance monitoring.
However, the relationship between training frequency and
perceived security effectiveness was inconclusive. Therefore,
it can be concluded that training method and training
compliance monitoring are the strongest predictors for
security effectiveness.
IMPLICATIONS
Results of this research can provide practical guidance to
information security practitioners and those setting the
policies within organizations. Four main implications can be
surmised based on this study.
Provide Training at Least Once a Year
Findings showed that those respondents whose organizations
provided training once a year had the highest rate of strongly
agreeing that their organizations secure their data effectively.
Participants from organizations that provided training more
than once a year did not greatly improve their numbers.
Employ Multiple Training Methods
In addition, based on the findings of this research, training
method appears to have the biggest impact on perceived
security effectiveness; furthermore, the use of multiple
methods of training produced the highest correlations to
perceived security effectiveness. As such, organizations should
consider focusing resources on training methodology. They
should strive to combine various tools, including:
• Computer-based training
• Policies and procedures
• Newsletters
• E-mail
• Leader-led training
• Video
• Posters
• Brochures
Ensure Compliance
This research also showed that training compliance
monitoring had a strong relationship with perceived security
effectiveness. It is not enough for organizations to merely
implement a security awareness training program. Policy
makers within organizations should strive to better monitor
the training for compliance; equally important is that they
ensure consequences are in place for noncompliance. As one
survey respondent stated, “Training sessions are performed
annually, and attendance and understanding are monitored.
However, no action is taken that I know if someone does not
participate in the training. It is just reported to management.”
Respondents from a recent Enterprise Strategy Group
survey stated that training users on confidential data
security policies was the most important measure for
protecting proprietary information.7 Yet, only 36 percent
of government workers are held accountable for knowing
information security policies and procedures via their annual
performance evaluation. In addition, only 48 percent were
tested throughout the year on what they learned in awareness
training.8 Therefore, while it is important for leadership to
monitor employees for adherence to policies, it should not
be performed only once a year. Such monitoring should be
an ongoing occurrence, especially considering that some
public-sector studies have shown that more than 80 percent of
breaches occur not because of malicious intent, but because
employees claim not to know about a policy or because they
simply ignored it.9
Teach Relevant Topics
This research revealed that topics such as social engineering
and data encryption appeared at the bottom half of the list
of the most popular training awareness topics within the
organizations of the respondents. However, current data
breach information is proving these to be critical areas. For
example, social engineering attacks continue to grow. In fact,
phishing, a social engineering technique, is included in the
2009 Verizon Business Supplemental Data Breach Report’s
ISACA JOURNAL VOLUME 4, 2011 5
 top 15 most common security attacks.10 Furthermore, 2 Knapp, Kenneth Joseph; “A Model of Managerial
encryption is also playing a more critical role when one Effectiveness in Information Security: From Grounded
considers that lost/stolen laptops, smartphones and removable Theory to Empirical Test,” dissertation, Auburn University,
storage devices are contributing factors to data breaches. In USA, 2005, http://etd.auburn.edu/etd/bitstream/
fact, according to a 2009 Ponemon Institute study, 36 percent handle/10415/708/KNAPP_KENNETH.pdf?sequence=3
of all data breach cases examined involved lost or stolen 3 Lease, D.R.; “Factors Influencing the Adoption of
laptop computers or other mobile devices. These types of Biometric Security Technologies by Decision Making
data breaches tend to be more expensive than other incidents, Information Technology and Security Managers,”
costing approximately US $225 per victim.11 As such, dissertation, Capella University, USA, 2005
practitioners should develop security awareness training that 4 Op cit, Knapp
places more emphasis on these relevant topics. 5 Caralli, Richard A.; Managing for Enterprise Security,
Carnegie Mellon University, USA, 2004, www.sei.cmu.edu/
CONCLUSION reports/04tn046.pdf
Security awareness training alone will not secure an 6 Op cit, Knapp
organization, just as technical solutions alone will not secure 7 Berrong, Stephanie; “Creative Approaches to Security
an organization. Although individuals are taught the secure Awareness Training,” Security Management, July 2009
way to interact with data, they may not always follow that 8 SecureInfo Corp., Information Security Awareness Report:
training. Clearly, information security must involve a layered The Government Workers’ Perspective, USA, 2007,
approach that includes both technical and nontechnical www.secureinfo.com/downloads/reports/SecureInfo-
solutions. Security awareness training is a vital nontechnical InfoSec-Report-Dec-2007.pdf
component to information security. As such, it is in the 9  Government Security, “Study Shows Fed Workers in Dark
interest of the public and private sectors to continue to About Security,” 31 May 2007, http://preview.govtsecurity.
research this component that directly impacts security’s com/news/fed-workers-in-dark/
weakest link: humans. Verizon Business, “Verizon Business Issues 2009
10

Supplemental Data Breach Report Profiling 15 Most

ENDNOTES Common Attacks,” PR Newswire, 9 December 2009,
1 Westervelt, Robert; “RSA SecurID Breach Began www.prnewswire.com/news-releases/verizon-business-
With Spear Phishing Attack,” SearchSecurity.com, issues-2009-supplemental-data-breach-report-profiling-15-
4 April 2011, http://searchsecurity.techtarget. most-common-attacks-78840502.html
com/news/article/0,289142,sid14_gci1529523,00. 11 Ponemon Institute, 2009 Annual Study: Cost of a Data
html?track=NL-102&ad=824622&asrc=EM_ Breach, PGP Corp., USA, January 2010
NLN_13603105&uid=10604598

The ISACA Journal is published by ISACA. Membership in the association, a voluntary organization serving IT governance professionals, entitles one to receive an annual subscription
to the ISACA Journal.

Opinions expressed in the ISACA Journal represent the views of the authors and advertisers. They may differ from policies and official statements of ISACA and/or the IT Governance
Institute® and their committees, and from opinions endorsed by authors’ employers, or the editors of this Journal. ISACA Journal does not attest to the originality of authors’ content.

© 2011 ISACA. All rights reserved.

Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For other copying, reprint or republication, permission must be obtained in
writing from the association. Where necessary, permission is granted by the copyright owners for those registered with the Copyright Clearance Center (CCC), 27 Congress St.,
Salem, MA 01970, to photocopy articles owned by ISACA, for a flat fee of US $2.50 per article plus 25¢ per page. Send payment to the CCC stating the ISSN (1526-7407), date,
volume, and first and last page number of each article. Copying for other than personal use or internal reference, or of articles or columns not owned by the association without
express permission of the association or the copyright owner is expressly prohibited.

www.isaca.org

6 ISACA JOURNAL VOLUME 4, 2011