Psychotherapy Research

ISSN: 1050-3307 (Print) 1468-4381 (Online) Journal homepage: http://www.tandfonline.com/loi/tpsr20

How to conduct a qualitative meta-analysis:

Tailoring methods to enhance methodological
integrity

Heidi M. Levitt

To cite this article: Heidi M. Levitt (2018): How to conduct a qualitative meta-analysis:
Tailoring methods to enhance methodological integrity, Psychotherapy Research, DOI:
10.1080/10503307.2018.1447708

To link to this article: https://doi.org/10.1080/10503307.2018.1447708

Published online: 13 Mar 2018.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=tpsr20
Psychotherapy Research, 2018
https://doi.org/10.1080/10503307.2018.1447708

CONSIDERATIONS OF HOW TO CONDUCT META-ANALYSES IN PSYCHOLOGICAL

INTERVENTIONS

How to conduct a qualitative meta-analysis: Tailoring methods to

enhance methodological integrity

HEIDI M. LEVITT

Department of Psychology, University of Massachusetts Boston, Boston, MA, USA

(Received 31 October 2017; revised 21 December 2017; accepted 22 December 2017)

Abstract
Although qualitative research has long been of interest in the field of psychology, meta-analyses of qualitative literatures
(sometimes called meta-syntheses) are still quite rare. Like quantitative meta-analyses, these methods function to
aggregate findings and identify patterns across primary studies, but their aims, procedures, and methodological
considerations may vary. Objective: This paper explains the function of qualitative meta-analyses and their
methodological development. Recommendations have broad relevance but are framed with an eye toward their use in
psychotherapy research. Rather than arguing for the adoption of any single meta-method, this paper advocates for
considering how procedures can best be selected and adapted to enhance a meta-study’s methodological integrity.
Method: Through the paper, recommendations are provided to help researchers identify procedures that can best serve
their studies’ specific goals. Meta-analysts are encouraged to consider the methodological integrity of their studies in
relation to central research processes, including identifying a set of primary research studies, transforming primary
findings into initial units of data for a meta-analysis, developing categories or themes, and communicating findings.
Conclusion: The paper provides guidance for researchers who desire to tailor meta-analytic methods to meet their
particular goals while enhancing the rigor of their research.

Keywords: qualitative meta-analysis; meta-synthesis; methodological integrity; psychotherapy; trustworthiness

Clinical or methodological significance of this article: This article describes the function of meta-analytic methods in
relation to various research goals, such as the development of theoretical models, comprehensive literature reviews, or
methodological reviews. It encourages researchers to design and carry out studies in a manner that strengthens the
methodological integrity of their work in relation to their specific goals.

Qualitative meta-analytic methods are tools that allow
researchers to aggregate and synthesize findings from
primary qualitative studies. These studies can assist
researchers in meeting a variety of goals, such as devel-
oping broadly-based theoretical understandings,
cataloging sets of findings, developing measures,
conducting comprehensive assessments of the state of
a body of literature, forming principles to guide
in-session practice, or examining the methods and
methodologies employed within a field. Many of these
goals are similar to those of quantitative meta-analyses.
Indeed, qualitative and quantitative researchers
often examine the same exact questions (e.g., why
is psychotherapy effective?). This overlap is genera-
tive as methodologists with expertise in quantitat-
ive, qualitative, and mixed methods have long
argued the deliberate varying of methodological
and epistemological approaches enhances research
programs and deepens understanding (e.g., Levitt,
Surace, et al., 2017; Shadish, 1986). Accordingly,
researchers from any perspective can strengthen
their framing of research questions by consulting
both qualitative and quantitative primary and
meta-analytic research in their literature reviews.
In this process, researchers will be aided by an
understanding of the logic of design that is

Correspondence concerning this article should be addressed to Heidi M. Levitt, Department of Psychology, University of Massachusetts
Boston, Boston, MA, USA. Email heidi.levitt@umb.edu

© 2018 Society for Psychotherapy Research

2 H. M. LEVITT
particular to qualitative methods and in which rigor
is associated especially with the strengthening of
inductive processes.
The current paper provides an overview of the
history of qualitative meta-analytic methods in psy-
chology. Varied forms of qualitative meta-analyses
that have been put forward, each entailing prescribed
sets of procedures (Barnett-Page & Thomas, 2009;
Timulak, 2013). Instead of recommending the adop-
tion of any one method whole cloth, however, the
current paper advocates for considering how pro-
cedures can best be selected and tailored (potentially
from across forms of meta-analysis) to allow research-
ers to meet their specific study goals. To provide gui-
dance on navigating this process, the author reviews
the steps of meta-analyses and considers critical
researcher decisions using the framework of meth-
odological integrity, which has been developed to
guide the adaptation of procedures in relation to the
logic of qualitative research.
The Development of Qualitative
Meta-analysis
Neither qualitative research, nor meta-analytic research
is new to either psychology or psychotherapy research-
ers (Wertz, 2014). Qualitative research methods have
become systematized only over the last half-century
as their procedures have come to be formally expli-
cated, with phenomenological (Giorgi, 1970) and
grounded theory researchers (Glaser & Strauss, 1967)
as early forerunners in this processes. These methods
began to catch on gradually in psychological research
and leading psychotherapy researchers were at the van-
guard of this turn, such as Robert Elliott, Clara Hill,
John McLeod, and David Rennie. They introduced
and popularized qualitative methods in psychology,
and established early standards for their evaluation
(e.g., Elliott, Fischer, & Rennie, 1999).
Qualitative meta-analyses began to emerge a few
decades later in nursing, sociology, and education
(e.g., Noblit & Hare, 1988; Paterson, Thorne,
Canam, & Jillings, 2001; Sandelowski, Docherty, &
Emden, 1997), although they remain relatively rare
in psychology. Researchers using multiple terms to
signify this form of review may have compromised
the visibility of this method (e.g., meta-ethnography,
meta-study, meta-synthesis). Although these
approaches all aggregate qualitative findings, they
each recommend differing sets of procedures. In
this article, I retain the term qualitative meta-analysis
as it is better recognized in both psychology and psy-
chotherapy research.
Ladislav Timulak spearheaded this method in the
field of psychotherapy research and contributed a
number of qualitative meta-analytic reviews (e.g.,
Timulak, 2007 on helpful therapy processes). Other
qualitative analyses have followed by multiple
research teams, focusing on issues such as experi-
ences of master therapists (Jennings et al., 2016),
the training of psychotherapists (McGillivray,
Gurtman, Boganin, & Sheen, 2015) and computer-
ized psychotherapy (Knowles et al., 2014). Levitt,
Pomerville, and Surace’s (2016) omnibus meta-
analysis reviewed psychotherapy studies with varied
foci.
Although direction on how to conduct qualitative
meta-analyses has come from outside psychology
(e.g., Sandelowski et al., 1997), Timulak (2009)
wrote a helpful article describing his meta-analytic
method with a focus on psychotherapy, and in
2013, he co-authored a paper on this topic with his
colleague, Mary Creaner. The current paper builds
on this work by discussing how to tailor meta-ana-
lyses for varied purposes and how to strengthen
methodological integrity within the meta-analytic
process.
Methodological Integrity
Methodological integrity is a concept put forward by
a task force of the Society for Qualitative Inquiry in
Psychology (SQIP; Levitt, Motulsky, Wertz,
Morrow, & Ponterotto, 2017) that generated rec-
ommendations for both designing and reviewing
qualitative research. The goal of this work was to
identify the methodological foundation for trust-
worthiness in qualitative research to provide a
clearer understanding of the underlying logic within
qualitative research design. In contrast to a cookbook
approach to research evaluation that constrains
research design by mandating adherence to estab-
lished methods, this framework was developed to
provide guidance to reviewers on how qualitative
research methods could best be adapted given a
study’s research goals, approach to inquiry or epis-
temology, and characteristics. Input was incorpor-
ated from researchers from across a breadth of
qualitative methods and perspectives to ensure the
framework’s capacity to be applied across
approaches. Recently, APA Style, which determines
the research standards for both the Publication
Manual of the APA and APA Style Central, has used
the concept of methodological integrity to guide
authors, reviewers, and editors in reporting primary
qualitative research, mixed methods, and qualitative
meta-analyses (Levitt et al., in press). Given its
broad applicability, the framework of methodological
integrity is applied in the current paper to the design-
ing of qualitative meta-analyses (Table I).
Table I. Conceptual issues and recommendations for designing a qualitative meta-analysis.
Issue
(1) Identifying and describing primary studies.
(2) Transforming primary research findings
into initial units of data in the meta-analysis.
(3) Organizing initial units into categories or
themes.
(4) Increasing Methodological Integrity
Question
How comprehensive does my search need to be?
How should I consider the quality of primary articles?
How should I consider the fit of primary articles with
the research question?
How should I label my initial units reflecting the
primary findings to facilitate their use in the meta-
analysis?
How many categories or themes should I create?
How should I label categories?
Should I use frequency counts of findings?
How can I review large numbers of studies?
Should I examine the methods used and their
influence upon the findings?
Should I restrict my literature to certain features (e.g.,
therapy orientation or diagnoses)
What procedures should I consider when using meta-
analyses to develop measures?
How should I manage my expectations, perspectives,
and assumptions in the analysis and write up?
How should I examine the expectations, perspectives
and assumptions of the primary researchers?
How should findings be discussed to enhance
methodological integrity?
Methodological recommendation
Consider whether your goal is to review the complete literature base or to collect enough
primary studies to reach a saturated set of findings or model that answers your question.
Consider restricting data collection to published studies, using methods that rate quality,
or coding methodological features.
Seek a diversity of studies within the scope of your research question or adjust your
question to fit the available studies. Articulate justifications for the fit between types of
diversity included or coded in the studies and your question. Be transparent in
describing your procedures for assessing fit (inter-rater coding, etc.).
Using the primary research labels is helpful so long as the connection to your research
question is clear, and the phrasing allows you to make connections across other studies.
If not, re-label findings to do so.
Consider what would be useful in the context of your goal (an article, a training, a book,
etc.).
Generate category labels that are descriptive and specific enough to serve as meaningful
answers to your research question. Keep labels grounded in the distinctions within the
data.
Reconciling conflicting answers will increase the coherence of your findings.
The use of frequency counts can be used to indicate the salience of findings or focus of the
researchers, rather than their validity or the proportion of cases in which they are
experienced.
The use of hybrid qualitative methods has been found helpful, in which researchers
develop an attuned saturated model using one qualitative method (e.g., grounded
theory, discourse analysis, thematic analysis) and then use content analysis to sort the
rest of the database into that model (maintaining the ability to adjust the model as
needed).
Examining methods can be an important function of meta-research but it may be
challenging to merge a meta-method study and a meta-analysis of findings within one
article. Researchers may need to write a separate article to examine methods and their
effect more closely.
Restricting your literature review to focus on certain features can allow for
recommendations that are tailored to that context. It can provide insight into dynamics
that are localized and can challenge understandings that are based more broadly.
Using methods that organize data in a hierarchical format can be helpful when developing
Psychotherapy Research 3
measures because this form suggests both items for an initial scale as well as factors with
which they might be associated in exploratory factor analyses.
Qualitative meta-analysts value being aware of their own expectations and being
transparent in the process of reporting findings. This transparency can strengthen
methodological integrity.
Researchers can examine the self-reflective statements of researchers, the perspectives
revealed by their questions or findings, or search authors’ other publications for
identification of their perspectives.
Findings that are contextualized appropriately, that generate new insight into a
phenomenon, that are coherent with one another and that lead to a meaningful and
useful contribution will be higher in methodological integrity.
4 H. M. LEVITT
There are two central components in the estab-
lishment of methodological integrity; both have
been conceptualized as having four central features.
The first process, fidelity to the subject matter is the
process by which researchers select procedures
that increase their findings’ allegiance to the
phenomenon under study. It is strengthened when
researchers collect data that includes variations in
the phenomenon that generate a more comprehen-
sive understanding of their research question (data
adequacy); when researchers recognize and are
transparent about the influence of their own per-
spectives and appropriately limit that influence
within their data collection (perspective management
in data collection); when they consider how their
own perspectives influenced or guided their analytic
process with the aim of strengthening their percep-
tiveness (perspective management in data analysis);
and when findings are rooted in the data analyzed
(groundedness).
The second process, utility in achieving research
goals is the process by which researchers select pro-
cedures that usefully answer their research questions
and address their aims (e.g., developing theory, dee-
pening understanding, raising critical consciousness,
identifying social and discursive practices). Utility is
strengthened when findings are considered in
context—relaying, for instance, their location, time,
and cultural situation (contextualization of data);
when data collected are rich enough to support
insightful analyses (catalyst for insight); when analyses
lead to insights that address the researchers’ ques-
tions and goals (meaningful contributions); and when
conflicting findings are reconciled or explained
(coherence among findings).
A central characteristic of the framework of meth-
odological integrity is that fidelity and utility are
conceptualized in relation to the researchers’ goals,
approaches to inquiry (e.g., philosophical assump-
tions), and study characteristics (e.g., the particular
subject matter, resources, participants, researchers).
Many examples can be provided of times when these
features could lead to changes in procedures. For
instance, interviewing procedures might change
when interviewing vulnerable participants to maxi-
mize the fidelity of the ensuing data. The types of
diversity sought in participants might shift in
relation to the forms of diversity that influence a
given topic to enhance the data adequacy. Or, a
critical qualitative approach would demonstrate
utility in relation to the goals of this approach
(e.g., social change, consciousness raising) in order
to lead to a meaningful contribution. In other
words, the fidelity and utility of procedures need
to be evaluated in relation to their function in the
research design and aim.
General Process of Meta-analysis
Many forms of qualitative meta-analytic methods
have been developed (e.g., meta-ethnography,
Noblit & Hare, 1988; meta-study, Paterson et al.,
2001; meta-summary, Sandelowski & Barroso,
2003). These methods prescribe varied sets of pro-
cedures, with some emphasizing cataloging findings,
developing new findings, conducting a critical analy-
sis of findings, or assessing the quality of findings (see
Barnett-Page & Thomas, 2009; Timulak, 2013 for a
detailed review). In addition, many qualitative
methods can be adapted to generate a meta-analytic
approach by using primary findings in place of raw
data (e.g., Kearney’s, 1998 grounded formal theory).
Before describing how to adapt qualitative meta-
analytic procedures to the goals of a given study,
the general processes that form the foundation of
most forms of qualitative meta-analysis will be
reviewed. I will draw from some of my own research
collaborations to provide examples of decision
points. This review will consider how to best adapt
procedures using the framework of methodological
integrity.
Identifying and Describing Primary Studies
The process of identifying studies for a qualitative
meta-analysis may be similar to the process of a quan-
titative meta-analysis. Typically, researchers will
need to use electronic databases (e.g., PsycINFO)
to locate the primary studies. Researchers will ident-
ify keywords and terms that will assist them in finding
studies that are relevant to their question. In their
reporting, they will identify the databases searched,
the time period searched, the keywords used, and
the languages of studies reviewed. Researchers may
wish to use a flow diagram to demonstrate their
process of identifying studies (e.g., Moher, Liberati,
Tetzlaff, & Altman, 2009; see Swift & Wampold,
2018 for a discussion of how this process in quantitat-
ive meta-analyses).
An exception can occur when the researcher’s goal
is to develop a new understanding or theory of the lit-
erature and plans to halt the collection of primary
studies once saturation is reached (e.g., Kearney,
1998). In this case, the researchers might not need
to locate all the research in a literature base but
enough to satisfy that goal. (For a detailed description
of this process and its tie to utility, see the forthcom-
ing section on Developing an understanding of find-
ings versus a description of an entire literature.)
Considering quality. As search terms often are
insufficient in assessing the eligibility of the primary
studies in light of a specific research question,

researchers will need to describe how they narrowed
down the studies reviewed. To do this, researchers
usually consider two issues—the fit of the primary
study to the meta-analytic study question and the
quality of the study. When considering the quality of
a study, a key decision is whether unpublished
research will be retained. Including only published
qualitative research can be a helpful form of quality
control (Timulak, 2009).
Criteria have been developed to evaluate the
quality of qualitative research for systematic reviews
(e.g., Critical Appraisal Skills Programme, 2017;
Paterson et al., 2001), however, characteristics of
these systems tend to be quite basic and are overwhel-
mingly present in published research (e.g., Was there
a statement of research aims? Was a qualitative meth-
odology appropriate?). A few meta-analytic
approaches (e.g., framework synthesis, thematic nar-
rative analysis, and thematic synthesis) include within
them guidance for rating the research quality of the
primary studies, for instance by assessing their ration-
ale, methods, and findings. These evaluations can
lead to the exclusion of studies if these ratings do
not meet an acceptable score. Because these
methods exclude certain studies, however, this
process would not permit researchers to conduct an
evaluation of the methodological state of the litera-
ture on a whole. Also, if researchers are interested
in critiquing the use of methods, excising methodolo-
gically problematic articles would not permit a com-
prehensive picture of the field and the analysis
would be limited. Also, a challenge in evaluating pub-
lished research is that the page limits for most jour-
nals have been developed for quantitative research
and may prohibit a full description of the methods
used, making it challenging to be certain of the meth-
odological rigor as procedures may go unreported.
In my experience, qualitative meta-analyses are
robust to the occasional weaker study as poor quali-
tative research findings tend to be characterized by
being less descriptive and vague. Because of this,
their findings can be interpreted in light of other
studies that display more insight and acuity within
the meta-analytic process. Limiting the primary
research to studies that have passed the test of peer
review then may be sufficient quality control.
Across forms of meta-analysis, researchers may wish
to document research quality by coding the specific
design features that enhance integrity in the literature
or by rating of the eight components of methodologi-
cal integrity (see Levitt, Surace, et al., 2017 for ques-
tions and principles that can guide this evaluation).
Considering fit. When evaluating the fit of
primary studies to the meta-analytic research
Psychotherapy Research 5
question, there are several things to keep in mind.
In qualitative research, greater diversity of partici-
pants within the scope of a question tends to be
helpful as it allows for a more comprehensive under-
standing of how a phenomenon is experienced across
a variety of people, improving the fidelity of a study
(Levitt, Surace, et al., 2017). Researchers will want
to establish the scope of the question that they are
asking and whether there is enough of a primary lit-
erature to support data adequacy. That is, is there a
number of and diversity of primary studies to
provide an answer to the researchers’ question in its
current form? If not, they could describe this as a
limitation of the research and direct future research-
ers to filling in that gap. For instance, if research on
insight in therapy had been conducted mostly from
the perspective of one type of client (e.g., depressed
clients), this would be a strong limitation to making
general claims.
Because the meta-analytic researchers are not
recruiting clients themselves, they may have to
adapt their question to the characteristics of the
data available in the primary studies for their
answers to have methodological integrity. For
instance, in the preceding example, it would be
appropriate for the researchers to narrow the scope
of their study question to insight within depressed
clients. Although researchers will begin with a
general study question, as they code the primary
research studies, they may shape their study question
to be descriptive of the primary data. Researchers
may wish to consider the cultural backgrounds of par-
ticipants, their presenting issues, their diagnoses,
their professional training, and other features that
are relevant to their research topic.
When designing inclusion and exclusion criteria to
remove studies from the initial returns from a search
for primary research, providing a justification of these
criteria in relation to the scope of the question will
help reviewers and readers make sense of the
decisions being made. For example, in our study on
clients’ experiences of psychotherapy, we excluded
case studies that focused upon one client (Levitt
et al., 2016). We explained that case study research
would be challenging to include both because our
question focused on typical experiences and case
studies often are used to represent more idiosyncratic
experiences.
When reviewing initial returns from a search, it is
likely that most of these studies can be culled based
upon the title and abstract of papers alone. Research-
ers will want to record the numbers of articles that
were eliminated within the examination of the title
and abstract versus full text. If researchers use mul-
tiple judges to make these determinations, a typical
process in quantitative meta-analyses, they also will
6 H. M. LEVITT
report the numbers of judges, and how agreement
was indicated or disagreement was resolved.
In generating a description of the primary studies
selected for analysis, researchers may or may not
want to use multiple judges to check each other’s
coding depending on how challenging that infor-
mation is to identify within a study. Typically,
researchers will identify features of each article that
relate to the context of the paper (e.g., the research-
ers’ location, the setting of the research, the psy-
chotherapy orientations examined) and features that
describe the methodological processes (e.g., the
numbers of participants, the central research ques-
tion, data collection methods, data analytic
methods, epistemological perspectives, the checks
on methodological integrity used). For instance,
Levitt, Pomerville, Surace, and Grabowski’s (2017)
meta-method study on the literature on clients’
experiences of psychotherapy described the use of
checks that included auditors, inter-rater reliability,
consensus, saturation, reflexivity checks, participant
checks, and triangulation. The recommendation in
identifying and describing the primary studies is to
consider the function of the procedures and
whether they strengthen the transparency of reporting
and the methodological integrity of the design with
regard to the goals of the meta-study.
Conducting Meta-analyses
After the primary literature is in hand, the central
processes in meta-analytic methods include generat-
ing units from the primary findings and labeling
them, creating categories based upon commonalities
and distinctions in the meanings within these units,
and then examining the relationship between these
categories to develop a central finding or a set of
main findings.
Generating units and labels. A first step in
qualitative meta-analysis is to form initial units that
each describe a finding from the primary research
and assign it a descriptive label. I find it helpful to
follow Rennie’s (2000) approach in borrowing from
phenomenology the process of generating meaning
units (e.g., Giorgi, 2009) that each describe one
central meaning. A finding in a primary study may
lead to more than one unit, if they include multiple
meanings that are relevant to the subject in the
meta-study. In this process, the groundedness of
units can be increased by remaining close to the orig-
inal researchers’ ideas as possible when labeling units,
but shaping them to answer the central question of
the meta-analysis as directly as possible (and findings
that are not relevant are excluded). Introducing
extraneous units and categories into an analysis can
make the process of developing findings confusing
and including irrelevant data can weaken the emer-
ging findings by compromising fidelity.
In the labeling of primary units, I often use the
primary researchers’ category or thematic titles
directly or with small clarifying amendments. This
is not always possible, however. Some qualitative
methods do not present their results via category
titles but use narrative descriptions instead. Also,
sometimes researchers form the category titles that
represent the research questions rather than the
answers—a practice that I discourage. For instance,
a study on clients who have interpersonal conflict
might have categories called “interpersonal conflict
in intimate relationships,” “interpersonal conflict in
the workplace,” and “interpersonal conflict in
therapy.” These category titles reflect the question
of the researcher about the contexts of interpersonal
conflict but they do not reveal anything substantive
about the character of the conflict in any of these set-
tings. In both these times, I will form my own labels
by summarizing the text to provide meaning unit
labels that distinguish the ways the phenomenon
transpires in the contexts.
Labels can be written in such a way to permit both
common and unique meanings to be identified across
studies. For instance, if one study describes tolerating
fear in a behavioral exposure intervention, a second
portrays the emotion-focused process of resolving
sadness during a focusing exercise, and a third
describes the experience of hypnosis to re-experience
stressful early experiences, labeling these units “be-
havioral exposure is effective,” “focusing resolves
depression,” and “hypnosis reduces stress” will lose
the common processes of engaging with emotions
within these interventions. It would be preferable to
avoid overreliance on the intervention labels and
instead articulate this common experience. For
instance, “behavioral exposure helps clients engage
with fear, contributing to their resolution,” “focusing
exercises help clients maintain contact with sadness
to alleviate it,” or “hypnosis structures engagement
with early stressors.” These labels that emphasize
process are more likely to be meaningfully grouped
together. For this reason, it can be helpful to avoid
jargon to increase the likelihood of a developing
insightful data that can lead to new understandings.
Creating categories and higher order
categories. The labeled units become the foun-
dation for categories (sometimes called themes or
simply findings) that describe patterns within the
meta-analytic findings. The meaning units from one
study, each summarizing a central finding, are
 Psychotherapy Research 7

compared to the units from the other studies in turn. I (6) Feeling and discussing sadness with the
conceptualize this process as akin to constant com- therapist led to feeling connection.
parison, in which each unit is compared with every
Grouping units 1, 3, and 5 into a category called,
other unit to identify points of similarity and differ-
“Sadness is scary because it is overwhelming and
ence, which then become the basis for categories
might lead to therapist judgment,” and grouping
that group those units together (Glaser & Strauss,
units 2, 4, and 6 together into a category called,
1967).
“Feeling sadness in session increases engagement, con-
Depending on the form of analysis in use, these
nection and acceptance” can be a crucial preliminary
categories may become the central findings in the
step in considering how to reconcile the meanings
meta-analysis, or the initial categories created may
within the units in an overarching category title that is
be subjected to a similar process of mutual compari-
more meaningful and grounded than simply saying
son in which they are grouped together to identify
that clients feel sadness in therapy. One such title in
broader, more encompassing themes. This process
our meta-analysis of clients’ therapy experiences
of seeking to integrate higher order categories may
(Levitt et al., 2016) was, “Fear of sadness and vulner-
continue until the set of findings appear to be convey-
ability prompts disengagement but experiencing and
ing discrete ideas that hold utility in light of the
exploring these emotions in therapy enhances engage-
research question at hand. Typically, it is challenging
ment and leads to acceptance” (p. 818). The reconcil-
to present more than eight main findings in a journal
ing of conflicting clients’ reactions guides therapists on
length article so when there are many initial cat-
how to intervene when they see clients avoiding their
egories, it can be especially helpful for researchers
emotions in session and demonstrates greater utility
to group them together so they become more man-
than a general vague category title. Being as specific
ageable to present. Alternatively, researchers may
as possible in terms of contexts and processes within
need to divide their analysis into separate articles
the findings is useful in this regard.
that focus on discrete sets of findings. For instance,
Third, when there are conflicts within the data, I
a meta-analysis might result in one paper that is
work to reconcile them in order to increase the coher-
focused upon the review of findings and a second
ence of my findings. For instance, if one finding
meta-method article focused on the use of qualitative
shows that clients do like when therapists have pro-
methods in a field of study.
fessional credentials but another states that they
Similar to the formation of initial unit labels, the
also are suspicious of credentials, therapists reading
categories are assigned labels to capture the pattern
the article will be uncertain how to draw guidance
observed in that set of data. There are a few tips
from that study. Conflicting findings allow research-
that can be helpful at this point. First, these labels
ers to identify patterns that may not have been articu-
are best when they make clear their connection to
lated before. To do this, researchers may ask when,
the central meta-analytic study question, for the
under what conditions, why, or contingent upon what dis-
reasons described previously. Second, when there
tinctive responses patterns hold. For instance, in this
are many units grouped together within a category
same meta-analysis, we developed the principle,
(>8), it can be useful to check if there are subcate-
“Professional structure creates credibility and clarity
gories that could be created to articulate the distinc-
but casts suspicion on care in the therapeutic
tive meanings among the units. The articulation of
relationship” (p. 820), to explain the varied functions
these common meanings in subcategory titles might
of credentials. By reconciling conflicting meanings,
inform the evolving broader category title. For
researchers enable more sophisticated theoretical,
instance, consider the following unit labels that
clinical, and research implications and increase the
might appear under a provision category title
utility of their research findings.
“Sadness in therapy”:

Identifying Goals and Tailoring

(1) Clients expressed worry about feeling over-
whelming sadness in therapy.
(2) Clients felt that exploring sadness in therapy
helped them accept their emotions.
(3) Clients felt vulnerable when sad topics arose
and tried to avoid the topic.
(4) Clients felt more engaged in sessions when
they got in touch with sadness.
(5) Clients worried that their therapist would be
critical of them if they cried.
Meta-analysis
While the previous section focused on processes that
are common across most forms of meta-analysis, this
present section is concerned with the tailoring of pro-
cedures to specific research goals. There are a variety
of reasons why one might wish to conduct a meta-
analysis, including the development of a new under-
standing, a need to reconcile conflicts in the litera-
ture, the identification of central findings in an
8 H. M. LEVITT

entire literature, the development of an outcome may extend deeper than those in the primary litera-
measure or a process measure, the desire to raise ture (e.g., Ma, Roberts, Winefield, & Furber,
critical consciousness about shortcomings or biases 2015). They do not need to analyze an entire litera-
in a literature, and an interest in reviewing the meth- ture, however, to develop and offer new understand-
odological soundness of a set of findings or the use of ings. Investigators being clear on their study aim will
a certain method. Research goals will be reviewed in be helpful as they set forth to design their meta-analy-
relation to adaptions that can be made to facilitate sis. This goal may be influenced, in part, by the
that goal and strengthen methodological integrity. number of primary studies available for analysis. A
distinction of qualitative meta-analysis is that the
numbers of primary studies can vary widely from 2
Cataloging Types of Findings versus to 100, with 12 being an average number (Timulak,
Re-analyzing Them 2009). Many psychotherapy qualitative meta-ana-
lyses seem to examine approximately 7–10 studies
When researchers describe meta-analyses, they typi-
(e.g., McGillivray et al., 2015; Timulak, 2007;
cally are referring to processes in which primary find-
Timulak & Creaner, 2010), which may well exhaust
ings are subjected to a synthesizing process of
the research that has been conducted in an area.
secondary analysis. In contrast, Sandelowski and
If the literature contains a large enough number of
Barroso (2003) have developed a method, qualitative
studies, however, it creates the possibility that
meta-summary, which collates findings into descriptive
researchers seek to develop a saturated understand-
groups but does not re-interpret them. Instead, it
ing of that literature in their meta-analysis. Saturation
guides researchers to consider the frequency of findings
is the point at which new data being analyzed ceases
across studies and to credit findings that appear more
to add new understandings to the analytic scheme
often with higher validity within an “effect size” that is
already developed, suggesting that the analysis is
“verifying the presence of a pattern or theme”
comprehensive (Glaser & Strauss, 1967). To
(p. 231). The use of frequency as a proxy for validity
provide an example, in our meta-analysis on clients’
in qualitative research can be seen as problematic,
experiences of psychotherapy (Levitt et al., 2016),
however (see Levitt, Butler, & Hill, 2006 for a descrip-
saturation was reached at 47 studies, as tested by
tion of this problem in primary qualitative research).
the addition of 20 more studies into the analysis
The concern with this calculation is that, even
that did not generate new categories. This could be
within the same area, researchers may not have the
a rationale to end the collection of primary research
same interests, ways of phrasing questions, capabili-
as the findings meet the research goal of developing
ties as researchers, or priorities in terms of generating
new understandings of the literature—even if all the
thematic foci. As a result, it can be that a finding
primary studies were not reviewed.
found only in one study, but by a highly attuned
When there is large number of studies and
research team, underlies and meaningfully explains
researchers wish to review them all, they might con-
superficial findings that are identified frequently
sider using a combination of methods. When we
often across other studies. Also, research can be con-
first conducted this analysis, we had planned to halt
ducted at times and regions when certain expla-
data collection at the point of saturation (reviewing
nations become more prevalent and accessible, even
67 studies) and submitted our paper to Psychological
though they might always be at play. For instance,
Bulletin. Feedback from reviewers, however, indi-
descriptions of fear of the government by immigrant
cated that they wanted our review to include all of
clients might emerge under one president and not
the research related to clients’ experience of psy-
under another, but its inconsistency in time does
chotherapy. A systematic search of the literature
not make the finding less valid. Or participants
revealed that there were 109 studies that met our
might agree with ideas that they fail to describe
inclusion criteria. Because we were using a grounded
because they think they are obvious, not relevant, or
theory meta-analytic approach, it would have been
not as pressing as other concerns. For these many
prohibitive to conduct constant comparison with all
reasons, the use of frequency to indicate the validity
the additional studies. As a result, we developed a
or representativeness of a finding within a population
hybrid method in which we used the categories that
(versus within a set of studies) is questionable.
we had developed in our grounded analysis that
were highly attuned to clients’ experiences and then
conducted a content analysis with the remaining lit-
Developing an Understanding of Findings
erature by sorting the unitized and labeled findings
versus a Description of an Entire Literature
into the already-established categories. Although we
Qualitative meta-analyses have been advocated for were open to adjusting the category labels to reflect
their ability to develop novel interpretations that incoming data, the hierarchy remained relatively
 Psychotherapy Research 9

stable. This additional analysis permitted us to been conducted, for instance, specifically examining
characterize a broader research literature and to research that is focused within specific orientations
speak more confidently about the prevalence of or diagnoses. It still can be challenging to identify
trends in our findings. The adoption of a similar enough research to conduct these meta-analyses.
hybrid approach to analysis might make feasible Within the research on clients in psychotherapy, for
other comprehensive reviews of large literature bases. instance, the vast majority of the primary qualitative
research studies appear to include clients from
across psychotherapy orientations (73.4%; Levitt
Reviewing Content versus Methods Being et al., 2016). This research can provide valuable
Used information, however, on processes and goals that
are associated within identified therapy approaches.
While some meta-analysts are interested primarily in
For instance, Timulak and Creaner (2010) examined
the content of the primary findings, others may be
8 qualitative studies on humanistic therapies and
more interested in examining how qualitative
identified 11 types of client-reported change. These
methods are used. A number of established methods
included a variety of outcomes, such as healthier
were developed to examine the use of both findings
emotional experiencing, experiences of appreciating
and methods within one qualitative approach, such
vulnerability, changed views of others, and empower-
as meta-ethnography to examine ethnographic
ment. This focused research usefully identified
studies (Noblit & Hare, 1988), and grounded formal
change processes that are not typically assessed
theory to review grounded theory studies (Kearney,
using standardized psychotherapy outcome measures
1998). While these methods can be adapted to
(Levitt, Stanley, Frankel, & Raina, 2005) and lead
review a wide range of methods, the parallel structure
toward a better understanding of how change
between the meta-analytic method and primary
unfolds within a treatment.
methods can facilitate a review.
Paterson et al. (2001) described a meta-study
process that incorporates a meta-analysis that
reviews a literature’s findings, a meta-theory that Process or Outcome Measure Development
reviews the theory and a meta-method that reviews
Within mixed methods programs of research, quali-
the methods used and considers how the methods
tative meta-syntheses can be used to develop
used to influence the findings reported. The meta-
measures that are grounded in a literature base.
method component can be used independently,
When developing a process measure, findings from
however, to examine how multiple methods are
a study can help researchers to identify the central
applied within an area of research. This approach to
change processes related to the resolution of types
research can lead to informative findings that can
of therapy events across primary studies. They also
advance qualitative methods in a given area. For
could lead to the development of typologies that
instance, a meta-method study indicated that it was
can help identify the internal processes that are
commonplace for qualitative psychotherapy research-
unfolding within a session. For instance, Li, Black,
ers to creatively integrate procedures from one
and Garland (2016) used a mixed method approach
method to enhance another approach, suggesting
including qualitative interviews, scale validation,
that reviewers’ evaluations should focus on methodo-
and a replication study to develop a process
logical integrity within a study rather than its adher-
measure of how mindfulness can be used to alleviate
ence to a given method to reflect this practice
psychological suffering. This process allowed them to
(Levitt, Pomerville, et al., 2017). Meta-method
begin their measure development with a set of
studies also can direct researchers toward methods
attuned items that conveyed the experiences they
that rarely are used but that might make a significant
were studying with fidelity.
contribution in a field. Because qualitative studies are
When developing an outcome measure, it can be
still relatively new to the field of psychology, examin-
useful to utilize a method of qualitative analysis that
ations of how they are being conducted can commu-
will result in at least two levels of categories, which
nicate and shape standard research practices.
can assist in identifying items that might be associ-
ated with one another in subscales. Grounded
theory meta-approaches can be particularly useful
Reviewing Research across or within a
in this respect as the hierarchical data structure
Certain Context
makes it possible to identify a large number of
Sometimes researchers’ goals focus on primary items that can then be reduced within a factor analy-
research conducted within a specific context or sis. For instance, Levitt identified a set of 50 items
setting. Some qualitative meta-analytic research has from a grounded-theory driven meta-analysis
10 H. M. LEVITT
(Levitt et al., 2016) and then, with colleagues Gra-
bowski and Minami (2018), developed a 15-item
measure to capture the change processes most criti-
cal to psychotherapy clients based upon an explora-
tory factor analysis. The advantage of developing
items from meta-analyses is that they can be both
robust, as they are confirmed across a number of
studies, and comprehensive, as they broadly reflect
the internal experiences of participants from
across a range of primary study questions and inves-
tigator lenses. Reflecting the qualitative research
from which they derive, they also permit the
measurement of phenomena that are challenging
to evaluate because they may be subjective, covert,
and/or context-dependent.
Additional Thoughts on Strengthening
Methodological Integrity
Through this paper, guidance has been provided on
how to use processes to improve methodological
integrity in meta-analyses and how to adapt pro-
cedures in relation to researchers’ goals. As
described, to strengthen the fidelity of their findings
with the primary literature, meta-analysts will wish
to consider the adequacy of their data in relation to
the scope of their research question. They can
increase the groundedness of their findings by generat-
ing units and initial categories that are faithful to the
primary findings under analysis. There are two com-
ponents of fidelity that have not yet been discussed,
however, which bear consideration.
The first is the management of the researchers’ per-
spectives during data collection. Throughout the
course of qualitative meta-analyses, researchers
being aware of their hopes and expectations can
allow them to consider how to collect data in a way
that tests these expectations rather than build upon
their assumption. Researchers may use a variety of
methods to assist them in keeping this awareness
forefront. These include using techniques such as
memo-ing, note-taking, and discussing expectations
and how to limit their effects within a research
team. In addition, meta-analysts strengthen fidelity
by managing their perspectives during data analysis.
This might occur in the same manner as during
data collection, but researchers also consider how
they employ viewpoints to increase the perspicacity
of their analyses. For instance, researchers conduct-
ing theory-driven analyses or using critical epistem-
ologies (e.g., feminist-multicultural approaches)
might instead deliberately use a certain theoretical
perspective to structure their analysis. Regardless of
whether researchers seek to limit or apply a perspec-
tive in their analysis, they increase fidelity by being
transparent about this process so that their findings
can be read in light of their analytic approach.
In addition to these forms of methodological integ-
rity, which also relate to primary data analysis, there is
a form that relates specifically to meta-analyses. This
is the consideration of the perspectives of the primary
researchers. Because many primary qualitative
research studies include reflexive statements reveal-
ing the positions or attitudes of investigator with
respect to the research question, meta-analytic
researchers can analyze this information. A problem
though is that, although many primary studies
(80.7% in one meta-analysis of psychotherapy
research; Levitt et al., 2016) contain some reflexive
statement by the researchers, far fewer (27.5% of
that literature) described how they adjusted
methods in response to their personal hypotheses.
Researchers can examine other information as well
such as the perspectives being revealed by research
questions being asked (or not asked), or biases
present in research presentations. To provide an
example of a study that examined the assumptions
of primary researchers, in that same meta-analysis
on clients’ experiences, the researchers found that
the primary researchers tended to ask questions
about either professional power or cultural power
but quite rarely about the intersection of the two—
revealing that researchers appeared to be motivated
by one interest or the other, but tended to ignore
the intersection of these types of power—generating
a limitation in the literature. Another way meta-ana-
lysts might examine researcher perspectives is to
adopt the approach used in quantitative analyses of
investigator allegiance in which publications are
searched for identification of authors’ expertise in
or bias toward a specific hypothesis (e.g., Munder,
Gerger, Trelle, & Barth, 2011; Spielmans & Flücki-
ger, 2018).
Through this paper, procedures that support the
utility of meta-analytic findings have been indicated
as well. Contextualization can be useful when present-
ing quotes, when describing the location and time-
period of the data collection in the primary literature
in the method section, and when describing limit-
ations of the meta-analysis in the discussion section.
By contextualizing their research, they assist readers
to better apply findings in their own contexts.
Researchers are more likely to generate insight when
they avoid jargon and code primary findings in such
a way that they can be compared meaningfully
across studies. By reconciling conflicting findings,
meta-analysts can create coherent findings that can
provide clarity in areas that had been complex and
confusing. Also, by generating meaningful contri-
butions, in relation to the research goals that are set
forth, meta-analytic researchers can provide

enlightening information to address a wide variety of
aims. Rather than confining their research design to
predetermined sets of procedures, by considering
these components of methodological integrity as
they design research, meta-analysts can adapt
methods to support their goals, just as reviewers can
use these ideas in the evaluation process.
Acknowledgment
The author thanks Zenobia Morrill for her comments
on a draft of this article.
References
Barnett-Page, E., & Thomas, J. (2009). Methods for the synthesis
of qualitative research: A critical review. BMC Medical Research
Methodology, 9, 1–11. Retrieved from https://www.ncbi.nlm.nih.
gov/pmc/articles/PMC3224695/
Critical Appraisal Skills Programme. (2017). CASP Qualitative
Checklist. [online] Available at: http://www.casp-uk.net/
checklists.
Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guide-
lines for publication of qualitative research studies in psychology
and related fields. British Journal of Clinical Psychology, 38, 215–
229. doi:10.1348/014466599162782
Giorgi, A. (1970). Psychology as a human science: A phenomenologi-
cally based approach. New York, NY: Harper and Row.
Giorgi, A. (2009). The descriptive phenomenological method in psy-
chology: A modified Husserlian approach. Pittsburgh, PA:
Duquesne University Press.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded
theory: Strategies for qualitative research. London: Wiedenfeld
and Nicholson.
Jennings, L., Sovereign, A., Renninger, S., Goh, M., Skovholt, T.
M., Lakhan, S., & Hessel, H. (2016). Bringing it all together: A
qualitative meta-analysis of seven master therapists studies from
around the world. In L. Jennings, T. M. Skovholt, L. Jennings,
& T. M. Skovholt (Eds.), Expertise in counseling and psychother-
apy: Master therapist studies from around the world (pp. 227–
273). New York, NY: Oxford University Press.
Kearney, M. H. (1998). Truthful self-nurturing: A grounded
formal theory of women’s addiction recovery. Qualitative Health
Research, 8, 495–512. doi:10.1177/104973239800800405
Knowles, S. E., Toms, G., Sanders, C., Bee, P., Lovell, K.,
Rennick-Egglestone, S., … Bower, P. (2014). Qualitative
meta-synthesis of user experience of computerised therapy for
depression and anxiety. Plos ONE, 9. doi:10.1371/journal.
pone.0084323
Levitt, H. M., Bamberg, M., Creswell, J. W., Frost, D., Josselson,
R., & Suárez-Orozco, C. (in press). Journal article reporting
standards for qualitative research in psychology: The APA pub-
lications and communications board task force report. American
Psychologist. doi:10.1037/amp0000151
Levitt, H., Butler, M., & Hill, T. (2006). What clients find helpful
in psychotherapy: Developing principles for facilitating
moment-to-moment change. Journal of Counseling Psychology,
53, 314–324. doi:10.1037/0022-0167.53.3.314
Levitt, H. M., Motulsky, S. L., Wertz, F. J., Morrow, S. L., &
Ponterotto, J. G. (2017). Recommendations for designing and
reviewing qualitative research in psychology: Promoting
Psychotherapy Research 11
methodological integrity. Qualitative Psychology, 4, 2–22.
doi:10.1037/qup0000082
Levitt, H. M., Pomerville, A., Surace, F. I., & Grabowski, L. M.
(2017). Meta-method study of qualitative psychotherapy
research on clients’ experiences: Review and recommendations.
Journal of Counseling Psychology, 64, 626–644. doi:10.1037/
cou0000222
Levitt, H. M., Pomerville, A., & Surace, F. I. (2016). A qualitative
meta-analysis examining clients’ experiences of psychotherapy:
A new agenda. Psychological Bulletin, 142, 801–830. doi:10.
1037/bul0000057
Levitt, H. M., Stanley, C. M., Frankel, Z., & Raina, K. (2005). An
evaluation of outcome measures used in humanistic psychother-
apy research: Using thermometers to weigh oranges. The
Humanistic Psychologist, 33, 113–130. doi:10.1207/
s15473333thp3302_3
Levitt, H. M., Surace, F. I., Wu, M. B., Chapin, B., Hargrove, J.
G., Herbitter, C., … Hochman, A. L. (2017). The Meaning of
Scientific Objectivity and Subjectivity: Methodologists’
Perspectives. Manuscript under review.
Levitt, H. M., Grabowski, L. M., & Minami, T. (2018). The client
critical experiences in therapy scale (CETS). Unpublished
manuscript, University of Massachusetts Boston.
Li, M. J., Black, D. S., & Garland, E. L. (2016). The Applied
Mindfulness Process Scale (AMPS): A process measure for
evaluating mindfulness-based interventions. Personality and
Individual Differences, 93, 6–15. doi:10.1016/j.paid.2015.10.027
Ma, N., Roberts, R., Winefield, H., & Furber, G. (2015). Utility of
qualitative metasynthesis: Advancing knowledge on the well-
being and needs of siblings of children with mental health pro-
blems. Qualitative Psychology, 2, 3–28. doi:10.1037/
qup0000018
McGillivray, J., Gurtman, C., Boganin, C., & Sheen, J. (2015).
Self-practice and self-reflection in training of psychological
interventions and therapist skills development: A qualitative
meta-synthesis review. Australian Psychologist, 50, 434–444.
doi:10.1111/ap.12158
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009).
Preferred reporting items for systematic reviews and meta-ana-
lyses: The PRISMA statement. Journal of Clinical Epidemiology,
62, 1006–1012. doi:10.1016/j.jclinepi.2009.06.005
Munder, T., Gerger, H., Trelle, S., & Barth, J. (2011). Testing the
allegiance bias hypothesis: A meta-analysis. Psychotherapy
Research, 21, 670–684. doi:10.1080/10503307.2011.602752
Noblit, G. W., & Hare, R. D. (1988). Meta-ethnography:
Synthesizing qualitative studies. Newbury Park, CA: Sage.
Paterson, B. L., Thorne, S. E., Canam, C., & Jillings, C. (2001).
Meta-study of qualitative health research: A practical guide to
meta-analysis and meta-synthesis. Thousand Oaks, CA: Sage.
Rennie, D. L. (2000). Grounded theory methodology as methodi-
cal hermeneutics: Reconciling realism and relativism. Theory &
Psychology, 10, 481–502. doi:10.1177/0959354300104003
Sandelowski, M., & Barroso, J. (2003). Creating metasummaries
of qualitative findings. Nursing Research, 52, 226–233.
Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/
12867779
Sandelowski, M., Docherty, S., & Emden, C. (1997). Focus on
qualitative methods. Qualitative metasynthesis: Issues and tech-
niques. Research in Nursing & Health, 20, 365–371. Retrieved
from https://www.ncbi.nlm.nih.gov/pubmed/9256882
Shadish, W. R. (1986). Planned critical multiplism: Some elabor-
ations. Behavioral Assessment, 8, 75–103. Retrieved from http://
psycnet.apa.org/record/1987-00075-001
Spielmans, G. I., & Flückiger, C. (2018). Moderators in psy-
chotherapy meta-analysis. Psychotherapy Research. doi:10.1080/
10503307.2017.1422214
12 H. M. LEVITT
Swift, J. K., & Wampold, B. E. (2018). Inclusion and exclusion
strategies for conducting meta-analyses. Psychotherapy
Research. doi:10.1080/10503307.2017.1405169
Timulak, L. (2007). Identifying core categories of client identified
impact of helpful events in psychotherapy: A qualitative meta-
analysis. Psychotherapy Research, 17, 310–320. doi:10.1080/
10503300600608116
Timulak, L. (2009). Meta-analysis of qualitative studies: A tool for
reviewing qualitative research findings in psychotherapy.
Psychotherapy Research, 19(4–5), 591–600. doi:10.1080/
10503300802477989
Timulak, L. (2013). Qualitative meta-analysis. In U. Flick (Ed.),
The Sage handbook of qualitative data analysis. (pp. 481–495).
Thousand Oaks, CA: Sage.
Timulak, L., & Creaner, M. (2010). Qualitative meta-analysis of
outcomes of person-centred/experiential therapies. In M.
Cooper, J. C. Watson, & D. Holledampf (Eds.), Person-centred
and experiential psychotherapies work (pp. 65–90). Ross-on-
Wye: PCCS Books.
Wertz, F. J. (2014). Qualitative inquiry in the history of
psychology. Qualitative Psychology, 1, 4–16. doi:10.1037/
qup0000007