Critical Analysis of the Stimpfl & Engberg Comparative Matrix Articles

David Comp
International Higher Education Consulting
http://ihec-djc.blogspot.com/

Stimpfl, J.R., & Engberg, D. (1997, Spring). What to know before you go:
Creating a comparison for research on study abroad programs. International Education
Forum, 17 (1), 7-21.

Stimpfl, J.R., & Engberg, D. (1997, Fall). Comparing Apples to Apples: An

integrated approach to study abroad program assessment. International Education Forum,
17 (2), 97-109.

As discussed in a previous assignment, this two-part study was one of three

studies from the late 1990’s that laid the ground work for my interest in researching study

abroad. Of the three studies/articles, Stimpfl and Engberg’s study has been the most

influential to my understanding of methods to researching study abroad and it has guided

my thinking and advocacy about future approaches to researching U.S. students studying

abroad.

Stimpfl and Engberg seek to identify problems in research studies on study abroad

and suggest a tool (Comparative Matrix) that researchers can use to aid their research and

evaluation of study abroad experiences and programs. For example, a student from

Loyola University of Chicago (LUC) who takes one class on the LUC Rome Campus for

two weeks during a summer, living with a U.S. roommate from LUC in a dormitory

setting and taught in English by LUC will have a very different educational experience

than a U.S. student (could be a LUC student) who studies full-time in Quito, Ecuador for

a semester or year, living with a host family or independent living setting and taking

classes in Spanish at The Pontificia Universidad Catolica del Ecuador (PUCE).

Analyzing data (quantitative and/or qualitative) obtained from either of these hypothetical

1
students may or may not produce results that can be generalized and applied across the

full spectrum of the types of study abroad programs available to U.S. students. This

challenge is at the heart of Stimpfl and Engberg’s purpose which was lacking in the field

until that time. Many of the areas/topics addressed by Stimpfl and Engberg were not new

at the time of their 1997 study. However, they do a very good job of identifying

historical concerns other researchers have raised about methods, sample sizes, and

validity and tie all of these concerns together, along with problem areas they also

identified, in a meaningful and well argued manner during their literature review and

critique.

During the literature review Stimpfl and Engberg evaluate a significant amount of

research literature on study abroad and they inform the reader that their review, while

extensive, was more of a selective process rather than comprehensive in nature. As in all

disciplines, it is nearly impossible to include every piece of research in a literature

review. Stimpfl and Engberg selected many of the more well-known research studies

conducted over the previous forty years. Although I haven’t quantified the number of

times the studies critiqued by Stimpfl and Engberg have been incorporated into other

research studies I am familiar with the works and have seen them referred to often by

other researchers. It was quite appropriate for Stimpfl and Engberg to be very selective

in choosing the more well know articles to analyze and critique as these are the studies

that have formed the base of knowledge in the field.

Stimpfl and Engberg’s investigation began by conducting an extensive review of

the relevant literature. In some cases, as Marshall and Rossman (1999) indicate, “the

literature review yields cogent and useful definitions, constructs, concepts, and even data

2
collection strategies” (p. 53). For Stimpfl and Engberg, the literature review revealed

data and themes relating to sojourner change and the factors that influenced these

changes. Stimpfl and Engberg then conducted extensive interviews with 37 former study

abroad students. While Stimpfl and Engberg did an excellent job selecting students that

participated on a wide variety of study abroad program types, as well as selecting by

gender and year in their undergraduate program, they had a relatively small sample.

While their research was qualitative in nature and these types of studies tend to have

small sample sizes this goes against one of the critiques Stimpfl and Engberg made about

sampling sizes used in the various study abroad studies they analyzed. Stimpfl and

Engberg (1997a) show their concern of studies with small sample sizes by stating “while

this may be convenient, it does not lead to easily generalizable data” (p. 15). I would

argue that Stimpfl and Engberg used a purposeful sample for their study based on the

description of their student selection process. It is unclear how and where they obtained

student information from which they selected their participants. Neuman (1997) states

that purposive sampling is an appropriate method of sampling if the researcher “uses it to

select unique cases that are especially informative” (p. 206).

Stimpfl and Engberg use an analytic induction approach to collecting and

analyzing the data. Robinson (1951) describes the steps necessary when using an

analytic induction/grounded theory approach to data analysis. During the initial stages of

the data collection process, the researcher develops an initial conceptual framework that

defines and explains the phenomenon under investigation. As additional data are

collected, the researcher incorporates these into the existing model while comparing and

contrasting the new data with the existing data. If the researcher identifies data that do

3
not fit with the existing framework, the definition and the explanation of the phenomenon

are modified so that the new data will fall under the framework. The conceptual

framework is continuously undergoing modifications as new data are collected and

analyzed. It is essential that the researcher search for cases that do not fit the existing

framework. Actively seeking new cases allows the researcher to redefine, reformulate,

and “fine-tune the conceptual framework until a predictive relationship describing the

phenomenon is established” (Stimpfl & Engberg, 1997b, p. 98). Stimpfl and Engberg do

a very good job of describing their analytic induction methods but a little more

description of the process would be helpful to the reader. Since Stimpfl and Engberg’s

research was an exercise in theory building it seems that using an analytical induction

approach to analyzing both the literature and participant interviews was very appropriate.

Stimpfl and Engberg’s analysis produced a Comparative Matrix that identified the

following three main categories: level of immersion, level of synthesis and level of

difference. Incorporating cross-cultural learning theory added a fourth category, level of

personal development, to the Matrix. The author’s added a modified Model of

Intercultural Sensitivity by Milton Bennet (1986). Stimpfl and Engberg fully

acknowledge that this is still a work in progress and that more research on applying the

Matrix to individual study abroad programs is necessary before it becomes a practical

tool. I fully agree with Stimpfl and Engberg on the need to test the Matrix with a variety

of study abroad program types but I also argue for additional research, analysis and

refinement to the Matrix itself prior applying and testing it on study abroad programs.

In sum, the Stimpfl and Engberg study and development of the Comparative

Matrix is certainly an asset for the field. The journal that published the article has been

4
out of circulation for over five years and the Stimpfl and Engberg articles are relatively

unknown in the field. However, the occasional graduate student and research scholar will

incorporate Stimpfl and Engberg’s study in their literature review. Unfortunately, this is

not the case for study abroad practitioners conducting outcomes assessment research

and/or study abroad program evaluation who seem oblivious to this study.

References

Bennet, M. (1986). Towards ethnorelativism: A development model of

intercultural sensitivity. In M. Paige (Ed.), Cross-Cultural Orientation: New

Conceptualizations and Applications (pp. 27-70). Lanham, MD: University Press of

America, Inc.

Carlson, J.S. & Widaman, K.F. (1988). The effects of study abroad during

college on attitudes towards other cultures. International Journal of Intercultural

Relations, 12, 1-18

Neuman, W. L. (1997). Social Research Methods: Qualitative and quantitative

approaches. (3rd. ed.). Boston, MA: Allyn & Bacon.

Robinson, W.S. (1951). The logical structure of analytic induction. American

Sociological Review, 16, 812-818.

Stimpfl, J.R., & Engberg, D. (1997a). What to know before you go: Creating a

comparison for research on study abroad programs. International Education Forum, 17

(1), 7-21.

Stimpfl, J.R., & Engberg, D. (1997b). Comparing Apples to Apples: An

integrated approach to study abroad program assessment. International Education Forum,

17 (2), 97-109.