RESEARCH METHODS IN

CS
LESLIE G. AGBULOS
Instructor
Software Development

A multi-disciplinary field
 We need to investigate
 Which tools, techniques, and processes they
use?,
 What social and cognitive processes
surrounding them?
 How individual software engineer develop
software?
 How teams and organizations coordinate their
efforts?
Computer Science Approaches Techniques
Research  1. Descriptive  3. Mixed
 1. Theoretical  2. Causal Approaches
Methods Comparative  1. Longitudinal
 3. Correlational Studies
 2. Empirical
Methods  4. Experiments and  4. Historical
 1. The Scientific
Quasi-experiments Research
Method  5. Ex Post Facto  5. Theoretical
 2. Simulation  2. Qualitative Education
Educational Approaches  6. Literature
Research  1. Case Studies Reviews
 1. Quantitative  2. Other  7. Action Research
Theoretical Methods

Mathematical techniques - used to reason about

computational systems, and standard mathematical
methods, including recursion and induction, are
employed.

Computer Science papers

 “mathematical” - 73%
 Information Systems approximately 12%
 Software Engineering to below 11%

Glass et al., in a survey of Computing journal and

conference articles (2004),
Theoretical Methods
Specific instances might include the following:
 semantics minus its syntactic “sugar”;
 complexity theory
 graph theory
Pros
 “Hard” results may be obtained which are repeatable and verifiable.
Cons
 The topic under investigation must allow a mathematical model.
 The researcher must be competent with the appropriate branch of
mathematics!
Empirical Methods

 “concrete” evidence
 characterised by an experimental
approach
 Two empirical approaches
Scientific Method
Simulation
Empirical Methods

 1. The Scientific Method

 an iterative process whereby the investigator
formulates an hypothesis based on observed
information, performs tests and/or
experiments to validate the hypothesis, and if
those tests are consistently passed, the
hypothesis is supported.
 characterised by a positivist approach where
objective, value-free facts can be observed and
used to test hypotheses.
 Empirical Methods
The Scientific Method
Pros
 There is a much literature on the Method
 The Method has been adapted for use in many scientific
disciplines.
 Useful for topics which overlap other scientific disciplines.
Cons
 Limited applicability in the Computing disciplines.
 Possible learning curve where other disciplines are
involved
 reductionist
The Scientific Method cont..
Empirical Methods

2. Simulation
Pros
 Appropriate to many computing activities
Cons
 Results may be difficult to validate.
 A variety of concerns have been expressed about
the validity of using simulations and of the
conclusions which may be drawn
Educational Research

Education researcher has available

a relatively large portfolio of
methods and techniques, many of
which are used throughout the
Social Science communities.
in any sense “precise”
Quantitative Approaches

 all about numbers and statistics.

 might involve relationships and patterns being identified by other
means
 use of a significantly large set of data to validate their existence.
Pros
 The method of analysis is one which a scientist can relate to easily.
Cons
 Care must be taken with formulating the hypotheses being tested
 Statistics is not to everyone’s liking.
 Some social scientists would claim that what can be learned from this
type of approach is very limited.
Causal Comparative

 the “independent variable” has already been set, and is not under the
control of the researcher.
Pros
 A standard approach where experimentation cannot be performed,
yet statistical data is available.
 Many cause-and-effect relationships can, in principle, be tested in a
single study.
Cons
 Alternative hypotheses for the observed phenomena must be
explored.
 A study may suggest the existence of multiple cause-and-effect
relationships.
Correlational Research

 This is an exploratory technique for finding associations between

variable
 Pearson or Spearman correlation tests.
 “what is the direction of the correlation?” and “what is its
magnitude?”
 refers to the latter
Pros
 Good for exploratory studies
Cons
 A correlation between two variables does not indicate cause or effect.
 Statistical tests should always be used with care.
Experiments and Quasi-experiments

 Experiments can be used to test a theory by considering “cause-

effect” hypotheses, and are typically used as part of the Scientific
Method.
 Experiments can also be used to explore, to investigate the
relationships of variables to each other, prior to formulating a
theoretical hypothesis
 experimentation can take place with either a pedagogic or a technical
focus.
 In an experiment, we seek to establish cause and effect.
 In a quasi-experiment, we merely find indicators of what might be
cause and effect, based on our understanding of the other factors
which may affect the exercise, and the validity will depend on how
carefully those factors have been addressed.
Experiments and Quasi-experiments
cont…
Pros
 Experiments yield explicit concrete evidence
which can be used to support an hypothesis.
 Experiments can be replicated.
 Quasi-experiments are relatively easy to perform.
 A quasi-experiment is sometimes the only
methodology available are highly constrained, or
if a desired experiment cannot be performed for
ethical reasons.
Experiments and Quasi-experiments
cont…
Cons
 Experiments with small samples, or run over too short a
period of time, may not yield sufficient data to provide a
result.
 A single experiment will typically only test a single
hypothesis.
 Some phenomena cannot be controlled or managed, in
which case experiments are impossible. This is a
particular issue if the research has an education focus.
 Results from quasi-experimental research do not exhibit
the same internal or external validity as experimental
results.
Experiments and Quasi-experiments
cont…
Examples
Technical experiments
 Investigating the performance of a neural-net based learning support
tool
Educational experiments
 A new tool for comparing document similarity is used to detect
instances of plagiarism.
 A new educational tool has been developed.
Educational quasi-experiment
 An exercise in testing the efficacy of a new learning tool or method
might involve its use on a group of students.
Ex Post Facto Research

 ‘retrospectively’
 involves exploring cause and effect
relationships for events
 “pseudo-experiment”
 research interest relates to social
issues
Ex Post Facto Research cont…

Pros
 For some areas of enquiry, this may be the only possible approach.
 Provides hypotheses which may be explored in subsequent experiments.
 Can be used in realistic scenarios where experiments cannot easily be set up.
Cons
 Lack of control over the data.
 Identifying control groups is problematic, and data samples cannot be
randomised.
 The technique does not establish cause and effect, it merely suggests it.
 Results may not be reproducible.
Qualitative Approaches

 interpretative
 the study of cases, rather than of whole populations or
samples of populations.
 may be used to explore data, discover possible
relationships, and thereby gain insight and understanding
by representation and interpretation of the social
environment.
 covers a diverse range of strategies from different
philosophical perspectives
 based on a relativist philosophy which holds that reality
can only be defined subjectively
Qualitative Approaches cont…
Pros
 A large corpus of material on qualitative research
practices is available.
 Qualitative approaches are appropriate for research using
non-numerical data.
 Counter the “reductionist” criticism of quantitative
approaches.
Cons
 The methods used are very different from those usually
employed by Scientists, and you are advised to “do your
homework” before commencing a qualitative research
exercise.
Case Studies

 the detailed analysis of a single thing,

which might for example be a person,
a system, an organisation, a course, a
process, a group, or a methodology.
 may be exploratory (piloting further
research), descriptive (narrative), or
explanatory (testing theories) (Yin,
2002).
Case Studies cont…
Pros
 Easy to disseminate to a non-technical audience.
 The effect of a “real context” can be observed.
 Comparison can be made with similar cases and situations.
 Contributing factors do not all have to have been
identified prior to the start of the study.
Cons
 Generalisations cannot be made easily.
 Cannot be reproduced easily (or even verified)
 There may be observer bias.
Other Techniques
 Ethnography
 “cultural anthropology”
 “naturalistic enquiry”
 studyof sociocultural phenomena, and may
be descriptive or analytic.
 Phenomenography
 studyand interpretation of other peoples’
experiences, perceptions of, and
understandings of specific phenomena.
Other Techniques

 Grounded Theory
 uses repeated stages of data collection to
obtain views from the study participants.
 Narrative research
 study individuals by asking them to provide
stories about their lives.
 Discourse analysis
 studyof conversation or other exchanges, such
as on-line messaging
Mixed Approaches

 multiple methodologies and data collection

techniques may be used to investigate a
particular problem.
 become increasingly popular
 may be able to achieve a more balanced result by
triangulating data sources.
 ways to combine methods
 sequential procedures
 concurrent procedures
 transformative procedures
Mixed Approaches
Pros
 Flexibility
 A fuller picture of the topic can be built up.
 Bias inherent in one approach can be compensated for.
Cons
 Care must be taken that each approach is valid and
reliable on its own.
 Care must be taken that the different approaches are
combined in a valid way.
 May be time-consuming to carry out the necessary work
for a number of different approaches.
Longitudinal Studies
 The observation of given entities over an extended period. Detailed
data is collected on those entities.
Pros
 Effective at establishing causal relationships.
 Eliminates chance occurrences.
Cons
 Time consuming
 Entities may disappear during the period of study
 Repeated viewing of the same data may change that data
 New circumstances may affect the original research plan.
 It may be difficult to get suitable participants.
 The data collected may be complex.
Historical Research

The process of performing historical research is as

follows.
1. Formulate hypothesis
2. Collect data
3. Evaluate data
4. Confirm or refute hypotheses
Historical Research
Pros
 This research approach is essentially the same as that
used by the much larger community of researchers in the
History of Science, which has produced a substantial
corpus of published material which can be consulted.
 Recurrent trends over time can be identified, and may be
assessed.
Cons
 Historical research is, by its very nature, a specialised
activity for a Computer Scientist, and is inappropriate for
most Computer Science Education research activities.
 Data sources may be inadequate or inaccurate.
 Author bias may be an issue.
Literature Reviews

 not strictly a research methodology

 will form part of any sound research exercise, and should
be of sufficient breadth and depth to give the reader
confidence that the research is appropriately situated.
Pros
 Information from a substantial literature review can be
used to support multiple research activities.
Cons
 Time-consuming.
Action Research

 Action research is a practical activity which takes

place “real-time”, and to solve immediate and
pressing problems.
 The process is as follows.
1. Planning
2. Action
3. Observation
4. Reflection
 “softest”
Action Research

Pros
 Can be used in almost any setting.
 Extremely flexible.
 Particularly suited to e-learning environments.
Cons
 Potentially high workload for the researcher.
 Difficult to focus activities.
Thank you…