1

Software Engineering Research

What is software engineering research?
What is software engineering?

CDT413: Advanced Software Engineering Empirical research methods in software engineering

Software Engineering Research Experiments and surveys

Case studies

Validity and reliability

2007

What Is Software Engineering? What Is Software Engineering?

1. The application of a systematic, disciplined,
quantifiable approach to the development, operation,
and maintenance of software; that is, the application
of engineering to software
2. The study of approaches as in 1
IEEE Standard Glossary of Software Engineering Terminology,
IEEE std 610.12-1990, 1990.
Software engineering is an engineering discipline that
is concerned with all aspects of software production.
Software engineers should adopt a systematic and
organised approach to their work and use appropriate tools
and techniques depending on the problem to be solved, the
development constraints and the resources available.
Ian Sommerville, Software Engineering, 8th edition, 2006

What Is Software Engineering Research? What Is Software Engineering Research?

Engineering research study methods, tools, etc. that What kinds of questions are "interesting"?
can be used to solve practical problems
What kinds of results help to answer these questions, and
May include invention of new methods, tools, etc. or what research methods can produce these results?
improvement of existing ones
What kinds of evidence can demonstrate the validity of a
But invention is neither necessary nor sufficient
result, and how to distinguish good results from bad ones?
As opposed to basic research study phenomena and
try to find the truth
 Types of Research Questions Types of Research Questions
Method or means of development
Method or means of development How can we do/create (or automate doing) X?
What is a better way to do/create X?
Method for analysis Method for analysis
How can I evaluate the quality/correctness of X?
Design, evaluation, or analysis of a particular instance How do I choose between X and Y?
Design, evaluation, or analysis of a particular instance
Generalization or characterization What is a (better) design or implementation for application X?
What is property X of artifact/method Y?
How does X compare to Y?
Feasibility What is the current state of X / practice of Y?

Types of Research Questions Types of Research Results

Generalization or characterization Procedure or technique
Given X, what will Y (necessarily) be? Qualitative or descriptive model
What, exactly, do we mean by X? Empirical model
What are the important characteristics of X? Analytic model
What is a good formal/empirical model for X? Notation or tool
What are the varieties of X, how are they related? Specific solution
Feasibility Does X even exist, and if so what is it like? Answer or judgment
Is it possible to accomplish X at all? Report

Types of Research Results Types of Research Results

Procedure or technique Notation or tool
New or better way to do some task, such as design, implementation, measurement, evaluation, Formal language to support technique or model (should have a calculus, semantics, or other
selection from alternatives, basis for computing or inference)
Techniques for implementation, representation, management, and analysis, but not advice or Implemented tool that embodies a technique
guidelines
Specific solution
Qualitative or descriptive model
Solution to application problem that shows use of software engineering principles may be
Structure or taxonomy for a problem area; architectural style, framework, or design pattern; non- design, rather than implementation
formal domain analysis
Careful analysis of a system or its development
Well-grounded checklists, well-argued informal generalizations, guidance for integrating other
results, Running system that embodies a result; it may be the carrier of the result, or its implementation
may illustrate a principle that can be applied elsewhere
Empirical model
Answer or judgment
Empirical predictive model based on observed data
Result of a specific analysis, evaluation, or comparison
Analytic model
Report
Structural model precise enough to support formal analysis or automatic manipulation
Interesting observations, rules of thumb
 Validation Techniques Validation Techniques
Analysis - I have analyzed my result and find it satisfactory through
Analysis
rigorous derivation and proof
data on controlled use
Experience
experiment

Example Experience My result has been used on real examples by someone other than me, and the evidence
of its correctness / usefulness / effectiveness is
narrative
Persuasion
data, usually statistical, on practice

Evaluation comparison of this with similar results in actual use

Example Heres an example of how it works on

Blatant assertion a toy example, perhaps motivated by reality
a system that I have been developing

Validation Techniques Software Engineering Research

Persuasion I thought hard about this, and I believe Question Result/method Validation
if you do it the following way,
a system constructed like this would Feasibility Qualitative model Persuasion
this model seems reasonable
If the original question was about feasibility, a working system, even without analysis, can be persuasive Characterization Technique Implementation
Evaluation Given the stated criteria, my result
adequately describes the phenomena of interest Method/means System Evaluation
accounts for the phenomena of interest
is able to predict because , or gives results that fit real data Generalization Empirical model Analysis
Feasibility studies, pilot projects

Blatant assertion No serious attempt to evaluate result

Selection Analytic model Experience

A Common Plan A Common, But Bad, Plan

Question Result/method Validation Question Result/method Validation

Feasibility Qualitative model Persuasion Feasibility Qualitative model Look it works!

Characterization Technique Implementation Characterization Devise a technique Implementation

Can X be done Build a Y Measure Y and Can X be done System Evaluation

better? compare to X better?
Generalization Empirical model Analysis Generalization Empirical model Analysis

Selection Analytic model Experience Selection Analytic model Experience

 Two Other Good Plans Empirical Research Methods
Question Result/method Validation Definitions and motivation
Can X be done at all? Qualitative model Look it works! Different types of methods
Characterization Technique Implementation Experiments and surveys

Method/means Build Y that does X Evaluation Case studies

Is X always true of Y? Empirical model Check proof Validity and reliability of empirical studies
Selection Formally model Y Experience Measurements
and prove X

Empirical Research Methods Empirical Research Methods

Based on observations Controlled experiments
As opposed to methods relying only on analysis, e.g. in Surveys
mathematics or theoretical computer science
Case studies
Two main categories of observations
Quantitative based on measurement

Qualitative explanations without measurement

Controlled Experiments Surveys

Researchers control independent variable(s) and Researchers observe a phenomena in a representative
observe dependant variables(s) subset of some population
Useful for studying isolated activities in controlled E.g. collecting data about a number of projects in different
environments organization through questionnaires and/or interviews

E.g. comparing the use of two programming languages Research in-the-large

using students as subjects

Research in-the-small
 Case Studies Empirical Research Methods
Researchers observe a phenomena in a real-life
Researchers
context
Control/observe Observe
E.g. collecting data about an ongoing project Design

Possible methods for data collection include interviews,

project participation, documents, artifacts (software), Independent variables
Study Dependent variables
Research in-the-typical Confounding factors

Validity of Empirical Studies Construct Validity

Construct validity Establishing correct operational measures for the
concepts being studied
Internal validity
Indirect measures are often used
External validity
E.g. reported working hours as a measure of effort
Conclusion validity

Construct Validity Internal Validity

Establishing a causal relationship, whereby certain
Researchers
conditions are shown to lead to other conditions
Control/observe Observe
Design As distinguished from spurious relationships

Independent variables
Study Dependent variables
Confounding factors
 Internal Validity External Validity
Establishing the domain to which a studies findings can be
Researchers generalized
Can an experiment using students as subjects be generalized
Control/observe Observe
Design to software professionals?
Can a case study of one project be generalized to other
projects in
Independent variables
Study Dependent variables The same/other organization
Confounding factors The same/other application domains
The same/other countries

Reliability (Conclusion Validity) Empirical Research Methods

Demonstrating that the operations of a study can be Hard science
repeated with the same results
Controlled experiments high level of control, repeatable

Surveys sampling, statistically valid

Soft science
Case studies needs careful documentation, multiple-case
studies can strengthen results