Академический Документы
Профессиональный Документы
Культура Документы
far@ucalgary.ca 2
Empirical SE SE Investigation
Fill the gap between research and practice by: What is software engineering investigation?
Developing methods for studying SE practice Applying “scientific” principles and techniques to
Building a body of knowledge of SE practice investigate properties of software and software
related tools and techniques.
Validating research before deployment in
industrial settings
Why talking about software engineering
investigation?
Because the standard of empirical software
engineering research is quite poor.
far@ucalgary.ca 3 far@ucalgary.ca 4
far@ucalgary.ca 5 far@ucalgary.ca 6
SE Investigation
Investigation:: What? SE Investigation
Investigation:: Where & When?
Person’s performance In the field
Tool’s performance In the lab
Tool’s usability
far@ucalgary.ca 7 far@ucalgary.ca 8
SE Investigation
Investigation:: How? SE Investigation
Investigation:: Characteristics
Hypothesis/question generation Data sources come from industrial settings
Data collection This may include people, program code, etc.
Data evaluation
far@ucalgary.ca 9 far@ucalgary.ca 10
far@ucalgary.ca 11 far@ucalgary.ca 12
Where Data Come From? Where Data Come From?
Second Degree Contact Third Degree Contact
Access to work environment during work time, Access to work artifacts, such as source code,
but not necessarily participants documentation
Example: Example:
Instrumenting systems Problem report analysis
Real time monitoring Documentation analysis
Analysis of tool logs
Off-line monitoring
far@ucalgary.ca 13 far@ucalgary.ca 14
far@ucalgary.ca 15 far@ucalgary.ca 16
far@ucalgary.ca 19 far@ucalgary.ca 20
Examples /2 Examples /3
Case study: research in the typical Survey: investigate in the large
You may have used software reliability After you have used SRE in many projects in
engineering (SRE) for the first time in a project your company, you may conduct a survey to
in your company. After the project is completed, capture the effort involved (budget, personnel),
you may perform a case-study to capture the the number of failures investigated, and the
effort involved (budget, personnel), the number project duration for all the projects. Then, you
of failures investigated, and the project duration. may compare these figures with those from
projects using conventional software test
technique to see if SRE could lead to an overall
improvements in practice.
far@ucalgary.ca 21 far@ucalgary.ca 22
far@ucalgary.ca 23 far@ucalgary.ca 24
Control /2 Control /3
A common mistake: ignoring other variables that may affect How to identify the dependent and
the values of a dependent variable.
Example: independent variables?
Suppose you want to determine whether a change in programming
language (independent variable) can affect the productivity
Example:
(dependent variable) of your project. For instance, you currently use A D
FORTRAN and you want to investigate the effects of changing to
Ada. The values of all other variables should stay the same (e.g., F &BZ
application experience, programming environment, type of problem,
etc.) D&C F
Without this you cannot be sure that the difference in productivity is Given : A, B, C
attributable to the change in language.
But list of other variables may grow beyond control! Using causal ordering:
A, B, C D F Z
far@ucalgary.ca 25 far@ucalgary.ca 26
far@ucalgary.ca 27 far@ucalgary.ca 28
far@ucalgary.ca 29 far@ucalgary.ca 30
Example: Blocking & Balancing Formal Experiments: Principles
You are investigating the comparative effects of three design techniques
on the quality of the resulting code. 3. Local control
The experiment involves teaching the techniques to 12 developers and Correlation: the most
measuring the number of defects found per 1000 LOC to assess the code popular technique to
quality.
It may be the case that the twelve developers graduated from three assess relationships
universities. It is possible that the universities trained the developers in among observational
very different ways, so that being from a particular university can affect data
the way in which the design technique is understood or used.
To eliminate this possibility, three blocks can be defined so that the first Linear and nonlinear
block contains all developers from university X, the second block from correlation.
university Y, and the third block from university Z. Then, the treatments
are assigned at random to the developers from each block. If the first Nonlinear correlation
block has six developers, two are assigned to design method A, two to B, is hard to be measured
and two to C. and may stay hidden.
far@ucalgary.ca 31 far@ucalgary.ca 32
far@ucalgary.ca 33 far@ucalgary.ca 34
far@ucalgary.ca 35 far@ucalgary.ca 36
Empirical Research Guidelines Contents
1. Experimental context
2. Experimental design
3. Data collection
4. Analysis
5. Presentation of results
6. Interpretation of results
far@ucalgary.ca 37 far@ucalgary.ca 38
far@ucalgary.ca 39 far@ucalgary.ca 40
far@ucalgary.ca 41 far@ucalgary.ca 42
2. Experimental Design /2 3. Data Collection
D6: For formal experiments, perform a pre- Goal
experiment or pre-calculation to identify or estimate
the minimum required sample size. Ensure that the data collection process is well
D7: Use appropriate levels of blinding. defined
D8: Avoid the use of controls unless you are sure
the control situation can be unambiguously defined. Monitor the data collection and watch for
D9: Fully define all treatments (interventions). deviations from the experiment design
D10: Justify the choice of outcome measures in
terms of their relevance to objectives of the
empirical study.
far@ucalgary.ca 43 far@ucalgary.ca 44
far@ucalgary.ca 45 far@ucalgary.ca 46
far@ucalgary.ca 47 far@ucalgary.ca 48
5. Presentation of Results 6. Interpretation of Results
P1: Describe or cite a reference for all procedures used. Goal
Report or cite the statistical package used.
P2: Present quantitative results as well as significance levels. Ensure that the conclusions are derived
Quantitative results should show the magnitude of effects merely from the results of the experiment
and the confidence limits.
P3: Present the raw data whenever possible. Otherwise,
confirm that they are available for review by the reviewers
and independent auditors.
P4: Provide appropriate descriptive statistics.
P5: Make appropriate use of graphics.
far@ucalgary.ca 49 far@ucalgary.ca 50
6. Interpretation of Results
I1: Define the population to which inferential
statistics and predictive models apply.
I2: Differentiate between statistical significance and
practical importance.
I3: Specify any limitations of the study.
far@ucalgary.ca 51 far@ucalgary.ca 52