Int. J.

Production Economics 80 (2002) 119128

Applying quality award criteria in R&D project assessment

Ville Ojanena,*, Petteri Piippob, Markku Tuominena
a

Department of Industrial Engineering and Management, Lappeenranta University of Technology, P.O. Box 20,
FIN-53851 Lappeenranta, Finland
b
R&D Center, Valtra Inc., P.O. Box 557, 40101 Jyvaskyl
a,
.
. Finland

Abstract
Due to the problematic characteristics and complexity of R&D, the applicability of Total Quality Management
(TQM) and quality award criteria in the assessment of a companys R&D process is very challenging. This study
concentrates especially on the application of the Malcolm Baldrige Quality Award (MBQA) Criteria to R&D
assessments at project level.
The applicability of quality award criteria in the assessment of R&D projects is rst discussed with the help of a
literature review at conceptual level. The meaning of different sub-areas of quality award criteria is analyzed from the
point of view of R&D activities and single projects. The measures, performance criteria and concrete measurable
aspects for R&D project evaluation are then derived on the basis of the analysis. In the empirical part of the study, the
analysis of the utilization of criteria for R&D project assessment is discussed from the viewpoint of a manufacturing
company that has successfully applied the Finnish National Quality Award Criteria based on the MBQA.
The study gives examples of the derivation of new R&D project measures from the quality award criteria framework.
r 2002 Elsevier Science B.V. All rights reserved.
Keywords: Total quality management (TQM); Quality award criteria; Self-assessment; R&D projects; Performance evaluation

1. Introduction
An important consequence of the introduction
of systematic approaches of quality management,
e.g. quality awards, is the increasingly widespread
use of their models and criteria for company selfassessment. Several similar types of national
criteria are used for assessment, e.g. the Malcolm
Baldrige Quality Award Criteria, the European
Foundation for Quality Management (EFQM)model, the Deming Prize and the Finnish National
*Corresponding author. Tel.: +358-5621-2670; fax: +3585621-2667.
E-mail address: ville.ojanen@lut. (V. Ojanen).

Quality Award Criteria. This study concentrates

on application of Finnish criteria to R&D project
assessments. The Finnish Criteria were formerly
based on the MBQA, but from the year 2001 they
are based on the EFQM-model. The framework
utilized in this study is based on the MBQA.
In a number of earlier studies, the Total Quality
Management (TQM) philosophy has been argued
to be an applicable approach also for the management of R&D. However, due to the problematic
characteristics and complexity of R&D, the
application of quality award criteria in the
assessment of a companys R&D process is very
challenging (see e.g. Bellary and Murthy, 1999;
Boyer, 1991; Fisher and Heywood, 1992; Kiella

0925-5273/02/$ - see front matter r 2002 Elsevier Science B.V. All rights reserved.
PII: S 0 9 2 5 - 5 2 7 3 ( 0 2 ) 0 0 2 4 7 - 5

120

V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119128

and Golhar, 1997; Lovett, 1992; Patino, 1997).

Companies have different types of R&D projects
and in addition to the process level, suitable
performance criteria are needed also at the project
level. There is a need in many organizations to
bring the criteria also to project-level assessment in
order to effectively control and manage different
types of single R&D projects and the whole project
portfolio.
The proposed systematic analysis approach of
this study has several phases. First, the application
of quality award criteria in the assessment and
evaluation of R&D projects is discussed with the
help of a literature review at conceptual level in
order to identify the possibilities and potential
problems involved. The meaning of different subareas of quality award criteria is analyzed from the
point of view of R&D activities and single
projects. The measures, performance criteria and
concrete measurable aspects for R&D project
evaluation are then derived on the basis of the
analysis. In the empirical part of the study, the
analysis of the applicability of derived measures
for R&D project evaluation is discussed from the
viewpoint of a manufacturing company that has
successfully applied the Finnish National Quality
Award Criteria.
The main goal of the study is to promote
effective R&D management by utilizing a systematic analysis approach based on the Quality Award
Criteria framework in order to achieve better
understanding of the meaning of each sub-area of
the framework to the assessment of R&D as a
whole and at the project level, as well as to propose
new measurement subjects and evaluation methods or concrete measures for R&D projects. In
order to evaluate the validity of the derivation
results of the analysis approach and to ne-tune
the results, the derived measurement subjects and
potential R&D project measures are compared
with the development needs of the R&D performance measurement in a case company.
The systematic analysis approach provides new
insights for companies to improve the performance
of their R&D through the effective use of a quality
award criteria framework. Generally, the results of
the study promote the communication of criteria
to the project level and taking different aspects

of the sub-areas of quality award criteria more

strongly into account in systematic R&D projectlevel evaluation in different organizations. The
study also gives examples of derivation of new
R&D project metrics from the quality award
criteria framework and comparison of the metrics
in the light of the development needs of R&D
performance measurement in a case company.

2. Quality management in R&D: A brief overview

The quality of the R&D process is one of the
critical success factors inuencing the performance
and success of a companys R&D activities.
According to a study by Cooper (1998), a high
quality new product process is the strongest
common denominator among high performance
businesses. The effective management of R&D
requires appropriate metrics for assessing the
quality of the process.
The principles of quality management have been
applied in R&D management in several organizations, and their applicability has been reported in a
number of earlier studies (e.g. Bellary and Murthy,
1999; Boyer, 1991; Fisher and Heywood, 1992;
Kiella and Golhar, 1997; Lovett, 1992; Patino,
1997). Due to the nature of R&D activitiese.g.
insecurity related to planning and decisionmaking, assessment of the contribution of R&D
to prots, long time lag, creative personnel,
coordination and control etc.there are barriers
hindering the effective use of these principles.
2.1. Quality management practices in R&D
theoretical considerations from previous studies
Total Quality Management (TQM) can be
dened as follows: A business improvement
philosophy which comprehensively and continuously involves all of an organizations functions in
improvement activities (Rosenau et al., 1996).
TQM is an approach for improving the competitiveness, effectiveness and exibility of a whole
organization. It is essentially a way of planning,
organizing and understanding each activity, and
depends on each individual at each level (Oakland,
1993). The implications emerging from the total

V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119128

quality movement embrace management and

organizational theory and incorporate a wide
range of quality strategies including customer
satisfaction, benchmarking, time-based competition, process simplication, and performance
evaluation. Quality management tools can be
applied not only to production but also to costcutting, capital investment, environmental concerns, and research and development activities
(Fisher and Heywood, 1992). The TQM philosophy has handed down useful legacies for R&D
management, e.g. the understanding of customer
needs, strengthening cross-functional and crossorganizational linkages and teamwork, formal
benchmarking, measurement of R&D performance and the establishment of unifying business
processes which integrated R&D into a broader
enterprise-wide context (Chatterji and Davidson,
2001).
Most TQM approaches strongly emphasize
measurement, especially in the quality assurance
and control areas (Oakland, 1993). According to
Oakland (1993), the measurement system must be
designed, planned and implemented to reect
customer requirements, give visibility to the
processes and the progress made, communicate
the total quality effort and engage a never ending
improvement cycle.
The introduction of the Malcolm Baldrige
Quality Award in 1987 was a milestone in the
evolution of total quality concepts in the US and

121

in the West in general. The launch of the European

Quality Award (EQA) in 1991, which took the
MBQA as its starting point, was a further advance
in TQM development (Conti, 1997). This study
focuses on the applicability of the Finnish
National Quality Award Criteria, formerly based
on the MBQA, but from the year 2001 based on
the EFQM-model. Since the empirical part of the
study describes the company that has successfully
applied the criteria and participated in the Finnish
National Quality Award contest in 1999, the
framework for assessment in this study is based
on the MBQA. A system perspective and causal
chains between the sub-areas of the MBQA are
presented in Fig. 1. The categories and weightings
of MBQA criteria in the year 2000 are presented in
Table 1.
Self-assessment can be seen as a by-product of
quality awards. The awards and self-assessments
have different aims, and therefore the means of
conducting assessments are different. For instance,
Conti (1997) suggests to reverse the direction with
respect to the traditional award assessment sequence, and to simply forget the weights in
assessments, even though a standard model like
the MBQA is a wise choice for starting.
The Finnish National Quality Award Criteria
had the MBQA as the starting point until the year
2000. According to a survey made by the Center
for ExcellenceFinland (2001), the Finnish Quality Award is at least moderately well known

Fig. 1. Baldrige criteria for a performance excellence framework: a system perspective (National Institute of Standards and
Technology, 2000).

V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119128

122

Table 1
2000 Criteria for performance excellenceitem listing (National Institute of Standards and Technology, 2000)
2000 categories/items
1

Point values

Leadership
1.1
Organizational leadership
1.2
Public responsibility and citizenship

85
40

Strategic planning
2.1
Strategy development
2.2
Strategy deployment

40
45

Customer and market focus

3.1
Customer and market knowledge
3.2
Customer satisfaction and
relationships
Information and analysis
4.1
Measurement of organizational
performance
4.2
Analysis of organizational
performance

125

85

85
40
45

85

45

Human resource focus

5.1
Work systems
35
5.2
Employee education, training, and
25
development
5.3
Employee well-being and satisfaction 25

85

Process management
6.1
Product and service processes
6.2
Support processes
6.3
Supplier and partnering processes

85
55
15
15

Business results
7.1
Customer focused results
7.2
Financial and market results
7.3
Human resource results
7.4
Supplier and partner results
7.5
Organizational effectiveness results

115
115
80
25
115

Total
points

2.2. Quality management practices in R&D at

Valtra Inc

40

One of the main differences between the Finnish

National Quality Award Criteria and MBQA
Criteria in the year 2000, was that there were
eight categories in the Finnish Criteria. The eighth
category included the issues of public responsibility and environmental effects. All the categories
of the Quality Award Criteria also cover the areas
of R&D, but they all cannot as such be utilized in
R&D assessment. The development of R&D
assessment with help of the Quality Award
Criteria, however, makes it easier for R&D to be
integrated in other activities of the company and
its business development.

450

1000

among 43% of company managers in Finland. The

quality award is best known in large companies, in
which 72% of managers know the Finnish Quality
Award well. 59% of these companies apply the
quality award thinking and utilize it in their
business development. The application of quality
awards is increasing especially in small companies.

The empirical part of this study is based on the

analysis of Valtra Inc, a business unit of the Partek
Corporation. Valtra develops, manufactures, markets and services tractors for agricultural, forestry
and municipal customers in more than 70 countries. It is the fth biggest manufacturer of tractors
in the world in its ranges.
Valtra has put a lot of effort for long term
quality management and continuous improvement, as well as taken many systematic
steps to reach its quality goals. Valtra was granted
the ISO 9001 quality certicate in 1993 as the rst
tractor manufacturer in the world, after comprehensive self-assessment and systematic development work.
After this the quality development work was
continued by applying the Malcolm Baldrige and
Finnish Quality Award criteria as well as developing the ISO14001 environmental management
system. The quality criteria and self-assessment
have helped Valtra to clarify its biggest strengths
and needed development areas as well as focus its
quality development efforts. Valtra obtained the
Finnish Quality Award in 1999 and certicated
ISO14001-system in 2000. Valtras quality development work continues by applying QS9000 tools
in its own operations and by training its suppliers
to use the QS9000 tools and continuously develop
their quality improvement capabilities.
A clearly dened product development process
is one of the key tools for quality management in

V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119128

Valtras R&D. The product development process

and other quality instructions are described in
Valtras quality manual. The product development
process, called product process, denes the needed
phases, control points and tasks, and related
documents for development projects. With the
help of the product process the participants in the
project and the steering group can check that all
required tasks are fullled and risks are under
control. One of the last tasks in each product
process is the assessment of executed projects to
clarify the success, strengths and weaknesses of
their execution and to continuously improve the
product process. The assessment is mainly based
on discussions and collected experiences. Some
quantitative criteria, like time and costs, are also
examined, but new criteria to clarify the fulllment
of the goals of projects and the effectiveness of
their execution are needed as well.

3. R&D performance assessment at the project level

Corporate performance as well as R&D performance can be measured at several levels. Typical
levels of assessment are company level, SBU level,
department level, process level, project level and
personal level. This study focuses on measurement
aspects at the project level. However, assessment at
the company or SBU level are also discussed, since
many project level measures are derived from
upper level measures.
3.1. Different measures for different R&D project
stages
R&D can be divided into stages in which
different evaluation techniques are applied (see
Pappas and Remer, 1985). Organizations often
execute many different types of R&D projects,
from fundamental basic research projects to
product improvement projects. Therefore, both
qualitative and quantitative evaluation techniques
can be needed. Quantitative techniques usually
follow a specic algorithm or predened ratio to
generate numbers that can be compared with other
projects and past experiences. Semi-quantitative
techniques are basically qualitative judgments that

123

are converted to numbers, and qualitative techniques are intuitive judgments (Pappas and Remer,
1985). According to Pappas and Remer (1985),
qualitative techniques are best suited for basic
research, semi-quantitative techniques for applied
research and quantitative techniques for product
development and improvements.
To better understand how the metrics vary,
Hauser and Zettelmeyer (1997) have introduced a
tier metaphor, which enables us to categorize a
diverse continuum of projects, programs and
explorations and focus on key characteristics.
Tier 1 is dened as basic research that attempts
to understand basic science and technology. Tier 1
explorations may have applicability to many
business units or may spawn new business units.
Tier 2 is dened as those activities that select or
develop programs to match the core technological
competence of the organization. Tier 3 is
dened as specic projects focused on the more
immediate needs of the customer, the business unit
and/or the corporation. For applied projects,
market outcome metrics are most relevant. In
their study, Hauser and Zettelmeyer present R&D
metrics, both qualitative judgments and quantitative measures, reported by interviewees, as well as
their relevancy for the Tiers. Integrated metrics
that contain an articulated but separable suite of
quantitative and qualitative techniques can be
exibly applied across all types of R&D (Werner
and Souder, 1997).
Project-level measurement results can be utilized
in several ways in R&D. Through measuring,
activities can be better diagnosed, and signicant
problem areas or bottlenecks which inuence the
overall effectiveness of projects can be clearly
detected. Cause and effect relationships in R&D
project performance should be claried in order to
understand the signicance of single projects for
the whole R&D. The performance should be
assessed in different phases of projects, and
therefore both leading and lagging indicators are
needed. Well-dened leading indicators can give
valuable information and early signals predicting
project success for project managers. The measures
of project outputs reveal the quality and overall
performance of the project. The measurement
results can be utilized for instance in more effective

124

V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119128

resource allocation and motivation of project

personnel. Several different frameworks can promote the dening of appropriate R&D project
measures. The MBQA framework is one of the
possible framework, because it is well known and
widely applied for company self-assessments. It is
a systematic and exible approach for dening the
assessment criteria.
3.2. Development needs of R&D project
measurements at Valtra Inc
At the moment Valtra has ve corporate level
and three own measures for product development.
The corporate level measures are: the average time
to market of projects, changes in product costs,
guarantee costs of products, the share of new
products from total sales and the complexity of
products (total number of items/total number of
products). Valtras own R&D measures are the
amount of new models per year, development of
the average horsepower of produced tractors and
the capability of R&D to keep dened project
timetables at project level. The difference between
the actual and planned project schedules affects
the bonuses for the R&D personnel. The performance of projects is also evaluated by comparing
the actual product costs and features with the cost
target and project specications, but there are no
xed systematically used measures for that.
New measures are needed in Valtra to comprehensively clarify the fulllment of project goals,
product specications and customer needs. New
measures could be also useful in order to follow
the quality of input information and the effectiveness of the execution of the different phases of
projects. Creation of new innovative solutions and
knowledge is becoming more and more important
in Valtras R&D. Therefore, the R&D measures
should also clarify innovativeness and development of knowledge both at project and departmental level. An essential development need in the
measurement is to link the measures of different
levels more accurately. Our approach to the
derivation process of new potential R&D measures
aims at enhancing this link.
Valtra has also started to apply the main
principles of the Balanced Scorecard approach

(Kaplan and Norton, 1996) to establish a new set

of measures for the whole company as well as for
R&D. Earlier it has utilized the Quality Award
Criteria to support and focus company development. The Quality Award Criteria describe the
success factors of the companys processes as well
as important results. Thus they also describe
important measurement areas for R&D projects.
Quality award criteria are utilized in the next
chapter to derive new measures for manufacturing
companies R&D projects.

4. Analysis of the utilization of quality award

criteria in R&D project assessment
In this chapter the application of quality award
criteria for R&D project assessment is analyzed
with the help of a systematic analysis approach.
The analysis of the utilization of criteria for R&D
project assessment is discussed from the viewpoint
of a manufacturing company that has successfully
applied the Finnish National Quality Award
Criteria. In later implementation of the new
measures derived on the basis of the analysis, also
the most signicant development needs related to
the current R&D project metrics are carefully
taken into account.
Fig. 2 below presents a systematic analysis
approach for the derivation of new R&D project
measures with help of the MBQA framework.
Each category involves several sub-areas (see
Table 1), which are analyzed, but only the main
categories are presented below. The meaning of
each category is rst described from the whole
R&D perspective. Then the meaning of the
category for projects and their assessments is
analyzed. Several signicant measurement subjects
can then be derived from the results of the earlier
phases of the analysis. In the last phase, new
potential measures or evaluation methods are
presented as a nal result of the analysis utilizing
the MBQA framework. The results of the analysis,
i.e. the new measure proposals will then again be
compared to the development needs related to the
present R&D metrics and the whole measurement
system. This comparison reveals the nal applicability of the derived potential measures. The

V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119128

MBQA Criteria

Meaning of
criteria for the
whole R&D

Meaning of
criteria for R&D
project
assessment

Derived
measurement
subjects

Development
needs of
present R&D
measures

125

Derived
potential
measures /
evaluation
methods

New measures
for R&D project
assessment

Fig. 2. The phased process of derivation of new R&D project-level measures.

derivation process was implemented in expert

meetings with company representatives. The process was implemented systematically according to
the phase-by-phase principle so that the derived
potential measures were results from the fourth
phase of the process.
In Table 2 below the derivation process of R&D
project-level measures in the Leadership-category
is presented as an example. This example is a part
of a much larger table. Similar derivations were
implemented for all the seven categories, but their
presentation in this context is not essential, as the
principles and utilization of the derivation process
are the focal point here.
In the following table (Table 3), the examples of
derivation results from different categories are
presented. The example measures presented are
chosen for their values of applicability and novelty
from the case companys point of view. The
derived measures were compared with the development needs of the company in order to nd the
most applicable measures for R&D project assessment.
In the following, we will evaluate the usability of
the derived, Quality Award Criteria-based measures from the viewpoint of Valtra Inc. and its
development needs in R&D performance measurement, which were presented in Section 3.2 above.
New measures are needed for both function and
project level measurement in order to support a
continuous development of the effectiveness of
R&D and its processes. At the project level new

measures are needed to clarify the fulllment of

project goals, specications and customer needs as
well as to follow the effectiveness of the execution
and creation of new innovative solutions and
knowledge.
The executed derivation process produced several measurement areas and measures to meet the
described development needs. Good new measures
to clarify the fulllment of customer needs, project
goals and specications are:
*

Amount of realized customer requirements

(must be needs) and requests (should be
needs) stated in advanced executed customer
surveys.
Amount of features customers consider as their
own (based on fulllment of customer requirements).
Performance in pilot customers satisfaction
surveys.
Amount of contacts between key customers,
sales companies and personnel of the project.
Share of forecasted and realized results of
projects from the goals of the company and its
R&D, for example share of stated increase in
sales target (2/10%).
Share of fully and partly fullled specications.

The net present value of the project divided by

its total working hours and development of goals,
as well as the realized lead times of different phases
of the product process in different projects could

V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119128

126

Table 2
The derivation process of R&D project-level measures in the leadership-category
MBQA
criteria
category

Meaning for company

R&D

Meaning for R&D

project assessment

Measurement
subjects

How are the values known and Value test of project

internalized by project
personnel, amount of right
personnel?
answers

(1) Company values,

How are the factors

taken into account in
R&D projects. E.g.
how is it conrmed that
a project supports the
(1) Company values,

(2) Innovative working

environment

(2) Innovative working

environment, and

1. Leadership How are the following

factors taken into
account in R&D
by senior management?

Implementation of values?

Creativity, idea promotion?

Risk-taking capability?
(3) Companys direction (3) Direction of R&D and Contribution of a project to
the whole company?
the company objectives?

be useful measures to follow the effectiveness and

efciency of projects. The quality of input
information could be followed by clarifying the
reasons for changes of specications and categorizing them into groups, and by following the share
of different groups in the total amount of reasons.
Innovativeness and learning are difcult areas to
measure, but they are also becoming more and
more important factors for the success of Valtra.
Potential new measures dened in the derivation
process for Valtra are:

Share of people who have produced new ideas

from the total amount of personnel working in
the project.
Share of realized ideas out of the total amount
of ideas presented in the project.
Match (coverage %) between the needed
knowledge/competence areas and available
knowledge in the project.
Assessment of critical competencies before and
after the project.

Derived potential
measures, evaluation
methods

Post-project review of
implementation of
values, e.g. reliability:
schedule-keeping
Amount of initiatives
Amount of people who
made initiatives
Amount of new ideas
to be implemented
Contribution of a project
to the sales increase objectives
Contribution of a
project to gain new
business segments

Number of days worked together with customers (for example in farms) in the project.

Measures that can be utilized at both the project

and the functional level are practical to control the
total amount of measures. Many of the above
measures can be utilized at both levels. The
executed derivation process has helped to create
common measures for both the project and
functional level as well as increased the understanding of links between functional level and
project level measurement. The validity and
reliability of the derived measures are assessed by
the company management in the light of the
development needs presented above.

5. Conclusions
Approaches like the utilization of quality award
criteria are widely recognized as good starting
points for self-assessment in companies. Award
assessments and self-assessments may, however,

V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119128

127

Table 3
Examples of derivation results from different MBQA categories
Leadership
Amount of people who made initiatives
Amount of ideas to be implemented per total amount of ideas
Contribution of a project to the sales increase objectives
Strategic planning
Categorized amount of reasons for changes in a project
Competence of project personnel vs. competence areas needed in a project (x% coverage)
Project management assessment of the fulllment of the strategy in a project (e.g. scale 15)
Assessment of resources and strategic competencies before and after the project
Customer and market focus
Ability of project personnel to enumerate the main markets and main customers
Contacts of project personnel and amount of visits to key customers sales ofces
Existence of collected customer need document (yes/no) and its clarity and scope
Number of project people who know the specication and have the specication document
Systematic satisfaction measurements of pilot customers
Amount of recurred complaints (because of the same issue)
Amount of realized customer requirements and requests stated in advanced executed customer surveys
Number of features based on fulllment of customer requirements
Number of present customers who change to the product of project and the speed of change
Information and analysis
Availability of measurement information for project personnel
Schedule objectives and schedule keeping in different phases of a project; Development of project lead time in similar types of
projects (project efciency index)
Quality and amount index of communication
Amount of problems/faults; prioritized most difcult problem areas
Human resource focus
Amount of initiatives per person
Number of days in further education
Development speed and level of a project team
Process management
Schedule keeping in a product process
Amount of problems in planning and support systems per day per user
Number of initiatives from subcontractors
Business results
New customers as a result of a project
Project net present value per total working hours of a project
Satisfaction of project partners
Amount of new applied technologies
Share of forecasted and realized results of projects from goals of company and its R&D
Schedule keeping

have distinct purposes. When assessing and

measuring performance at the R&D project level,
the main purposes for assessing different types of
projects need to be carefully claried.

The aim of the research was to study the

application of quality award criteria in the
assessment of R&D projects by combining theory
and practice so that new potential measures could

128

V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119128

be derived with the help of our phased systematic

analysis approach. The literature review helped to
identify the possibilities and potential problems
related to the issue at conceptual level. The meaning
of different sub-areas of quality award criteria was
analyzed from the point of view of R&D activities
and single projects. New measures, performance
criteria and concrete measurable aspects for R&D
project evaluation were derived on the basis of the
analysis. In practice, the analysis of the application
of criteria for R&D project assessment was
discussed from the viewpoint of a case company
that has successfully applied the Quality Award
Criteria in its business development. Example
derivations of measures for R&D project assessments were described in the study.
Similar types of new measures for project
performance assessment were derived in the
analysis of different categories of quality award
criteria. For e.g. common measurements like
schedule keeping, the satisfaction level of project
personnel, customer satisfaction measurements,
the achievement level of project objectives and
number of initiatives of project personnel could be
found in several categories. This reects the basic
principlese.g. customer focus, concentration on
results, innovation and learningof quality management practices and quality awards. In addition
to common assessment methods, which are often
qualitative in nature, also more concrete, new
potential measures were derived as a result of the
systematic analysis approach.
The executed derivation process helped to create
common measures for both the project and
functional level, and increased the understanding
of links between functional level and project level
measurement. Measures that can be utilized at
both the project and functional level are practical
to control the total amount of performance
measures of a companys R&D.
Further studies within this issue concentrate on
the utilization of the executed derivation approach
and derived measures in companies R&D management and continuous improvement of R&D
activities. The links between project and functional
level measurements as well as the links between
utilization of Quality Award Criteria framework
and the Balanced Scorecard approach in compa-

nies self-assessment and performance measurement of R&D are also potential topics for deeper
future studies.

References
Bellary, A., Murthy, D.N.P., 1999. New product development
process and total quality management. Proceedings of
Portland International Conference on Management of
Engineering and Technology. Portland, Oregon, USA.
Boyer, S.M., 1991. Total quality management and new product
development. Total Quality Management 2 (3), 283289.
The Center for ExcellenceFinland (CEF), 2001. Suomen
laatupalkinnon tunnettuus ja k.aytto. nousussa [Internet
document], [referred 5.4.2002], Available at: http://
www.sly./laatupalkinto/index64.html (in Finnish).
Chatterji, D., Davidson, J.M., 2001. Examining TQMs
Legacies for R&D. Research Technology Management
44 (1), 1012.
Conti, T., 1997. Optimizing self-assessment. Total Quality
Management 8 (2), 515.
Cooper, R.G., 1998. Benchmarking new product performance:
results of the best practices study. European Management
Journal 16 (1), 117.
Fisher, J., Heywood, C., 1992. Total quality management of
Canadian R&D activities. CMA Magazine 66 (7), 2528.
Hauser, J., Zettelmeyer, F., 1997. Metrics to evaluate R, D&E.
Research Technology Management 40 (4), 3238.
Kaplan, R.S., Norton, D.P., 1996. The Balanced Scorecard
Translating Strategy into Action. Harvard Business School
Press, Boston.
Kiella, M.L., Golhar, D.Y., 1997. Total quality management in
an R&D environment. International Journal of Operations
and Production Management 17 (2), 184198.
Lovett, J.R., 1992. Doing the right things right. All the time.
Research Technology Management 35 (5), 3538.
National Institute of Standards and Technology, 2000. Criteria
for Performance Excellence [pdf document], [referred
5.4.2002], Available at: http://www.quality.nist.gov/
PDF les/2000 Business Criteria.pdf.
Oakland, J.S., 1993. Total Quality Managementthe Route to
Improving Performance, 2nd Edition. Butterworth-Heinemann, Oxford.
Pappas, R.A., Remer, D.S., 1985. Measuring R&D productivity. Research Management 28 (3), 1522.
Patino, H., 1997. Applying total quality to R&D at Coors
Brewing Company. Research Technology Management
40 (5), 3236.
Rosenau, M.D., Jr., A. Grifn, Castellion, G.A., Anschuetz,
N.F. (Eds.), 1996. The PDMA Handbook of New Product
Development. John Wiley & Sons, New York.
Werner, B., Souder, W., 1997. Measuring R&D performance
state of the art. Research Technology Management 40 (2),
3442.