Project evaluation tools

Decisions on where to invest the company's resources to achieve a technological

innovation have a major impact on the future competitiveness of the company.
Therefore, trying to get involved in the right projects is worth an effort, both to avoid
wasting the company's time and resources in meaningless activities, and to improve
the chances of success.
However, in a continuous improvement context, ideas for change and projects which
need significant resources might be prioritised rather than selected, with a view to all
projects eventually being addressed.
In short, project evaluation aims at analysing research and development projects, or
activities or ideas, for any or all of the following purposes:
Getting an overall understanding of the project.
Making priorities among a set of projects.
Taking a decision about whether or not to proceed with a project.
Monitoring projects, eg by following up the parameters analyzed when the
project was selected.
Terminating projects and evaluating the results obtained.

Why and when are they used?

To provide information to assess the value of a potential project with particular
reference to the estimation of costs, resources and benefits, in order to take a decision
about whether or not to proceed with the project. A secondary use is for monitoring
and terminating projects.

How do they work?

A very long list of specific techniques have been developed over the years, and are
still being developed and used today, both within companies and in the academic
world. Most of those techniques can be described by means of some common
elements which form the backbone structure of any project evaluation technique.

Figure 1
Main elements of Project Evaluation

The inputs:
the data on important aspects of the projects and the business environment
which are needed to analyze them. The inputs will be assembled from various
sources, and care should be taken to ensure its certainty, although some inputs
will surely be very subjective. At the end of the day, an evaluation can only be
as good as the data that go into it.
Table 1
Typical inputs to Project Evaluation
Typical inputs
Technological the technical activities which will have to be undertaken, maturity of
technology, company's technological position
Internal

potential technical success, familiarity with the area of the project,

role of individuals and of different functions within the organization

Financial

expected benefit, likely cost, both of project and consequent actions

Market

size and attractiveness of the market, competitive position

Business

clarification of objectives, fit with company's strategy, level of topmanagement support, key success factors

Weighting: as certain data may be given more relevance than other (eg of
market inputs compared with technical factors), in order to reflect the
company's strategy or the company's particular views. The data is then
processed to arrive at the outcomes.
Many techniques include balancing between projects, as the relative value of a
project with respect to other projects is an important factor in situations of

competition for limited resources. Portfolio manageportment techniques are

specifically devoted to deal with this factor.
The techniques might also incorporate how the results of the evaluation have to
be communicated among the interested parties, as well as the way to proceed
in order to make the final decision. Eventually, no technique should be allowed
to take decisions, as this is a management responsibility. Even when expert
systems are used managers have the final word.
Project evaluation usually assumes that there is a choice of projects in which to invest,
but businesses may only be able to do R&D in collaborative projects or contract
situations and if they are not asked to be involved in any projects there is no choice
available in practice. Even in these situations, project evaluation is paramount to grasp
what might be expected from pursuing each opportunity.
With the apparent increase in networking, companies could move from one
collaborative project to another as opportunities arise. Project evaluation in such
situations actually means strategic direction (deciding which projects to accept).
However accepting a project which is essentially a contract or order is quite different
from selecting projects in which the company is making the financial investment.
Companies are less likely to decline project opportunities and this could imply a
deviation from the preferred strategy of the company. Even in these cases, project
evaluation techniques can and should be used to assess the value and risk of those
opportunities, in order to know whether the company's resources could be better used
for other purposes.
Nevertheless, all these different situations and contexts can have a specific answer
within project evaluation techniques. The answer can be either to use different
techniques for different situations, or, better, to introduce variations within the same
technique (eg by using different inputs or different weighting depending on the
context).

Specific techniques
Project evaluation methods have evolved in response to changing needs, although 'old'
techniques are still in use today. The earlier methods were based on financial
assessment, and even now this forms the back-bone of most practical methods.
One basic classification of all potential techniques might be:
Techniques mainly or uniquely based on a financial assessment.

 Techniques mainly based on human judgment.

Learning techniques, which explicitly take account of past experience in order
to improve future decisions.
Most of the techniques in practical use by industry incorporate a mixture of financial
assessment and human judgment. A more detailed list of types of techniques is shown
in table 2.
Table 2
Type of Project Evaluation techniques
Techniques
Financial ratio
methods

Short description
Among the longest established, easily-applied methods
Criticised as of limited accuracy
The key is the ratio (financial benefit) /(cost) in which the
estimations of benefits should be agreed by marketing

Cash flow analysis

Requires the estimation of cash outflows and inflows

It can be sofhisticated by considering discounting factors

Score index
methods

They generalize the simple Ratio Analysis by considering

the probability for technical and market success
The estimations of probability are usually made by
experts
Some of these methods include discount factors: some of
the measures are Internal Rate of Return (IRR) and
Return of Investment (ROI)

Mathematical
methods

They are based on the optimisation of allocation of R&D

resources through mathematical programming

The success of the methods depends strongly on how well
the benefit is understood and how well the input data is
converted into variables
The algorithms tend to be customised and they can
incorporate experience through expert systems
Matrix methods

They employ subjective considerations for proper

measurement of management information
Matrix methods use correlation techniques in order to
identify relationships that constitute the basis for
decision-making

Check-lists

Method that includes the reminders of the factors which

are important in decision-making
Simple and rapid way to assess a project with little effort
Can be considered as starting point for more sophisticated
methods as SWOT analysis

Relevance and
decision trees

It is an approach for structured thinking

It requires a very clear objective or long term goal,
establishing a clear differentiation between goals and
means for achieving them
Critical path analysis and decision trees are examples of
these methods

Multicriteria &
table methods

Scoring procedures to incorporate judgments based on a

number of criteria
Various criteria may be used such as economic and
financial factors and concepts or decision theory
The scoring of criteria is usually complemented with the
use of weight factors in order do distinguish
the importance of each criteria
QFD

It is used in many fields of design and engineering

It is based on the identification of customer requirements
and the means for achieving those requirements
Includes a scoring procedure with weighting of the factors

Experience based
methods

They are based on the analysis of conditions that are

usually related to the success or failure of a project
(quality of execution, synergies, etc.)
Success or failure is predicted according to the answers to
questions on those mentioned factors
The inputs for those factors should come from members
in different departments

Vision

The vision is that of an individual (a Chief Executive or a

product champion) defying conventional wisdom and
bringing about a breakthrough
It is common when information is scarce and it can be
sustained by irrational methods
When it works it has many virtues: speedy, incisive and

changing the scene

Many techniques used today are totally or partially software based, which have
some additional benefits in automating the process. In any case, the most
important issue, for any method, is the managers' interpretation of the direct
outcomes.
There is no best technique. The extent to which different techniques for project
evaluation can be used will depend upon the nature of the project, the information
availability, the company's culture and several other factors. This is clear from the
variety of techniques which are theoretically available and the extent to which they
have been used in practice. In any case, no matter which technique is selected by a
company, it should be implemented, and probably adapted, according to the particular
needs of that company.

Checklist
A checklist is a reminder of the factors (a list of factors) which are important in
making a decision. Most useful criteria for evaluating any type of research or
development project are essentially independent of the business field and the business
strategy. These criteria include technical and commercial details, research and
development realities, legal and financial factors, company targets and company
strategy, etc.
The requirements for the use of this technique are minimal, and the effort involved in
using it is normally low. Another advantage of the technique is that it is very easily
adaptable to the company's way of doing things. However, checklist can be a starting
point for more sophisticated methods where the basic information can be used for
better focus. One simple and useful example is a SWOT analysis, where projects are
assessed for their Strengths, Weaknesses, Opportunities and Threats.
Therefore, this technique can be developed further and the analysis interaction and
feedback can be easily managed using simple information technology. Different ways
to sophisticate the technique might be:
To include some quantitative factors among the whole list of factors.
To assign different weights to different factors.

 To develop a systematic way of arriving to an overall opinion on the project.

A simple checklist could be one made up of a lists of factors which have been formed
to affect the success of a project and which need to be considered at the outset. In the
evaluation procedure a project is evaluated against each of these factors using a linear
scale, usually 1 to 5 or 1 to 10 (see figure 2). The factors can be weighted to indicate
their relative importance to the organisation.
Figure 2
Example of how to use a checklist

A value in this technique lies in its simplicity but by the appropriate choice of factors
it is possible to ensure that the questions address, and are answered by, all functional
areas. When used effectively this guarantees a useful discussion, an identification and
clarification of areas of disagreement and a stronger commitment, by all involved, to
the ultimate outcome.
Checklist example
Table 3 shows an example of a checklist, developed by the Industrial Research
Institute, that, in princpiple, could be applied to any type of R&D activities: research,
product development and process development.
Table 3
List of potential factors for project evaluation
Corporate Objectives

Fits into the overall objectives and strategy

Corporate image

Marketing and Distribution

Size of potential market

Capability to market product

Market trend and growth
Customer acceptance
Relationship with existing markets
Market share
Market risk during development period
Pricing trend, proprietary problem, etc.
Complete product line
Quality improvement
Timing of introduction of new product
Expected product sales life
Manufacturing

Cost savings
Capability of manufacturing product
Facility and equipment requirements
Availability of raw material
Manufacturing safety

Research and development

Likelihood of technical success

Cost
Development time

Capability of available skills

Availability of R&D resources
Availability of R&D facilities
Patent status
Compatibility with other projects
Regulatory and legal Factors

Potential product liability

Regulatory clearance

Financial

Profitability
Capital investment required
Annual (or unit) cost
Rate of return on investment
Unit price
Payout period
Utilization of assets, cost reduction and cash-flow

Using the list of table 3 as a reference, any company should be able to develop its own
checklist. This adaptation could take several forms:
By sophisticating the checklist as suggested above.
By choosing part of or adding different factors that better suit the company and
its environment. For example, if the company strongly relies on external
resources, either from other companies or from technological infrastructures,

either for the development or for the production activities, this should probably
be reflected in the list.

Cash flow analysis

In its simplest form this approach requires the completion of a standard form which
asks for estimates of the expected cash outflows and inflows for the project. This can
be done on a yearly basis for longer term projects or on a shorter time scale for shorter
term projects, even down to weekly.
Table 4
Cash outflows and inflows
Cash Outflow

Cash Inflow

Net Cash Flow

1997

C0

B0

B1-C1

1998

C1

B1

B2-C2

1999

C2

B2

B3-C3

2000

C3

B3

B4-C4

When portrayed as a cash flow diagram a typical project might look as shown
in figure 3.
Figure 3
Diagram to plot the evolution of cash flow over time

A frequently used form of analysis applied to this data is to calculate the break even
point. This is the point at which the cumulative net cash flow is equal to zero and
hence the cash inflow has covered all the cash outflows (see figure 4).

Figure 4
Break even point

The reason for this measure being seen as important is that the shorter the time scale
to break-even, or payback, the less risk there is that the environment will change
dramatically and hence seriously affect the estimates used in the cash flow
projections. However, the analysis completely ignores the likely returns after the
break even point and therefore is not really a good indicator of the real potential of a
project. In some cases, people choose to ignore the time value of money, although it is
not difficult to handle this by including cost of capital payments on the net cash
outflows as part of the costs.
An easier way to do this, however, and one commonly encountered is to use the
discounted cash flow approach. Here the cost of capital is incorporated directly into
the calculation in a way which also takes into account the actual year of expenditure.
The result is a figure which represents the present value of both the cost and benefit
stream and ultimately a net present value which is the difference between the two. For
example:

A net present value greater than zero therefore indicates that a project will produce a
surplus after all costs are paid and is therefore likely to be worth undertaking. It is also
possible to calculate a Benefit to Cost ratio which can provide another indication of
value.
In some cases it may not be easy to identify the true cost of capital and at the same
time some people prefer to think in terms of rate of return on investment (ROI). In
such circumstances it is not difficult to identify what has been termed the internal rate
of return which is calculated by setting PVB = PVc and solving the resulting equation
for i. This can be done very simply, as can the NPV calculation, using widely
available computer programmes.

Decision analysis
There is inevitably uncertainty surrounding the information which is used in any
evaluation exercise. The extent of this will depend upon the type of project and the
environment in which the organisation operates. One way of incorporating this into
the evaluation procedure is through the use of probability estimates. This can be done
in a number of different ways. For example, it is possible to adjust the calculations
made by other techniques using estimates of the probability of commercial success
and/or of technical success to create a rank index as follows
Rank Index = Pt PB (B-C)

The potential variability in costs and benefits is very likely to be influenced by the
time to completion and by the competitive environment. It is possible to take this into
account using a simple 3 estimate approach. This requires that people are prepared to
consider pessimistic and optimistic values as well as the most likely. This approach
has been used in the planning of highly complex technical projects as is described in
the project planning section.
However these methods tend to ignore the multi stage process which most projects go
through. A useful way of illustrating this is through the use of the decision tree
approach. Figure 5 clearly illustrates that it is possible to stop a project at intermediate
stages if progress is not up to expectations. In this case there are few stages and
following each one there is an estimate showing the probability of success (s) or of
failure (f).
Figure 5
Example of a decision tree

Watch out for

Typical problems that might arise in the implementation and use of project evaluation
are as follows:
Thinking that subjective estimates are exact representations of future outcomes.

 People who are not committed to the process and who provide information
without due thought to its likely accuracy and the effect of this on the selection
decision.
Unnecessary change of goals and resource conflicts with other projects which
will lead to an inability to achieve the desired outcomes.
The technique becoming a routine that the project has to suffer, being not
applied with the intensity and effort required. Furthermore, any technique needs
to be applied rigurously to avoid that those with a long experience in dealing
with it can learn how to 'cheat' it.
Changes occuring in the company context should have an impact in the
application of project evaluation. Although there might be a learning process
involved in order to master the technique, hence requiring it is applied
consistently over a period of time, at the same time it needs to continuosly
adapt itself to every new situation.

Further information
The report: EIRMA Working Group 47, 'Evaluation of R&D projects'. It
provides an overall explanation and analysis of project evaluation specific
techniques, including both a method to assess which technique might be more
suitable for a specific company and context, and specific cases of application in
companies.
The application of simple financial methods can be seen in many books either
on financial management or on project evaluation.