Вы находитесь на странице: 1из 4

A REVIEW OF INFORMATION TECHNOLOGY INVESTMENT EVALUATION METHODOLOGIES: THE NEED FOR APPROPRIATE EVALUATION METHODS

Benny Ranti, ranti@infosolusi.co.id Graduate Program in Information Technology, University of Indonesia Salemba Campus, PUSILKOM UI Building, Jalan Salemba Raya No. 4, Jakarta 10430
ABSTRACT For years, many academics and practitioners have been debating about the worth or business value of information technology (IT) and information systems (IS) investment. Despite large investments in IT over many years, some studies said that IT investment produces insignificant benefits known as the productivity paradox. Later, the paradox has been declined by other studies reporting that there appears positive relationship between IT investment and organization performance. The difficulties in measuring especially benefits are often the cause for the uncertainty about the expected benefits of IT investments. Organizations seeking positive relationship between organization performance and IT investment have often disregarded the IT investment evaluation because of its intangible and difficult nature. Or if it was done, the evaluation was performed inefficiently and ineffectively because of the absent of the appropriate evaluation methods. By taking into account the positive impact of IT investment, this paper reviews the model, type, and benefit of IT investment, as well as the evaluation methodologies. This paper suggests that the development of a more complete evaluation method is highly needed because it is said that cannot measure, cannot manage. Keywords: IT investment, organization performance, IT evaluation method

1.

MODEL, TYPE, AND BENEFIT

1.1. Model of IT Investment The main logical notion of a company investing money in information technology (IT) and information systems (IS) solution is to see a positive impact to the organization performance. In the past thirty years there have been some IT investment models introduced by researchers. The Synthesized Model of IT Investment developed by McKeen et al [1] combined all the above mentioned models. This model shows a chain of transformational process of IT investment before impacting the organization performance. The process is started with the form of IT governance that must be defined either on project or on-going basis. This leads to a specific investment and deployment before IT can be used by the organization to enhance its performance. The level of conversion effectiveness (i.e., the mix of internal managerial and external environmental factors) determines the degree to which each of these variables translates into organizational performance. Determining the impact of IT investments on organizational performance has been and will

continue to be an important research concern for both practitioners and academics [2]. The most commonly cited causes of the difficulty to appraise IT investments as brought up by Giaglis [3]: 1. The intangible nature of the benefits. 2. The benefits of IT are realized in the long run. 3. Strategic and competitive advantages are inherently difficult to quantify. 4. The benefits of IT are indirect and therefore indistinguishable from several confounding factors. 5. The theories and techniques available are inappropriate for understanding and capturing the value of IT. 1.2. Type of IT Investment From business perspective there are many different reasons for investing in IT which lead to some types of IT investments. Each type of IT investment requires different evaluation criteria depending on the purpose of the investment and the benefits to be achieved.

Prosiding Konferensi Nasional Teknologi Informasi & Komunikasi untuk Indonesia 3-4 Mei 2006, Aula Barat & Timur Institut Teknologi Bandung

112

Based on the combination of IT Investment Matrix by Lucas [4] and Zees classification [5], the type of IT investment can be defined as follows: 1. Mandatory IT. Required to satisfy regulatory requirements, to meet internal organizational requirements, and to provide IT applications that are competitive necessity. For example: changes to the payroll system, financial consolidation system for a multinational company, and join an industry-specific EDI service. 2. Efficiency and Effectiveness Improvement. Aimed at reducing or avoiding operational and labor costs, increasing business productivity and revenue, and monitoring business activities. For example: IT may contribute in various ways and its contribution can be evaluated with measures of cost, quality, and speed of internal business processes. 3. Strategic IT. Aimed at gaining a sustainable advantage over competitors and improving the organizations share of, or position in, existing and new markets. For example: the application of IT as a product or as a service, often involving Internet functionality. 4. IT Infrastructure. These are investments in the basis on which IT applications are built. They do not offer direct benefits, but enable the benefits of other IT investments to be realized. For example: WAN. 5. IT Research. Executed to ensure that the business is not left behind by technological progress. For example: in various ways. 6. Transformational IT. Must be combined with changes in management philosophy, good for fast response organization (risky to change structure, but high potential rewards). For example: Virtual Organization. 1.3. Benefit of IT Investment In line with the strong hope of CEO and senior management to see real improvements to the business performance as a result of IT/IS investments, the concept of IT benefit has been expanded to IT value. The definition of IT value is to always relate IT benefits to business performance. In other words, all benefits produced must be quantified or measured economically. Henceforth, the term IT benefit is used interchangeably with IT value with the meaning of IT value. So far there have been 2 (two) types of IT benefits commonly used or known, i.e., tangible and intangible. In between there are quasi intangibles. The increasing need to assess and measure IT benefit has produced a more detailed type of IT benefits. Based on observation of types of IT benefits

introduced among others by Remenyi [6] and Parker [7][8], 4 (four) types of IT benefits called IT Benefit Matrix (following Remenyis term) have been produced as follows: 1. Easy-to-Quantify Tangible (EQT) or Hard Benefit: IT benefits which directly affect the companys profitability and the effect can be directly and objectively measured. For example: reduced cost and increased revenue. 2. Hard-to-Quantify Tangible (HQT): IT benefits which can also be seen to directly affect the companys profitability, but the precise effect cannot be directly measured. For example: better information and improved the companys security. 3. Easy-to-Quantify Intangible (EQI): IT benefits which can be measured, but the impact does not necessarily directly affect the companys profitability. For example: increased customer satisfaction and improved staff satisfaction. 4. Hard-to-Quantify Intangible (HQI) or Strategic/Soft Benefit: IT benefits which cannot be easily measured and the impact does not necessarily directly affect the companys profitability. For example: improved market reaction and increased company image. HQT and EQI can be said as a derivation of quasi intangible. 2. EVOLUTION OF IT EVALUATION

In line with the development of IT investment model in the past thirty years, researchers also proposed methodologies for evaluating or measuring IT business value. There may be a number of objectives for IT investment evaluation including [2]: 1. As part of the process of justification for a project. 2. Enabling an organization to make comparisons of the merit of a number of different investment projects competing for limited resources. 3. Providing a set of measures which enable the organization to exert control over the investment. 4. Enabling organizations to gain competitive advantage, to develop new business, to improve productivity and performance, as well as to provide new ways of managing and organizing. In general, IT investment evaluation (or IT evaluation in short) methodologies have been influenced by quantitative, qualitative, or combined approaches. Cronk and Fry [9] conducted a research in the evolution of IT evaluation methodologies. They roughly divided the evolution into four phases based upon:

Prosiding Konferensi Nasional Teknologi Informasi & Komunikasi untuk Indonesia 3-4 Mei 2006, Aula Barat & Timur Institut Teknologi Bandung

113

Type of measurement. Can be qualitative or quantitative type of measurement. 2. Level of measurement. Can be at system, process, or organizational level. 3. Degree of complexity. Starting as early in the mid 1960s which focused on efficiency at the system level, the fourth phase, starting at the end of 1990s, introduced the so-called multi-dimensional methodology. This methodology combined the strengths of each type of measurement (qualitative and quantitative) and level of measurement (system, process, and organization). Some of multi-dimensional methodologies fit in this phase including Information Economics and Balanced Scorecard. 1. 3. IT EVALUATION METHODOLOGIES

3.

4.

The developments of IT evaluation methodologies in the last few decades have produced dozens of methods both unclassified and classified by authors into a number of ways, e.g., qualitative, probabilistic, tangible, intangible, and risk. Based on observation of quite a number of methodologies, the following briefly explained multidimensional methodologies are the most cited ones (excluding by the authors/creators): 1. Information Economics (IE). [11][12][13] IE is a variant of cost-benefit analysis, tailored to cope with the particular intangibles and uncertainties found in IT projects. IE retains ROI calculations for those benefits and costs which can be directly determined through a conventional cost-benefit analysis. However, the decision making process is based on a ranking and scoring technique of intangibles and risks factors associated with the IT investment. IE identifies IT performance measures and use them to rank the economic impact of all the changes on the organizations performance caused by the introduction of IT. Surrogate measures are often used for most intangible. 2. Real Options Valuation (ROV). [11][13][14] ROV aims to put a quantifiable value on flexibility. The technique was applied to leasing, mergers and acquisitions, and manufacturing. ROV uses three basic types of data: current and possible future business strategies; the desired system capabilities sought by the company, and the relative risks and costs of other IT choice that could be used. ROV can help assess the risks associated with IT investment decisions, taking into account that business strategies and system requirements may change.

5.

6.

Balanced Scorecard (BSC). [11][12] BSC is a set of financial and operational measures that provide a balanced presentation of both the financial and operational impacts of a system giving senior managers a comprehensive view of a system's value. BSC focuses attention on four areas or perspectives that are most critical to any business: finance, customers, internal processes, and employee learning and growth. Additional information was taken from Robinson [15]. Economic Value Added (EVA). [11][14] As a metric, EVA equals net operating profit minus appropriate capital charges. Using EVA as a yardstick to assess the performance of individual departments, including IT, on a monthly, quarterly and yearly basis can help with decisions on new projects. Conflicting and confusing goals (like revenue growth, market share or cash flow) are replaced with a single financial measure for all activities. Additional information was taken from Shand [16]. Return on Management (ROM). [6][13] ROM measures the impact of IT on business unit performance based on the added value to an organization provided by management. The assumption is that in the modern organization, information costs are the costs of managing the enterprise. If ROM is calculated before and after IT is applied to an organization, then the IT contribution to the business, which is so difficult to isolate using more traditional measures, can be assessed. ROM defines the managements valueadded as everything left after subtracting all the direct operating costs from the value-added due to direct labor. ROM proposes an index of the total performance of management due to the introduction of IT. The index is obtained by dividing the management value-added by the costs of management. ROM focuses on the most important impact of IT, i.e., on the value-added by management generated by the IT in excess of managements total costs. Multi-Objective Multi-Criteria (MOMC). [6][13] MOMC explicitly recognizes the existence of many points of view and more than one set of values in the decision to invest in IT. MOMC does not rely on monetary measurements of value, instead it works via an iterative procedure to establish preferences and utilities. Where there are many stakeholders, the best IT investment is that which will deliver the highest aggregate utility or which provides the highest overall measure of satisfied preferences.

Prosiding Konferensi Nasional Teknologi Informasi & Komunikasi untuk Indonesia 3-4 Mei 2006, Aula Barat & Timur Institut Teknologi Bandung

114

4.

CONCLUDING REMARKS

Although IT evaluation methodologies have been progressing considerably in the last few decades, moving from system level, quantitative efficiency metrics to complex multi-dimensional ones, much remains to be uncovered. For example, the causal relationships issue brought by Cronk [10], which suggests that the IT value outcome should be a result of the interaction among the system, the user and the organization (called Holistic Construal). According to Remenyi [6], each methodology leads to the development of a measure or metric which allows IT to be evaluated. Sometimes metric is compared to a corporate or industry standard or sometimes relative metrics are compared for competing systems. However, in most cases a single measure is not sufficient to make an evaluation. Two or three metrics combined will usually be required but six or seven measures should be avoided. It can be concluded that the main constraint in evaluating IT investment is the lack of any single appropriate method for understanding and capturing the value of IT. This is to confirm Giaglis fifth statement [3] as explained before. Thus, the research in developing a more complete IT investment evaluation method is highly needed. In doing that, in order not to reinvent the wheel, combining some existing methods or improving one existing method by adding the strengths from other methods to overcome its weaknesses, can be the effective way to move forward. Based on the writers experience in using Information Economics (IE), the following brief scenario might be applied to improve or enhance IE to become a more appropriate methodology: 1. Financial Approach. The original simple ROI calculation of IE can be replaced with a more rigorous EVA or ROV measurements. 2. Non-Financial Approach. The original Business and Technology Domains of IE can be complemented with Human Domain to fulfill Cronks Holistic Construal [10]. By limiting only to the 6 (six) above methodologies, there are numerous ways in combining strengths of each methodology. It definitely needs an in-depth and continuing research to produce the better, the next better ones, and so forth. 5. REFERENCES

[1] James D. McKeen et all, in Mo Adam Mahmood and Edward J. Szewczak (eds), Measuring Information Technology Investment Payoff:

Contemporary Approaches, Idea Group Publishing, 1999. [2] Chad Lin and Graham P. Pervan, A Review of IS/IT Investment Evaluation and Benefits Management Issues, Problems, and Processes, Chapter 1, Idea Group Publishing, 2001. [3] George M. Giaglis et al, The I.S.S.U.E Methodology for Quantifying Benefits from Information Systems, Department of Information Systems and Computing Brunel University, United Kingdom, 1999. [4] Henry C. Lucas, Jr., Information Technology and the Productivity Paradox Assessing the Value of Investing in IT, Oxford University Press, 1999. [5] Han van der Zee, Measuring the Value of Information Technology, IRM Press, 2002. [6] Dan Remenyi et al, Effective Measurement and Management of IT Costs and Benefits, Butterworth-Heinemann, 1995. [7] Marilyn M. Parker, Strategic Transformation and Information Technology Paradigms for Performing While Transforming, Prentice-Hall, 1996. [8] Marilyn M. Parker and Robert Benson, Information Economics Linking Business Performance to Information Technology, Prentice-Hall, 1988. [9] Marguerite Cronk and Graham Fry, IT Evolution: how far have we come?, http://www.sabusinessreview.co.za, July 2001. [10] Marguerite Cronk, Understanding Information System Evaluation through Holistic Construal, http://www.sabusinessreview.co.za, December 2000. [11] Tracy Mayor, A Buyers Guide to IT Value Methodologies, CIO Magazine, July 15, 2002. [12] Vector Research Incorporated, Guide for Managing Information Technology (IT) as an Investment Measuring Performance, Prepared for Department of Defense the United States of America, 1997. [13] Joseph Wen and Cheickna Sylla, in Mo Adam Mahmood and Edward J. Szewczak (eds), Measuring Information Technology Investment Payoff: Contemporary Approaches, Idea Group Publishing, 1999. [14] Sarv Devaraj and Rajiv Kohli, The IT Payoff Measuring the Business Value of Information Technology Investments, Prentice-Hall, 2002. [15] Robin A. Robinson, Balanced Scorecard, Computer World, January 24, 2000. [16] Dawne Shand, Economic Value Added, Computer World, January 24, 2000.

Prosiding Konferensi Nasional Teknologi Informasi & Komunikasi untuk Indonesia 3-4 Mei 2006, Aula Barat & Timur Institut Teknologi Bandung

115