Chan, Y. L. and B. E. Lynn. 1991. Performance evaluation and the analytic hierarchy process. Journal of Management Accounting Research (3): 57-87.
Summary by Mohamed Gomaa
Ph.D. Program in Accounting
University of South Florida, Spring 2002
Balanced Scorecard Main Page | Performance Measurements Main Page
The main purpose of this article, as stated by the authors, is to review the need for multiple performance criteria to measure long-run profitability, the potential difficulties of using such measures, and to present the analytic hierarchy process and its potential use in synthesizing multiple performance measures. The authors also describe a case and apply the analytic hierarchy process to illustrate the way in which it can be used to transform multiple criteria into a composite performance measure.
MULTIPLE PERFORMANCE CRITERIA VS THE SINGLE INDICATOR
Performance evaluation has been a difficult problem for businesses that can be viewed from several perspectives.
1. The shareholders’, Board of Directors’ or CEO’s perspective:
·The perspective of the organization as a whole.
2. The upper management perspective:
·The perspective of the organization or its segments as it pertains to the effectiveness of the unit or the individual managing the unit.
3. The operational perspective:
·The perspective of the management and supervisors as it relates to the efficient day-to-day operations.
Performance evaluation has been treated by management accountants as a by-product of their connection with the accounting system in the organization. If profit maximization is the main economic goal of the firm, then the profit measure will be the most important measure of performance. Kaplan (1984), believes that while financial measures are important, other measures such as product innovation, product leadership, employee skills and morale, or customer loyalty, can be much better indicators of future profitability than annual profits. The authors believe that a firm is effective in the long run only if it achieves its goals and that performance evaluation is closely connected to goal setting in that it feeds back information to the system on how well strategies are being implemented. Several possible reasons given by the authors for why businesses have not embraced the multiple performance criteria performance evaluation include:
1. The cost of designing and collecting multiple criteria is high,
2. Non-profit performance criteria are difficult to interpret and
evaluate because of their subjectivity, and
3. A single comprehensive measure is often needed for
THE ANALYTIC HIERARCHY PROCESS
The authors define the analytic hierarchy process as a participation-oriented methodology that can aid coordination and synthesis of multiple measures. It models the way in which the human mind structures a complicated problem. It has been applied in a variety of decision and utility contexts to help groups arrive at decisions where varying values are held by group members. It is ideally suited to help resolve problems that arise when multiple criteria are used in performance evaluation.
An Example of the Analytic Hierarchy Process
The authors provide an example scenario where a senior executive must decide on which of three candidates to promote. The decision maker will compare three criteria (leadership, human relations, and financial management ability) in pairs to develop a ranking. The comparisons would be:
1. Is leadership more important than human relations skills for
2. Is leadership more important than financial management ability
for this job?
3. Are human relations skills more important than financial
management ability for this job?
A ratio scale of 1 to 9 is added to the ordinal ranking provided by the responses to these questions to provide the relative importance of one criterion over another. The pairwise comparisons are summarized in a square matrix and the eigenvalues and preference vector are then computed to determine the relative ranking of the criteria. Then each of the candidates is compared on the basis of the criteria as follows:
1. Is candidate A superior to Candidate B in leadership skills?
2. I s candidate A superior to Candidate C in leadership skills?
3. I s candidate B superior to Candidate C in leadership skills?
Contributions of the Analytic Hierarchy Process to Performance Evaluation
The analytic hierarchy process can allow for multiple viewpoints to be incorporated into the priority ranking by being used at many levels of the organization. Its most important contribution to performance evaluation is that it provides a systematic approach for weighting performance to provide a comprehensive performance measure. This measure can be used to assess the overall performance of the organization, to rank organizations or segments of organizations, to serve as input to incentive compensation schemes, and as input into decisions about the organization.
PERFORMANCE EVALUATION AND THE ANALYTIC HIERARCHY PROCESS – A CASE EXAMPLE
The authors provide an example of a hypothetical decentralized organization with division located in several countries. This decentralization meant the need for a means of monitoring and evaluating the division managers. The corporation had three indicators to assess divisional performance and to set bonuses:
1. Return on investment.
2. Net income per unit of labor.
3. Change in market share.
After analyzing these indicators, they decided to reaffirm the companies goals:
1. To sustain growth in market share through the production of
quality products which meet customer needs.
2. To enhance profitability through efficient production and
distribution of the products.
3. To provide a healthy working climate for employees.
4. To maintain the reputation of flexibility and innovation in the
Since the divisions of the organization were located in several countries, the effects of foreign currency fluctuations had to be taken into consideration. They added six new performance indicator measures:
1. Operating margin ratio.
2. Proportion of net income contributed by foreign exchange
gains or losses.
3. Percentage of employee turnover in the year.
4. Customer satisfaction.
5. Product and technology innovation.
6. Total operating cost variance.
To apply the analytic hierarchy process, a questionnaire was given to the division managers and two Divisional Vice-Presidents to compare the relative importance of the nine evaluation criteria. The results formed a 9 x 9 matrix from which the preference vector (eigenvalues) was computed. Based on the performance vector, the nine evaluation criteria were ranked as follows: profitability, product and technology innovation, hedging effectiveness, operating efficiency, marketing effectiveness, customer satisfaction, productivity, operating effectiveness, and employee morale.
After the priorities were established, the two vice presidents were asked to complete a second questionnaire which requires pairwise comparisons between the divisions on the basis of the actual values of each of the nine evaluation criteria separately.
The results of the analytic hierarchy process showed that one of the divisions was the best in terms of some factors while another was the best in terms of others. It was difficult from the information provided to determine which division has the best overall performance. These results were different from the company’s original evaluation scheme. The authors believe that the analytic hierarchy process experiment was successful for this firm even though the results were different from the company’s original three-factor weighted scheme.
Advantages of this Methodology
1. Both quantitative and qualitative measures can be included in
the evaluation scheme.
2. It allows for multiple input in the setting of priorities of the
3. Subjectivity in setting priorities for evaluation criteria and
assessing divisional performance is reduced.
4. Consistency in judgment is improved.
5. The evaluation of performance against standards can be
incorporated in the process.
6. The resultant vector from the analytic hierarchy process
provides a composite performance measure which can be
used for other purposes (e.g., the allocation of bonuses to
Limitations of This Methodology
1. It requires substantial time and effort.
2. It can be difficult and complicated to understand because of
its quantitative nature.
3. Although it is systematic, it still requires managers to make
subjective judgments about interpretation of the qualitative
4. The relative ranking of the original alternatives may be
reversed when an identical alternative is added to the list.
Note: This study was somewhat confusing to us because the company sounds like a real organization although Chan and Lynn include the word hypothetical after the company name in parentheses on page 69. Kasanen, Lukka and Siitonen (1993) list this paper as a study in which a construction is developed further, but which lacks, in one way or the other, practical implementation. For a case study where the authors used the technique in an insurance company, see Chan, Y. L. and B. E. Lynn. 1993. Organizational effectiveness and competitive analysis: An analytic framework. Advances In Management Accounting (2): 85-108. I suppose the 1993 study passes the semi-strong market test of a managerial construction mentioned by Kasanen, Lukka and Siitonen on page 253 of their paper mentioned above.