THE ANALYTIC HIER ARCHY PROCESS AHP FOR BUSINESS INTELLIGENCE SYSTEM EVALUATION

Abstract The aim of the article is a presentation of the report of the latest studies carried out at the Faculty of Management and Economics of Services (University of Szczecin, Poland) on analyzing the quality level of two business intelligence system (Power BI and Tableau Public). Analysis was based on the multi-criteria analytic hierarchy process AHP method, a tool that is used for determining the main criteria of BI system evaluation. By structuring the problem based on the hierarchy, it is possible to better understand the level of quality, the criteria to be used and the alternatives to be evaluated. The proposed research concept can be used to analyze business intelligence problems within the framework of specific subjects, such as system quality, quality of analysis, data cleaning and data connectors, visualization etc. Assuming a proper (the best description of the nature of BI systems) selection of descriptive characteristics and transforming them into real determinants, the AHP concept can be used to improve decisions on quality issues in BI system evaluation. #0#


Introduction
One of the key problems related to the development of civilization in the 21st century concerns the proper use of available information and converting it into useful knowledge. For managers and economists, the most important issue is the selection of such data analysis tools that will enable the use of the analysis results to make rational decisions and gain a competitive advantage on the market. One of the solutions that allow to carry out in-depth

Adam Stecyk
research and analysis of business development opportunities are business intelligence tools. As a reminder, there are many definitions of BI systems (Ranjan, 2009, p. 34, Hashmi, 2006, p. 113, Nedelcu, 2015, p. 345, Muntean, 2012, but for the purposes of this study, it is assumed that the concept of business intelligence concerns the collection and management of information to identify associations between key elements of the enterprise, enabling rational decisions based on reliable and up-to-date analyzes. In other words, business intelligence is "the capability of the organization or company to explain, plan, predict, solve problems, think in an abstract way, understand, invent, and learn in order to increase organizational knowledge, provide information to the decision process, enable effective actions, and support establishing and achieving business goals" (Wells, 2008, p. 67).
This outlined perspective indicates the need for a qualitative assessment of specific components at all levels: from the economic operator's strategy, through the timeliness of information, to the assessment of the quality level of the systems and tools. Because the issue of assessment is closely related to the problem of measuring the identified factors that determine individual quality areas, there is a need to build a model based on the appropriate selection of existing methods that can be used in assessing BI tools. The choice and adaptation of methods is subjective and remains the responsibility of the researcher analyzing a specific object in terms of quality assessment and measurement.
In the literature on the subject, many theoretical models of quality assessment and their application in the form of analytical methods and techniques can be found: the evaluation of quality systems based on the SERVQUEL and SERVPERF methods (Cronin, Taylor, 1994, p. 23;Stecyk, 2016, p. 232), as well as methods based on IT quality level assessment and user satisfaction models, such as the comprehensive assessment model (DeLone, McLean, 2003, p. 98); satisfaction assessment model and key system dimensions (Cai, Jun, Pham, 2007, p. 17) and a model of system quality measurement and user satisfaction (Doll, Torkzadeh, 1988, p. 39).
The second group of tools that can be used for solving decision problems are multicriteria methods. The best known are PROMETHEE I and II, MAPPACC, PRAGMA, artificial neural networks (Diech, Korbicz, Rutkowski, Tadeusiewicz, 2000, pp. 45-88) or the AHP (analytic hierarchy process) and ANP (analytic network process) methods (Saaty, 2002, p. 56) The aim of the article is to use the multi-criteria analytic hierarchy process method (AHP) to evaluate two business analytic framework, such as Microsoft Power BI and Tableau Public, according to the adopted criteria.

Characteristics of the analytic hierarchy process method AHP
The AHP method enables the practical application of a multi -criteria decision-making concept to a given research problem. It was developed in the seventies of the last century and has, so far found application in such areas as economics, management, transport, politics, education, medicine, technology, etc. The main assumption of the AHP method is the relative scale of assessments/priorities made using reversible pairwise comparisons for both countable and uncountable criteria. Figure 1 presents a model view of the AHP method, which consists of the following stages: 1. Determination of the main objective (the purpose of the study, e.g. selection decisions, determination of priorities, etc.). 2. Selection of the basic criteria (if the researcher deems it valid also sub-criteria) according to which the given problem will be analyzed (C1-C4). 3. Selection of alternatives (solutions), which will be assessed according to the adopted criteria (A1-A3).

Vol. 28/2, 4/2018
The analytic hierarchy process AHP for business intelligence system evaluation 4. Conversion of the adopted factors to the numerical form, using the fundamental scale 1-9, and creation of a square pairing matrix. 5. Creation of a normalized matrix on the basis of priority vectors and calculation: the largest own size of the matrix λ max , the consistency index C.I. (responsible for the lack of consequences of comparisons) and the consistency ratio C.R. (responsible for the coherence of comparisons in pairs). 6. Calculation of local priorities (according to points 4 and 5) for alternatives to each criterion. 7. Optional performance of the sensitivity analysis, giving the answer as changes in the weights assigned to individual criteria, may affect the final result.
Goal An important element of the method is the fundamental Saaty scale from 1 to 9, used to assess individual criteria and alternatives, which allows the application of expert knowledge and experience of the decision maker (or group of decision makers). Its main purpose is to indicate the number of times a specific element outweighs another in relation to the criterion being assessed. Table 1 presents the main assumptions of the discussed scale.

Adam Stecyk
The basic task of experts in the AHP method is to make reversible comparisons in pairs between selected criteria, for which a ij = 1/a ji and a ii = 1. Expert judgment is entered into a square pairing matrix (n × n) A = [a ij ], in which it performs n (n -1)/2 of these comparisons. The consequence of this is a pairwise comparison of each criterion using the fundamental scale 1-9 and assigning the inverse of the evaluation for the second element. In the case of expert group evaluations analyzing a specific decision problem, the geometric mean of all expert assessments should be used to calculate the final assessment. An example of a pairwise comparison matrix is shown in Table 2. Source: own elaboration.
The next stages of using the AHP method are based on the calculation of a normalized matrix for selected criteria and the largest own size of the λ max matrix. The author of the method proved that pairwise comparisons are all the more consistent, when the λ max value is similar to the number of matrix elements n. On this basis, the calculation of the C.I consistency index was proposed, according to the formula: and consistency ratio C.R., 100% . . . .

C I
where R.I. is a random consistency index, generated from several thousand matrices and proposed by the author in the form of Table 3. It is assumed that the value of C.R. for matrix (3 × 3) and (4 x 4), should be adequate accordingly, less than or equal to 5% and 8%, while for larger matrices it should not exceed 10% (C.R. ≤ 10%). In that case consistency ratio C.R. is accepted, and the comparisons made are considered consistent. If the number of 10% is exceeded by the C.R. the criteria evaluation should be repeated in order to get rid of the incompatibility of comparisons in pairs. The next stage of the AHP analysis is the application of the same analytical technique to the sub-criteria Vol. 28/2, 4/2018 The analytic hierarchy process AHP for business intelligence system evaluation (if identified), and next, to the proposed alternatives, which will allow determining the priorities within the adopted hierarchical structure.

Application of the AHP method to evaluate business intelligence system
Based on the concept of the AHP method, the following preliminary assumptions regarding the levels of the decision hierarchy to assess two business intelligence systems were adopted: Level 1  The basic problem in the analyzed study is the identification of key criteria determining the quality level of BI tools. The initial analysis of the issue made it possible to identify 13 major determinants, which were verified by using a combination of heuristic methods, such as the method of concepts transferring, the definition method, the incompetence method, and the analogy method. In this way, the original list was limited to 8 factors and formed the basis for building a matrix of assessments in pairs. In the next stage, the values in the columns were summed up and the matrix was normalized by dividing all the matrix values by the sum of the relevant column. Next, the weights of the selected criteria were calculated (the arithmetic mean for all rows of the normalized matrix, Table 4).
In order to calculate the value of λ max , the product of the weight vector and the non-normalized score matrix in pairs was used, which gave the possibility to calculate the sum for particular rows of the new matrix, which were then divided appropriately by the weights from the standardized matrix. The arithmetic mean of the obtained results gave the required value of λ max = 8.87.
In the next stage, the consistency index C.I. = 0,1237 was calculated, as well as the consistency ratio (for n = 8, R.I. = 1.40) C.R. = 0.0884 (8.84%). According to the adopted assumptions, the C.R. can be accepted (it is less than 10%), which means that the comparisons made within the accepted comparison criteria are consistent. At this stage of the analysis one can conclude about the importance of individual criteria for decision-makers to assess the evaluation of business intelligence systems. The most important criterion in this study is the in-depth analyzes (IDA = 0.33) and simple analyzes (SA = 0.23); the least important are the interface (I = 0.03) and tools for sharing and publication (SP = 0.04). Figure 2 presents weights for selected criteria in the evaluation of BI systems.
The next stage of the analysis is the calculation of the so-called local priorities for selected alternatives (Power BI and Tableau) in relation to the eight selected criteria. As the analytical technique has been presented in relation to the level of criteria, the method of calculating local priorities is presented only for the most important criterion, in-depth analyzes (Tables 5-6). The final results for other criteria are shown in Table 7.  The construction of the eight matrix for each criterion and two alternatives (Power BI and Tableau Public) made it possible to calculate local priorities for each criterion. Then, weighed priorities (the product of local priorities and criteria weights) were calculated, which consequently enabled calculating the results of final preferences (sum). Conducted multi-criteria analysis using the analytic hierarchy process method (AHP) gave the answer to the question about the quality level of two business intelligence systems (Power BI and Tableau Public), assessed against each other and against eight selected criteria. The results indicate a slight preference for Power BI (51%) towards the Tableau Public (48%). In addition, the conducted analysis indicated the level of significance of selected criteria and may be a starting point for further, in-depth analyzes, eg. by performing a sensitivity analysis and changes in the weightings of individual criteria against each other.

Conclusions
To sum up the above considerations, it should be noted that the adoption of specific methodological assumptions in relation to the problem under consideration took into account all the difficulties and imperfections of the method (complexity of the problem, lack of universal methods, the problem of accuracy and reliability of criteria selection and evaluation, etc.). The above statement is in line with K. Popper's thesis: For the empirical method, it is characteristic that the system under review is available for falsification in all imaginable ways. The purpose of this method (the methods of overthrowing) is not to save the life of unserviceable systems, but rather to throw them all into the struggle for survival and to choose the one that is the best match (Popper, 2002, p. 121). The outlined perspective indicates the necessity of continuous improvement of methods and models used to measure and evaluate the IT systems, in order to understand the impact of technology on other spheres of social and economic life.