A Critical Assessment of Quality and Validity of Composite

A Critical Assessment Of Quality And Validity Of Composite-PDF Download

  • Date:13 Sep 2020
  • Views:3
  • Downloads:0
  • Pages:20
  • Size:271.88 KB

Share Pdf : A Critical Assessment Of Quality And Validity Of Composite

Download and Preview : A Critical Assessment Of Quality And Validity Of Composite

Report CopyRight/DMCA Form For : A Critical Assessment Of Quality And Validity Of Composite


surprising that composite indicators are widely used for transforming multi dimensional phenomena. into a single or a fewer more manageable number of dimension s. This is certainly the case with measuring innovation Annually published country rankings for the. WIPO INSEAD Global Innovation Index GII or the European Commission s Summary Innovation. Index SII continue to attract broad public attention 2 Over the last decade the underlying concepts. selection of indicators and modeling choices weighting aggregation methods for these indices have. been critically assessed Grupp and Mogee 2004 Schibany and Streicher 2008 Grupp and Schubert. 2010 Gault 2013 Hollanders and Foray 2016 among others but also gradually refined Sajeva et. al 2005 Hollanders and van Cruysen 2008 Saisana and Filippas 2013 Along the way composites. have exposed shortcomings in their component indicators triggering changes in the data collection. and dissemination process While composite indicators are apparently here to stay the very fact that. there is an ongoing critical discussion keeps the agora model Barr 2001 2010 of indicator. development alive Hardeman and Vertesy 2016, This paper aims to support this discussion by assessing the quality and validity of the most commonly. used composite indicators of innovation In our proposed framework the quality of an indicator relates. to its statistical properties Saltelli 2007 Saisana et al 2005 2011 while the validity of an index. relates to the link between component indicators or aggregates and the conceptual framework JRC. OECD 2008 Saltelli et al 2013 In the case of country level innovation indicators validity depends. on how well an innovation index quantifies what is relevant for innovation policy The paper. concludes that both quality and validity are more normative concepts than what is usually. acknowledged This becomes evident from taking an evolutionary approach recognizing the co. evolutionary dynamics of indicator development within changes in the global economic environment. the goals of science technology and innovation policies the availability of data as well as analytical. and measurement tools In this context it is striking that the past decade has brought more. incremental rather than radical changes in composite indicators of innovation. 2 Measuring innovation with composite indicators, Before assessing composite indicators of innovation we need to address some critical questions such. as 1 How to define innovation 2 What is the meaning of indicators and how are they used or. misused for innovation policy 3 What is the connection between innovation policies and. indicators 4 What are the most widely used composite indicators of innovation The aim of this. section is to give an overview of some of the underlying concepts and practices. 2 1 Innovation from concept to measurement, Our paper focuses on composite indicators that aim to compare countries in terms of innovation The. third wave of the Oslo Manual reflects the strongest common understanding of innovations the. introduction of a new or significantly improved product good or service or process a new marketing. method or a new organizational method in business practices work place organization or external. relations OECD Eurostat 2005 While this definition is broad in the sense that it encompasses not. only technological innovation its focus is the firm that brings novelty to the broadly understood. market Aggregating the innovative activities of many firms to define and measure the innovative. activities of geographical areas such as countries is not straightforward and entails making many. choices related to the weighting of different types of activities i e Should innovations of all types of. all degree of novelty in all economic activities count equally If not what are the right weights to. See i e Global Innovation Rankings The innovation game The Economist 17 Sep 2015 or Innovation in. Europe Improves But Only in a Handful of Countries The Wall Street Journal 14 Jul 2016. Moreover scholarship on innovation showed that the innovation process occurs not only within firms. rather in a complex interaction that involves other firms as well as research organizations and. universities public funding agencies regulating bodies or users In other words if one aims to. measure how countries innovate the performance of national systems of innovation Lundvall 1992. 2007 Nelson 1993 Freeman 1995 need to be defined and quantified challenges that multiply the. already existing complexities of measuring the innovation performance of a set of firms 3 Furthermore. comparability of countries is influenced by structural scale and time factors that are central to. innovation in a firm Given the breadth of what national innovation systems encompass when it. comes to measurement authors often focus on selected elements of it or for a certain purpose i e. technological capabilities or innovative capacities scientific technological or innovative interactions. the creation of new inventions etc In sum complex often untraceable set of conceptual choices are. inherently involved in the measurement of innovation at aggregate levels. Driven by the definitions multiple means are used to innovations direct measures of innovations. following the object the innovation itself or subject the innovator approach and making use of. surveys or expert opinions exist alongside indirect measures of the innovative process These include. information on the innovation efforts R D or non R D innovation expenditures personnel etc and. outcomes such as patents or changes in the broader economic performance or structures see Smith. 2005 Gault 2013 Accordingly statistical data stemming from the various ways to measure. innovation is loaded with a similar depth of variety as the reliability of data from innovation surveys. as well as indirect measurements may be diverse Nevertheless a significant part of innovation survey. data can be considered of sufficient quality for the purposes of robust econometric analyses Mairesse. and Mohnen 2010, The measurement of innovation system performance is a more complex task which depending on. the purpose may focus on the system functions Hekkert et al 2007 its ability to achieve certain. outcomes or directly capture the performance of its elements such as actors networks and. institutions We conclude that beyond the identification of what is measured and how is measured the. outcomes of measurement are also influenced by why we intend to to define and measure innovation in. a firm or in a country Depending on whether the aim is to make an informed judgement to achieve. high returns on investment or to understand the dynamics of long term economic development. technological change and catch up see Fagerberg and Srholec 2008 innovation performance may. turn out to differ significantly,2 2 Indicators for policy indicators as models.
In the context of evidence based policies metrics are ever more frequently called upon to legitimate. policy interventions Wilsdon et al 2015 The ability to measure innovation is perceived crucial for. the design of effective and efficient policies Grupp and Mogee 2004 and the lack of measurement is. often a way to hide weaknesses in the innovation system Benchmarking and evaluation of distant. mostly similar entities the key purposes of using indicator is therefore highly informative. Nevertheless especially because of the widespread political implications of governance by. indicators there is reason to exercise caution Davis et al 2012 In the domain of science. technology and innovation indicators the cautionary remarks are also reflected in the scholarly. literature both in terms of technical critiques Grupp and Mogee 2004 Grupp and Schubert 2010 as. well as normative critiques related to development process or legitimacy see Barr 2010. Debates around the development and use of indicators are centered on the understanding that. quantification is always a conversion with involve a certain degree of simplification of reality and the. Not to mention that while national institutions matter the geographical scope of interactions that influence. innovation often encompasses many countries, societal pressure calling for comparisons Indicators seen more optimistically offer learning and self. discovery through the measurement comparison and monitoring process and may offer early warning. signal show trends and bottleneck and identify strength and weaknesses On the other hand. indicators are models and thus selective Normative choices including the ability to manipulate by. selection and weighting made under uncertainty are inherent in any kind of indicators Saltelli et al. 2013 but at the same time it is important to recognize that indicators gain meaning only in a socio. political context There is a two way interaction between indicators and policies indicator. development is shaped by policy needs and the policy discourse Godin 2002 while in turn. indicators influence the policy discourse, 2 3 Innovation policy perspectives and the demand for indicators. The aspects of innovative performance governments are interested in monitoring and influencing are. related to the innovation policy paradigm Is innovation policy conducted in isolation or as part of a. coordinated set of policies What is the rationale for intervention. If the rationale of innovation policy is limited to encouraging firms to engage in innovation and. entrepreneurial activities governments are concerned with providing framework conditions in which. firms can operate and recognizing and removing obstacles understood as market or system failures. This approach is likely associated with a neutral understanding of innovation the more innovation. the better i e larger share of firms reporting to innovate larger spending on innovation etc. regardless of the types of innovation the size or sectoral belonging of firms that innovate or how the. outcomes relate to other policy goals such as employment if the kind of innovation pursued is not. replacing jobs with machines or redistribution of incomes It is assumed that the outcomes of. innovation and technological change will diffuse in the economy spillover effects will lead to social. gains These implications are not the main concern of innovation policy understood in this narrow. sense On the other hand however scientific and technological activities skills and capacity to. innovate or engage in entrepreneurial activities what may be a precursor to the innovation process. is of interest, In another understanding innovation is perceived as a tool to achieve strategic aims Paraphrasing. Lundvall and Borr s 2005 such innovation policy is essentially about the promotion of production. diffusion and use of innovations in order to realize national or supra national objectives An objective. may be economic development and catch up for which innovation serves to accelerate growth and. increase competitiveness of industries Other objectives could be the improvement of military. capabilities or in contrast finding new ways to tackle hitherto unresolvable societal problems i e. improvements in health environment or the survival of humanity Tools to achieve such policies. with fostering system transitions Hekkert et al 2007 or creating new markets Mazzucato 2016. The main distinction of this approach is that innovative activities of a certain character are preferred. over another depending on the aim, Needless to add there may be more approaches for innovation policy and these two approaches may. be extremes However the key point is that they are rarely followed consistently in practice All too. often an amalgam of the two emerges from the political process of harmonizing the strategies of. different stakeholders ministries agencies business associations etc This is not without problems. as central questions such as the effectiveness of certain innovation policy measures depends on the. perspective chosen The different approaches may necessitate fundamentally different monitoring. mechanisms, How does this influence the demand for and the use of composite indicators While the strength of.
scoreboards lies in allowing interested parties to monitor a variety of related processes in accordance. with their preferred approaches and strategies there are considerable overlaps as well as a few trade. offs between processes that indicators measure Carefully constructed composite indicators have the. potential to summarize indicators without a significant loss of information highlight an unobserved. concept or normative choices and help analyze desired aspects of innovative performance Saltelli. 2007 Ravallion 2012 However as statistics does not offer solutions for normative decisions one. cannot emphasize enough the need for clarity in the approaches and strategies for innovation policy. even for the purposes of seemingly neutral practices such as monitoring Without this any indicator. composite indicators included can be of limited support to policy. 2 4 Composite indicators of innovation, It is not the main purpose of this study to offer a complete survey or a taxonomy of composite. indicators of innovation Nevertheless a brief overview of composite indicators of innovation is in. place here to inform our selection of the indices we investigate closer in our study. Composite indices of innovation come in different shapes and sizes Some are side products of. scoreboards or dashboards of indicators published by international organizations or think tanks while. A Critical Assessment of Quality and Validity of Composite Indicators of Innovation D niel V rtesy1 Joint Research Centre of the European Commission DRAFT PAPER SUBMITTED TO THE OECD BLUE SKY III FORUM ON SCIENCE AND INNOVATION INDICATORS Do not cite without author s consent 29 Jul 2016 Abstract Composite indicators are widely used to capture both the process and the outcomes of complex

Related Books