IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 61, NO. 8, AUGUST 2012 2107 Measurement Fundamentals: A Pragmatic View Luca Mari, Paolo Carbone, Senior Member, IEEE, and Dario Petri, Fellow, IEEE Abstract—Measurements are more and more required in an increasing variety of human activities to acquire reliable information useful for effectively supporting decision-making processes. Furthermore, the entities whose properties are to be measured and the measuring systems are becoming increasingly complex, hardly to be modeled and managed. In this rapidly evolving scenario, several issues concerning the fundamentals of measurement science and technology arise. The purpose of this paper is to discuss some aspects of the crucial question about which evaluation processes can be considered measurements. Rather than focusing on formal conditions or technological constraints, we propose a pragmatic characterization of measurement, under the assumption that a better comprehension of the concept can be achieved by identifying and discussing the basic features which justify the reliability attributed to measurement results. With such an interdisciplinary approach, this work aims at promoting a broad discussion among interested researchers, even working in different scientific disciplines, so to increase the synergies among different research areas and to improve the body of knowledge about measurement fundamentals. Index Terms—M/Measurement, M/Modeling, M/Multivariable systems, U/Uncertain systems, U/Uncertainty. I. I NTRODUCTION N MANY, if not all, human activities, measurement is considered a fundamental process to obtain reliable information on the empirical world, in view of the Galilean motto of measuring what is measurable and making measurable what is not yet. Nowadays, a systematic adoption of measurement is particularly solicited by the widespread application of technoscience in the social, industrial, economical, . . . fields. Once specifically aimed at evaluating physical quantities, measurements are, today, more and more required in biology, medicine, economy, sociology, psychology, . . . Furthermore, measuring systems are becoming complex entities, in which the customary measurement techniques have to be extended to multivariate measurands, and the measurement of physical quantities has to be often complemented by the measurement of nonphysical properties, sometimes called “weakly defined measurement” [1] or simply “soft measurement.” An excellent reference on this matter is the document Evolving Needs for Metrology in I Manuscript received November 9, 2011; revised January 4, 2012; accepted January 9, 2012. Date of publication April 30, 2012; date of current version July 13, 2012. The Associate Editor coordinating the review process for this paper was Dr. Wendy Van Moer. L. Mari is with Università Carlo Cattaneo (LIUC), 21053 Castellanza, Italy (e-mail: lmari@liuc.it). P. Carbone is with the University of Perugia, 06125 Perugia, Italy (e-mail: carbone@diei.unipg.it). D. Petri is with the University of Trento, 38050 Trento, Italy (e-mail: petri@disi.unitn.it). Digital Object Identifier 10.1109/TIM.2012.2193693 Trade, Industry and Society and the Role of the BIPM [2], which states, for example, that, currently, “an estimated 80% (of the world trade) is affected by standards and regulations” and that, according to various studies, “the cost to producers and service providers of complying with standards can be 10% of production costs.” Of course, measurement is the basis to assess such compliance. The Bureau International des Poids et Mesures (BIPM) document lists some of the application areas where the role of measurement is increasingly critical: They include “transport; information technology, navigation, and telecommunications; electronics and optics; electromagnetic and ionizing radiation; energy; climate change and environmental and pollution control; clinical chemistry and laboratory medicine; food safety; antidoping; pharmaceuticals; and forensics and security.” This evolving scenario arises several significant issues for measurement science, not only at the operative level—for example, about the possibility to apply the now-standard procedures of uncertainty evaluation specified by the Guide to the Expression of Uncertainty in Measurement [3] in such diverse fields—but also, and primarily, in reference to the state and the nature of measurement science itself. A fundamental critical problem relates to the very concept of measurement: Is its acceptation, as commonly understood in the measurement of mechanical, optical, thermal, electrical, . . . quantities, already adequate and directly applicable in this broader context? The question is not only lexical, i.e., whether a single entry in a vocabulary may accommodate all usages of the term or only related to the, however important, goal of mutual understanding, which is the basic goal that drove a number of leading international organizations to gather into the Joint Committee for Guides in Metrology and to produce the International Vocabulary of Metrology (VIM3) [4]. Rather, an inquiry on a widely shared meaning of “measurement” is useful to give a convincing justification to the customary claim of the “special reliability” of measurement itself, which is surely not assumed in the case of, e.g., subjective judgment or guess: The public trust attributed to measurement results and the resource spending acceptable for measurement should not be approved for such other activities. If the generic process of assigning a quantity value to a quantity of interest is termed “evaluation” (as in the abstract case of “function evaluation”), so that measurement, subjective judgment, and guess are all examples of evaluations, the problem may be then formulated: What does it characterize measurement as a specific kind of evaluation? Since the problem is not conceptually new (although perhaps this formulation is), some answers have been already proposed in the past. At least three well-known general standpoints can be mentioned. 0018-9456/$31.00 © 2012 IEEE 2108 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 61, NO. 8, AUGUST 2012 First standpoint: According to Euclid (Elements, Book V, Definitions 1–3), “a magnitude is a part of a magnitude, the less of the greater, when it measures the greater; the greater is a multiple of the less when it is measured by the less; and a ratio is a sort of relation in respect of size between two magnitudes of the same kind.” The hypothesis is assumed here that a quantity is measurable because it can be represented as the ratio of (integer) numbers. While this standpoint has shaped the traditional concept of measure, the lack of any specifications on the way that values are obtained (“according to my experience, I can see that this object is 1–2 m long” expresses, in fact, a ratio of two “magnitudes”) makes it useless for our purpose. Furthermore, even in the case of physical quantities, it is, today, customarily accepted that ordinal properties can be measurable (the VIM3 calls them “ordinal quantities” and mentions four examples: Rockwell C hardness, octane number for petroleum fuel, earthquake strength on the Richter scale, and subjective level of abdominal pain on a scale from zero to five). This makes “Euclidean quantities” a specific, although very important (quantity calculus applies only to them), case of a broader class. The aforementioned considerations highlight that the issue of measurability relates in principle to the very general concept of property, for which some more or less synonymous terms can be used, such as “feature,” “aspect,” “characteristic,” . . ., but whose definition, if even possible, is not a matter of measurement science. Second standpoint: Measurement is what is performed by a (physical) measuring system, i.e., a properly calibrated and operated instrument realizing a physical transduction effect. Despite its operational roughness, this standpoint has shown its effectiveness for centuries, and it can be considered the (default) acceptation in physical sciences and engineering. On the other hand, bounding measurement to physical instrumentation removes our problem without solving it [5]: The lexical move of making “measurement” and “physical measurement” synonymous is plainly inappropriate in the BIPM perspective of the aforementioned “evolving needs” for metrology and toward soft measurement. Third standpoint: As the outcome of a critical analysis on the possibilities of applying measurement in social sciences, measurement has been axiomatized as a morphic mapping from quantities to quantity values and, actually and more generally, from properties to property values. Morphisms are structurally constrained mappings which require that, in assigning a property value to a property, empirical observable relations between properties are preserved in abstract relations between property values [6]–[8]. This condition, which is in principle trivial, allows expressing the empirical relations occurring between instances of properties by means of properly chosen linguistic entities. For example, a morphic constraint in the ordinal case is: If a resistor x1 experimentally behaves by exhibiting a higher resistance than a resistor x2 , then the resistance value assigned to x1 must be greater than the resistance value assigned to x2 . Hence, the derived so-called rep- resentational theories of measurement [9] generalize the Euclidean standpoint by releasing some algebraic constraints yet ensuring that measurement results actually reflect the behavior of the examined property. However, representational theories alone are still unable to discriminate between, e.g., measurement and subjective morphic judgments [10]. As a synthesis, our claim is that the traditionally available positions are not adequate to found an encompassing but scientifically well-defined concept of measurement, able to cope with the current “evolving needs”: The new challenges require a new characterization. In the following section, measurement is presented as an evaluation process and, despite this stillgeneric description, some of its basic features are introduced. In Section III, measurement is characterized as an objective and intersubjective evaluation, according to a well-defined technical meaning of the concepts of objectivity and intersubjectivity. This allows arguing, in Section IV, on the twofold nature of measurement, being at the same time a model-based and an operative evaluation, whose implications are discussed in Sections V and VI with respect to measurement uncertainty, also in reference to the critical case study of the measurability of research quality. II. M EASUREMENT AS E VALUATION The background assumption that we are introducing— measurement is a specific kind of evaluation—is still very generic, and still, it throws some light on some fundamental features that measurement shares with any other evaluation. Pointing out that measurement is an evaluation abstractly characterizes it according to a black-box model, as a process realizing a functional transformation of an input entity to an output entity, called, in this case, the measurement result. This concept of evaluation as transformation is very general. For example, the mathematical transformation which takes a real number x and produces the real number 2x is an evaluation, in the usual sense that x, the input entity, is evaluated by means of the transformation and 2x is the resulting value. Output entities of evaluations are not required to be (real) numbers: They could be not only vectors or tensors but also subsets or more complex entities such as probability density functions and even linguistic (ordered or nonordered) labels. On the other hand, not any transformation is an evaluation, as it occurs, e.g., in the case of chemical reactions. Evaluation outputs and measurement results, in particular, are information (and not empirical) entities—let us call them symbols, according to a customary terminology, adopted in particular in the representational theory of measurement—although, of course, a physical support is required for conveying (transferring, storing, presenting, . . .) them. Hence, measurement is a map between the empirical world and the symbolic world, aimed at associating symbols to the (still unspecified, for the sake of generality) empirical entity under measurement in order to describe it. A dependence relationship is assumed between measurement-process input and output, typically interpreted according to a causality principle: Outputs are effects of inputs. This is particularly the conceptual basis for all the MARI et al.: MEASUREMENT FUNDAMENTALS: A PRAGMATIC VIEW developments related to measurement-uncertainty-evaluation techniques, which explain time variability of output values by assuming that at least one component of the empirical world (the entity under measurement, the measuring system, or the surrounding environment) had changed, due to the presence of so-called hidden variables, introduced to maintain causality (it is well known that in quantum physics, the relations between causality and measurement uncertainty are much more complex, but we will not consider this subject here). No further constraints—and, in particular, no specifications on the structure of the process—are imposed on evaluations as such. Moreover, measurement is an informative evaluation, aimed at acquiring and conveying information on objects of the empirical world. According to the Shannon’s theory of information, this implies that the set of possible values includes at least two elements and that a prior probability distribution can be associated to such set so that at least two elements (taking into account the discrete case) have a nonnull probability to be chosen as measurement result. This excludes constant functions from the candidates to measurement and highlights that measurement itself can be thought of as a process of selection of an element from a previously chosen set. Furthermore, the mathematical theory of information guarantees that extending the set of values does not reduce—and usually, in fact, increases—the average quantity of information, evaluated as an entropy and conveyed by an evaluation [11]. A further general condition characterizing measurement as a specific kind of evaluation relates to the entity under measurement. In fact, very diverse entities can appear as the input of the process: The VIM3 writes about “phenomena, bodies, or substances,” and by widening the context, pieces of software, individuals, industrial processes, organizations, and any perceivable thing can also be considered as such. Let us introduce the term object under measurement to denote them. As a basic fact, the relation between objects under measurement and measurement results is generally many-to-many, i.e., the same object can be evaluated to different results, and different objects can be evaluated to the same result. The first condition is interpreted by assuming that any object has multiple aspects, i.e., properties, and that subject to measurement is not the object as such but a property of it. Hence, measurement is an informative property evaluation. This basic assumption is fully consistent with the functional modeling as far as such general properties (such as electrical resistance, pain intensity, and attitude to scientific research) are superposed to the mappings, according to the logic propertyi (objectj ) = resultij where propertyi (objectj ) represents the single instance of a general property (such as the resistance of this resistor, the intensity of the pain felt by this patient, and the attitude to scientific research of that Ph.D. student) for the given object. Therefore, measurement assigns a symbol—the measurement result—to the object under measurement, so that the symbol intends to provide descriptive information on the current state of the object with respect to one of its properties, called the measurand, i.e., the property intended to be measured [4]. 2109 While this general characterization leads to some nontrivial consequences, what has been presented so far hardly can be accepted as sufficient to define measurement: Some further conditions are required. III. M EASUREMENT AS O BJECTIVE AND I NTERSUBJECTIVE E VALUATION The fundamental intuition that measurement is a property evaluation whose results convey reliable information on the measurand is not related to the nature of the object under measurement or of the measurand nor to the algebraic structure of the set of property values and, in principle, not even to the structure of the measurement process itself. Rather, such reliability can be characterized in terms of two general features expected for measurement results, which are supposed to convey. 1) Information specific to the measurand and, therefore, to a given property of the object under measurement. This means that the provided information should be independent of any other property of the object or the surrounding environment, which includes both the measuring system and the subject who is measuring. This corresponds to guaranteeing that measurement results actually provide information about the measurand and not of some other property. It is a sometimes trivial, sometimes very complex to satisfy, condition about the appropriate attribution of information to its claimed object: Hence, it is a requirement of objectivity. 2) Information interpretable in the same way by different users in different places and times. This corresponds to guaranteeing that measurement results are expressed in a form independent of the specific context and only referring to entities which are universally accessible, so that the meaning of a measurement result is unambiguous and can be easily reconstructed in principle by anyone, possibly on the basis of suitable conventions: Hence, this feature expresses a requirement of intersubjectivity. Physical measurements usually embed these features directly in the structure of the measuring instrument, designed and operated so to behave as a transducer whose empirical output, called the instrument indication, ideally should depend only on the measurand, or a property functionally related to it, thus assuring that the information it provides relates specifically to the object under measurement and therefore confirming objectivity. Moreover, the instrument indication should be mapped to measurand values through instrument calibration, which makes the instrument output traceable to a primary measurement standard. Hence, different instruments traced to the same standard provide comparable information so assuring intersubjectivity. On the other hand, such conditions of objectivity and intersubjectivity do not imply any specific constraint on the realization of the measuring process, so that, in principle, they can be introduced as requirements also in the case of the evaluation of nonphysical properties. These two features are independent with each other, in the sense that an evaluation might produce objective but nonintersubjective results (as in the case of the usage of an 2110 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 61, NO. 8, AUGUST 2012 uncalibrated measuring system), or vice versa intersubjective but nonobjective results (e.g., if they are expressed in the customary format for quantity values, i.e., number times measurement unit, but they have been obtained just at random). This is the reason why both of them are considered to be required here ([12] proposes to call “premeasurement” an objective but still nonintersubjective evaluation). For the condition of intersubjectivity to be fulfilled, measurement results are required to convey information not only on the measurand value but also, reflexively, on the actual trust attributed to that value. The classical assumption that quantity values are real-valued numbers (based on the hypotheses that “numbers are in the world,” as Kepler once wrote along the Pythagorean tradition, and that many physical phenomena are continuous or at least their variation is such) hides this requirement, under the wishful, but wrong, thinking that progressively increasing the quality of measurement would reduce, at then eliminate, all measurement errors, and the “true value” of the measurand would be discovered accordingly. Anyway, even if measurement might be hindered by errors (e.g., using a measuring instrument which is no longer calibrated), an appropriate intersubjective communication of measurement results calls for a statement about the degree of belief that the measurer attributes to the provided information. On this basis, the position emerged that a measurement result should specify the amount of information it is aimed at providing and, thus, that it is not complete if the uncertainty of the stated measured property value is not included. Specifically, the VIM3 defines “measurement uncertainty” as a “nonnegative parameter characterizing the dispersion of the quantity values being attributed to a measurand, based on the information used.” Measurement uncertainty is thus an overall concept, which synthesizes the effect on the measurement result of multiple contributions (including, where applicable, measurement errors), from both the experimental and the modeling steps of the measurement process, into a single piece of information. In practice, neither objectivity nor intersubjectivity is Boolean (i.e., yes–no) features. Indeed, the result of any measurement is affected by the way the measuring system was calibrated, and it conveys information not only on the measurand but also, unavoidably, on other properties of the surrounding environment. The fact that these two features admit intermediate levels emphasizes the need of introducing a further pragmatic component in measurement: The threshold over which the evaluation results are “sufficiently” objective and “sufficiently” intersubjective to be considered measurement results is set in reference to the expected use of the provided information, typically the support to a decision-making process. This perspective agrees with the so-called goal–question–metric paradigm [13] and considerations reported in the ISO 9000:2005 standard [14]: Any measurement process must first consider the intended use of the experimentally obtained information, in order to satisfy requirements predefined by the final user of that information. It is worth noticing that the evaluation of measurement uncertainty alone does not include the definition of thresholds and, thus, it may lead to different interpretations by varying the context: As an example, while measurement uncertainty of 10 kg could be considered unacceptable in several contexts, it Fig. 1. Measurement uncertainty as an essential tool for decision making. might be fit when it comes to extraction operations in mineral engineering. In particular, expressing uncertainty as a suitable standard deviation, as recommended by the Guide to the expression of uncertainty in measurements (GUM), allows comparing the quantity of the available information to the minimum quantity assumed by design as needed for effectively supporting the decision making, which is called the target measurement uncertainty (“measurement uncertainty specified as an upper limit and decided on the basis of the intended use of measurement results,” according to the VIM3). Hence, target measurement uncertainty might be thought of as a quantitative means to formalize the concept of “sufficient” objectivity and intersubjectivity with respect to the decision under consideration. In a favorable context, in which decisions can be actually made with the support of measurement results, the relations among the chained concepts of defined by specification (as expressed, e.g., by a nominal value and a tolerance for the property under consideration), required for decision, and experimentally obtained quality of information can be then shown as in Fig. 1. As a consequence, the pragmatic component of measurement could be stressed up to a point that an evaluation producing results whose uncertainty is greater than the required target measurement uncertainty would not be acknowledged to be a valid measurement, because, in fact, its results are not useful to support its intended use. When measurement results are exploited at a later time and possibly in a different context, information about their intended use might not be available at measurement time. This conclusion is still applicable, and measurement uncertainty still provides a means to judge the quality of a measurement by comparison with pre- or postdefined thresholds: Also in these cases, if targets of uncertainty thresholds cannot be met, measurement results cannot be used. Concerning the expression of measurement uncertainty, it could be also noted that the extension from quantities with unit to generic properties generates a (mainly still open) issue: Indeed, the GUM framework does not apply to ordinal quantities, nor as a consequence to nominal properties, since mean value and standard deviation are not empirically meaningful for them [15]. While other, algebraically weaker, statistics are available and well known (e.g., median and percentiles in the ordinal case), a widespread agreement on this subject is still to be reached. A plausible backgrounder might be the mentioned Shannon’s theory of information, which builds on the assignment of probability density functions to value sets and allows interpreting uncertainty in the selection of values by means of an entropy function, from singletons (no uncertainty and null entropy) to uniform distributions (complete uncertainty and maximum entropy). MARI et al.: MEASUREMENT FUNDAMENTALS: A PRAGMATIC VIEW Fig. 2. 2111 Measurement as a two-stage process. IV. M EASUREMENT AS A M ODEL -BASED AND O PERATIVE E VALUATION The customary structure of a physical measuring system assumes that by means of measurement, an empirical entity (the measurand) is represented by an information entity (the measurand value including its uncertainty). This effect of bridging two different “worlds” (as, for example, K.R. Popper called them, the “World 1 of physical entities” and the “World 3 of products of human minds” [16]) results from a two-stage process: 1) a modeling stage, i.e., a set of conceptual activities needed to define the measurand and to model its relationship with the measurement result; this is achieved through a proper modeling of the whole experimental environment, called measurement context in the following, in which measurement is expected to be performed, thus including the object under measurement, the measuring system, the subject who is measuring, and the empirical surrounding environment; 2) an operative stage, in which—on the basis of the knowledge about the measurement context provided by the modeling stage—experimental activities are performed, so implementing the mapping between the empirical world and the world of symbols. In its turn, the operative stage is organized as a two-step process: 1) an experimental step, in which the measuring instrument interacts with the object under measurement and as a result of the interaction, it produces an observable output, i.e., an instrument indication, which is (ideally) caused by the measurand; 2) a representational step, in which, from such indication, the cause which produced it is reconstructed and symbolized, so properly assigning a value, and generally a related uncertainty, to the measurand; this activity is performed by using the information about the measurement context achieved at the descriptive level and, in particular, the information provided by the calibration process of the measuring system. These two stages allow implementing and, in any effective measuring system, are expected to implement the requirements of objectivity and intersubjectivity of measurement results. In the simplest case, the measuring instrument input is the measurand, so that the two operative steps are sometimes considered as inverse with one another: The experimental step instances the transduction function, which is then inverted in the representational step which, for this reason, is sometimes called measurand reconstruction. According to this perspective, the whole operative stage of measurement is said to implement an identity function. This position is, in fact, wrong because of a wrong ontological superposition: The measuring instrument input is the measurand, i.e., a property, not its value. This basic case is generalized by admitting the measuring instrument not to be perfectly selective, i.e., the indication it produces depends not only on the measurand but also on other influence properties, so that the mapping from indications to measurand values must also include the information on the effect of such influence properties and properly correct or compensate it. This description of measurement as a modeling and operative process is sketched in the diagram of Fig. 2: Once the model of the measurement context has been obtained, the measurand is acquired in the experimental step, in which the measuring instrument produces an indication that may depend on influence properties. Also, the state of the object under measurement—and, consequently, the measurand—may be affected by influence properties. Finally, on the basis of the information contained in the model of the measurement context, the values measured for the influence properties, and the information obtained in calibration, the representation stage handles the transducer indication and the known values of influence properties to provide the measurement result. The further, and most significant, generalization introduces the hypothesis that the property subject to measurement and the property intended to be measured (i.e., the measurand) may be different and, therefore, that the input property of the transducer carries information on a property which is different from the measurand. This difference can take into account the modification induced on the object under measurement by its interaction with the measuring instrument, and as such, it may be corrected by suitably defining the measurand reconstruction function. This is typically the case in which the measurand (e.g., the intensity of electrical current in a given circuit) and the property actually subject to measurement (the intensity of electrical current in the circuit when coupled with a given measuring instrument) are instances of the same general property input of the transducer (intensity of electrical current). On the other hand, such two entities might be instances of different general properties, as when the property subject to measurement is a current and the measurand is a voltage. In 2112 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 61, NO. 8, AUGUST 2012 these cases, traditionally called derived (or indirect) measurements, the concept of measurand reconstruction is complex, since it has to include the relation between the two general properties (e.g., as expressed by a physical law; in the example, the Ohm’s law), and thus at least some aspects of a domain theory. These cases are usual in soft measurement, where the measurand is sometimes a very complex entity, such as the attitude of a given individual to a given task or the performance of a given R&D department, and the information on it is obtained via a set of indicators, the properties actually subject to measurement. A substantive theory is then crucial to justify that the searched information on the measurand can be derived from such indicators [17], [18]. Whenever such a theory is not available, not an uncommon case in soft measurement, measurement validation becomes a critical issue: An objective support has to be given to the claim that measurement results actually refer to the stated measurand. Common approaches to validation in this case are based on the use of statistical correlation techniques and cause–effect analysis. The influence of both the measuring system and the modeling stage on the information provided by the measurement result will be analyzed in the following sections. When dealing with modeling activities, the specific case of research quality measurement will be considered in order to support the discussion with a hot topic in many academic and research institutions, whose proper application could benefit the knowledge of measurement fundamentals. V. I NFLUENCE OF M EASURING S YSTEM ON M EASUREMENT U NCERTAINTY The availability of a measuring system, i.e., a coordinated set of one or more measuring instruments, together with a measurement procedure, i.e., a detailed description of how such system has to be operated to obtain measurement results, is the customary condition assumed for performing measurement. Hence, from the metrological characteristics of such systems and the knowledge of the measurement procedure, some critical components (and sometimes all components) of the uncertainty budget are derived. A complete metrological characterization of any specific class of measuring systems requires the specification of multiple features (e.g., sensitivity, selectivity, resolution, dead band, drift, response time, frequency response, measuring interval, operating conditions, energy requirements, lifetime, . . .), which can be classified according to different criteria (e.g., features related to dynamic versus static behavior). On the other hand, we claim that at a very fundamental level, the system behavior is characterized by a single general feature: its stability, i.e., the “property of a measuring instrument, whereby its metrological properties remain constant in time.” This requirement of constancy in time applies in reference to two complementary kinds of measurement conditions (definitions taken again from the VIM3): 1) the repeatability condition of measurement, i.e., the “condition of measurement out of a set of conditions that includes the same measurement procedure, same operators, same measuring system, same operating conditions and same location, and replicate measurements on the same or similar objects over a short period of time”; 2) the reproducibility condition of measurement, i.e., the “condition of measurement out of a set of conditions that includes different locations, operators, measuring systems, and replicate measurements on the same or similar objects.” Under repeatability conditions, the short-range stability of the system behavior is assessed. It is a check which can be performed independently of any calibration: With the only assumption that the (even unknown) input property to the transducer and the influence properties are not (significantly) changing during the repetitions, a sample of indications is obtained, and the information it conveys can be expressed by means of a scale (or dispersion) statistic [19], such as the sample standard deviation. Hence, such short-range stability is related to measurement precision, i.e., the “closeness of agreement between indications or measured quantity values obtained by replicate measurements on the same or similar objects under specified conditions” (VIM3 definition), where the “specified conditions” are repeatability conditions. In a complementary way, under reproducibility conditions (but including a single measuring system), the long-range stability of the system behavior is assessed. If the check is performed after the transducer has been calibrated, its capability of maintaining the calibration state, as stated in the calibration diagram, is verified: A “reference” indication, as obtained while calibrating the system, is compared with a suitable location (or position) statistic [19], such as the sample mean value, computed on the sample of the repeated indications. Hence, such long-range stability is related to measurement trueness, i.e., the “closeness of agreement between the average value obtained from a large series of test results and an accepted reference value” (definition from [20], where “test results” are interpreted as “indications”; the VIM3 definition “closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value” is practically unusable since an infinite number of measurements is required). Accordingly, measurement precision and measurement trueness can be interpreted as the complementary components of an encompassing feature of a measuring system, its accuracy, which conveys the basic information required to evaluate the instrumental measurement uncertainty, i.e., the “component of measurement uncertainty arising from a measuring instrument or measuring system in use” (VIM3 definition). VI. I NFLUENCE OF M ODELING ACTIVITIES ON M EASUREMENT U NCERTAINTY: M EASURING R ESEARCH Q UALITY As an application example of the effect of modeling activities on the validity of measurement results, let us consider the measurement of research quality. This concept is somewhat elusive, but it evokes the need of quantifying as much as possible the outcomes of a research process. This is an issue of great social and economic impacts in nations. Deciding MARI et al.: MEASUREMENT FUNDAMENTALS: A PRAGMATIC VIEW about the fundability of research groups or programs or about the ex-post effectiveness of governmental funding policies is steadily becoming a mechanism to show accountability in spending public money. At the same time, while scientists know the value and worth of peers, there is an information asymmetry between the scientific world and the general public that can be addressed by using suitable and possibly welldesigned measurement procedures. Whichever the model, a well-designed research quality measurement process originates from a consensus-based definition of research quality. This has to be reached at least in the context where the decisions based on measurement outcomes have effects and imply agreement between stakeholders. For example, in the research assessment exercise [21], research has a formal definition while the concept of “research quality” is implicitly defined by the indicators chosen as quality determinants. This is similar to what is intended as software quality in the ISO 9126-1 standard [22], i.e., by using a set of suitably chosen indicators, the characteristic is indirectly determined. Moreover, according to what has been suggested earlier, any measurement process must firstly take into account the intended use of the experimentally acquired information. As an example, if research quality is measured for promoting research personnel or for deciding about fundability of research programs, different sets of indicators have to be chosen to properly support the subsequent decision-making activities. For wide-range research quality measurements, where the final customer is the general public and the society at large, the indicator set does not contain too specific attributes but rather generic ones. Among others, a specified requirement in this latter case is often the determination of research impacts within the society. The research environment also plays a role: An environment rich in industrial activities most probably supports scientific and technological research by means of economical resources and creative stimuli. All mature research quality models try to measure also this attribute by introducing indicators related to environment. The measurement procedure designed to achieve the predefined goals results in the definition of a multiattribute model that contains both qualitative, and thus judgmental information, and quantitative indicators. This is the case for the many research quality frameworks and assessment procedures proposed in scientifically advanced countries. In all proposed models, the measurand is multidimensional and often comprises attributes related to outcomes (e.g., publications and patents), impacts (e.g., amount of social and economical benefits), and environment (e.g., types and level of facilities in the research environment). These main attributes include subattributes that are measured by informed experts using an ordinal scale, sometimes treated as a quantized ratio scale, so that weighted summations are calculated at a later stage of data processing. Bibliometric data also come often into play to allow judgments based on indirect sources. Models include the characterization of both enabling factors, such as the amount of available facilities, and results, such as the number of excellent research products, as done for the assessment of organizational quality in the excellence model proposed by the European Foundation for Quality Management [23]. In any case, each indicator is characterized by uncertainty that is often hard or noneconomical to estimate and thus is not available 2113 or not communicated. Therefore, process validation is required to assure that the intended use of the produced measurement results is achieved. Research quality models are often modified, both to reflect changes in views about the importance of single attributes as expressed by their corresponding weights and to reduce the consequences of adaptive behaviors in researchers adopting opportunistic strategies. Does this approach lead to a measurement, as seen from the perspective of the requirements that such a procedure must obey? The representation condition—which requires research of better quality to be measured so that symbols/numbers assigned to it preserve this order—might not be properly satisfied. In fact, although experts are trained so to develop discriminative capabilities in evaluating research quality determinants and definitions of research quality attributes are refined to reduce semantic uncertainties, the representation condition could be not fulfilled. Similar considerations apply also to the concepts of objectivity and intersubjectivity. The level of comprehension in the multidimensional concept of research quality could not assure a proper objectivity, i.e., the evaluation of indicators measuring attributes and subattributes is not guaranteed to capture the entire essence of the measurand and nothing else. As a counterexample, it is well known that confusing factors may influence prejudicially the evaluation by experts, as in the case of the Matthew effect [24], a phenomenon by which the evaluation of research outcomes or proposals is influenced significantly by the reputation of the evaluated subject. This may lead to increased probability of fundability of high-reputation researchers, thus leading to overfunding already rich research subjects. Regarding intersubjectivity, it can be observed that while harmonization sessions may be performed beforehand to guarantee that experts consistently respond in similar ways to similar stimuli, a higher variability in responses can be expected than what is usual in other quantitative disciplines, when different subjects perform the measurement. The concepts of repeatability and reproducibility all have to do with the uncertainty attributed to measurements and with the replication of the experiments under specified conditions. Uncertainty under repeatability conditions may be considered to be low because experts will give equal or similar judgments if asked again over short periods of times. Similar considerations apply to the concept of reproducibility, here to be intended as the precision in measurement obtained when possibly different experts evaluate different subjects performing research with nominally equal quality attributes. The amount of agreement between evaluations will provide information about reproducibility. The difficulty here is associated with the replication of the measurand in different experimental contexts and again with the harmonization of evaluation styles by experts. The concept of calibration is too loose in this context to be applied, also because universal standards or references can hardly be defined in measuring research quality. Although many attributes are applicable only marginally with respect to what happens in hard measurements, measuring research quality using this or similar approaches may provide valid information. This seems to be confirmed indirectly by the fact that some nations are using research evaluation processes 2114 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 61, NO. 8, AUGUST 2012 recurrently and by means of validation sessions performed using correlations based on citation and peer-review analyses [25]. The issue whether such an evaluation process is a measurement cannot be answered in a distinct and clear way, and consensus on this subject is not obtained among stakeholders. In a strict sense, the facts that uncertainty estimates are not available and that the desirable measurement characteristics can hardly be achieved suggest that this procedure is likely a generic evaluation rather than a measurement. However, the needed degree of belief in statements about research quality may be conveyed not only by explicitly stated uncertainty, as suggested in Section III but, when this is missing, also through the reputation of the measurer (e.g., as in the case of national agencies) and by the degree of rigor, transparency, and discipline exhibited by carrying out the evaluation process. [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] VII. C ONCLUSION Discussion about measurement fundamentals is being continuously enriched because of the new challenges offered by advancements in science and technology. Understanding differences in hard and soft measurement characteristics helps in the adoption of the best behaviors when trading off economic and engineering measurement strategies and adds to progress in science by fostering model recognition and consensus among interested parties. The pragmatic approach described in this paper starts from the recognition that soft measurements ask for additional analysis about models and properties to produce meaningful experimental results. We proceeded accordingly by addressing the general definitions of measurement characteristics adopted when taking hard measurements and by analyzing their applicability in the realm of soft measurements. As a by-product of this analysis, an improvement of the body of knowledge about fundamentals of any measurement process has also been achieved. Interesting reasoning opportunities have arisen to find similarities and interpretative viewpoints to be applied to specific soft multiattribute measurements. Still, some key points remain open for discussion: Validation, uncertainty, calibration, and other fundamental measurement concepts hardly apply to soft measurements. Nevertheless, measurements of nonphysical properties are becoming widespread, and decisions are taken on this basis, having economic impact. Our views are that these may return valid information and that rigorous, disciplined, and transparent modeling and operative activities are still the principal mechanisms to guarantee the final user about the reliability of the provided information. R EFERENCES [1] L. Finkelstein, “Widely, strongly and weakly defined measurement,” Measurement, vol. 34, no. 1, pp. 39–48, Jul. 2003. [2] Bureau International des Poids et Mesures, Evolving Needs for Metrology in Trade, Industry and Society and the Role of the BIPM. (Kaarls Report). [Online]. Available: http://www.bipm.org/utils/en/pdf/Kaarls2007.pdf [3] Joint Committee for Guides in Metrology, Evaluation of Measurement Data—Guide to the Expression of Uncertainty in Measurement, 2008. JCGM 100:2008. (GUM, originally published in 1993). [Online]. Available: http://www.bipm.org/en/publications/guides/gum.html [4] Joint Committee for Guides in Metrology, International Vocabulary of Metrology—Basic and General Concepts and Associated Terms, 3rd ed., [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] 2008. JCGM 200:2008. (VIM). [Online]. Available: http://www.bipm.org/ en/publications/guides/vim.html G. B. Rossi, “Measurability,” Measurement, vol. 40, pp. 545–562, 2007. D. Krantz, R. Luce, P. Suppes, and A. Tversky, Foundations of Measurement, vol. 1. New York: Academic, 1971. D. Krantz, R. Luce, P. Suppes, and A. Tversky, Foundations of Measurement, vol. 2. New York: Academic, 1989. D. Krantz, R. Luce, P. Suppes, and A. Tversky, Foundations of Measurement, vol. 3. New York: Academic, 1990. F. Roberts, Measurement Theory: With Applications to Decision Making, Utility, and the Social Sciences. Reading, MA: Addison-Wesley, 1979. L. Mari, “Beyond the representational viewpoint: A new formalization of measurement,” Measurement, vol. 27, no. 2, pp. 71–84, Mar. 2000. L. Mari, “Notes towards a qualitative analysis of information in measurement results,” Measurement, vol. 25, no. 3, pp. 183–192, Apr. 1999. A. Frigerio, A. Giordani, and L. Mari, “Outline of a general model of measurement,” Synthese, vol. 175, no. 2, pp. 123–149, 2010. V. R. Basili, G. Caldiera, and H. D. Rombach, “The goal question metric approach,” in Encyclopedia of Software Engineering, vol. 1, J. J. Marciniak, Ed. New York: Wiley, 1994, pp. 528–532. International Organization for Standardization, Quality Management Systems—Fundamentals and Vocabulary, Geneva, Switzerland, 2005. ISO 9000:2005. S. S. Stevens, “On the theory of scales of measurement,” Science, vol. 103, no. 2684, pp. 677–680, Jun. 1946. K. R. Popper, Knowledge and the Body—Mind Problem—In Defense of Interaction. Evanston, IL: Routledge, 1994. L. Finkelstein, “Problems of measurement in soft systems,” Measurement, vol. 38, no. 4, pp. 267–274, Dec. 2005. L. Mari, V. Lazzarotti, and R. Manzini, “Measurement in soft systems: Epistemological framework and a case study,” Measurement, vol. 42, no. 2, pp. 241–253, Feb. 2009. International Organization for Standardization, Statistics—Vocabulary and Symbols—Part 1: General Statistical Terms and Terms Used in Probability, 2006. ISO 3534-1. International Organization for Standardization, Accuracy (Trueness and Precision) of Measurement Methods and Results—Part 1: General Principles and Definitions, 1998. ISO 5725-1. Assessment Framework and Guidance on Submissions, Jul. 2011. REF2014. [Online]. Available: http://www.hefce.ac.uk/research/ref/pubs/ 2011/02_11/02_11.pdf International Organization for Standardization, Software Engineering— Product Quality—Part 1: Quality Model, 2001. ISO/IEC 9126-1:2001. The EFQM Excellence Model. [Online]. Available: http://www.efqm.org R. K. Merton, “The Matthew effect in science,” Science, vol. 159, no. 3810, pp. 56–63, Jan. 5, 1968. C. Oppenheim and M. A. C. Summers, “Citation counts and the research assessment exercise, Part VI: Unit of assessment 67 (music),” Inf. Res., vol. 13, no. 2, Jun. 2008. [Online]. Available: http://informationr.net/ir/ 13-2/paper342.html Luca Mari received the Laurea degree in physics from the University of Milano, Milan, Italy, in 1987 and the Ph.D. degree in measurement science from the Politecnico di Torino, Turino, Italy, in 1994. Since 2006, he has been a Full Professor of measurement science with Università Carlo Cattaneo (LIUC), Castellanza, Italy, where he teaches courses on measurement science, statistical data analysis, and system theory. He heads the Department of Quantitative Methods, the Laboratory on RFId Systems, and the Ph.D. School at LIUC. He is also currently the Chairman of the Technical Committee 7—Measurement Science—of the International Measurement Confederation (IMEKO) and an International Electrotechnical Commission (IEC) expert in the Working Group 2 (VIM), Joint Committee for Guides in Metrology, Bureau International des Poids et Mesures. He is the author or coauthor of several scientific papers published in international journals and international conference proceedings. His research interests include measurement science and system theory. MARI et al.: MEASUREMENT FUNDAMENTALS: A PRAGMATIC VIEW Paolo Carbone (M’94–SM’09) received the Laurea and Dottorato di ricerca from the University of Padova, Padova, Italy, in 1990 and 1994, respectively. From 1994 to 1997, he was a Researcher with the Third University of Rome, Rome, Italy. From 1997 to 2002, he was a Researcher with the University of Perugia, Perugia, Italy, where he has been a Full Professor since 2002 and has been teaching courses in instrumentation and measurement and in reliability and quality engineering. He has been involved in various research projects, sponsored by private and public funds. His research objective is to develop knowledge, models, and systems for the advance of instrumentation and measurement technology. He is author/coauthor of more than 130 papers and has appeared in international journals and conference proceedings. He is currently the Head of the Department of Electronics and Information Engineering, University of Perugia. Dr. Carbone was the Chairman of the International Measurement Confederation (IMEKO) International Workshop on Analog-to-digital converters Modeling and Testing (2003 and 2011 editions). He served as an Associate Editor of IEEE T RANSACTIONS ON C IRCUITS AND S YSTEMS —PART II from 2000 to 2002 and of IEEE T RANSACTIONS ON C IRCUITS AND S YSTEMS —PART I from 2005 to 2007. 2115 Dario Petri (F’09) received the M.Sc. (summa cum laude) and Ph.D. degrees in electronics engineering from the University of Padova, Padova, Italy, in 1986 and 1990, respectively. From 1990 to 1992, he was an Assistant Professor with the Department of “Electronics and Information Engineering,” University of Padova. In 1992, he joined the University of Perugia, Perugia, Italy, as an Associate Professor and then as a Full Professor of measurement and electronic instrumentation in 1999. Since 2002, he has been with the Department of “Information Engineering and Computer Science,” University of Trento, Trento, Italy, where he was the Chair of International Ph.D. School in “Information and Communication Technology” from 2004 to 2007, was the Chair of information engineering study programs from 2007 to 2010, and is currently the Head of the department. He is an author of over 200 papers published in international journals or in proceedings of peer-reviewed international conferences. Dr. Petri has chaired the Italy Chapter of the IEEE Instrumentation and Measurement (I&M) Society from 2006 to 2010. He is currently the Vice Chair of the IEEE Italy Section. Also, since 2008, he has been a Cofounder and General Chair of the Ph.D. School, “International Measurement University,” IEEE I&M Society. He is an Associate Editor and the Vice President for conferences of the IEEE T RANSACTIONS ON I&M.