Model Evaluation

Class-modeling techniques, classic and recent, are studied with special reference with the new applications to data sets characterized by many variables, frequently noisy variables without importance in the characterization of the studied class. UNEQ (based on the hypothesis of multivariate normal distribution and on the Hotelling T 2 statistics), SIMCA (with a model built on the class principal components), POTFUN (Potential Functions Modeling, where the probability distribution is estimated by means of the potential functions), MRM (Multivariate Range Modeling, where the model is obtained with the range of the original variables and of discriminant functions) are compared by means of the sensitivities and specificities of the models evaluated both by means of cross validation and with the model forced to accept all the objects of the modeled category. The parameters used to evaluate the performance of class-modeling techniques are critically reviewed. The performances of class-modeling techniques, both in classification and in modeling, have been evaluated on real data sets, with the original variables and on subsets of variables obtained after elimination of nondiscriminant variables. The effect of noisy variables and of deviation from the underlying hypotheses are discussed. Therefore, a "perfect" class model has 100% sensitivity and 100% specificity.

A classification of spatial simulation models of fire and vegetation dynamics (landscape fire succession models or LFSMs) is presented. The classification was developed to provide a foundation for comparing models and to help identify the appropriate fire and vegetation processes and their simulation to include in coarse scale dynamic global vegetation models. Other uses include a decision tool for research and management applications and a vehicle to interpret differences between LFSMs. The classification is based on the four primary processes that influence fire and vegetation dynamics: fire ignition, fire spread, fire effects, and vegetation succession. Forty-four LFSMs that explicitly simulated the four processes were rated by the authors and the modelers on a scale from 0 to 10 for their inherent degree of stochasticity, complexity, and mechanism for each of the four processes. These ratings were then used to group LFSMs into similar classes using common ordination and clustering techniques. Another database was created to describe each LFSM using selected keywords for over 20 explanatory categories. This database and the ordination and clustering results were then used to create the final LFSM classification that contains 12 classes and a corresponding key. The database and analysis results were used to construct a second classification key so managers can pick the most appropriate model for their application based on computer resources, available modeling expertise, and management objective. Published by Elsevier B.V.

As a unified discipline, econometrics is still relatively young and has been transforming and expanding very rapidly over the past few decades. Major advances have taken place in the analysis of cross sectional data by means of semi-parametric and non-parametric techniques. Heterogeneity of economic relations across individuals, firms and industries is increasingly acknowledged and attempts have been made to take them into account either by integrating out their effects or by modeling the sources of heterogeneity when suitable panel data exists. The counterfactual considerations that underlie policy analysis and treatment evaluation have been given a more satisfactory foundation. New time series econometric techniques have been developed and employed extensively in the areas of macroeconometrics and finance. Non-linear econometric techniques are used increasingly in the analysis of cross section and time series observations. Applications of Bayesian techniques to econometric problems have been given new impetus largely thanks to advances in computer power and computational techniques. The use of Bayesian techniques have in turn provided the investigators with a unifying framework where the tasks of forecasting, decision making, model evaluation and learning can be considered as parts of the same interactive and iterative process; thus paving the way for establishing the foundation of "real time econometrics". This paper attempts to provide an overview of some of these developments. JEL Classification: C1, C2, C3, C4, C5

The international workshop on Policy Modeling for Industrial Energy Use was jointly organized by EETA (Professional Network for Engineering Economic Technology Analysis) and INEDIS (International Network for Energy Demand Analysis in the Industrial Sector). The workshop has helped to layout the needs and challenges to include policy more explicitly in energy-efficiency modeling. The current state-of-the-art models have a proven track record in forecasting future trends under conditions similar to those faced in the recent past. However, the future of energy policy in a climate-restrained world is likely to demand different and additional services to be provided by energy modelers. In this workshop some of the international models used to make energy consumption forecasts have been discussed as well as innovations to enable the modeling eScholarship provides open access, scholarly publishing services to the University of California and delivers a dynamic research platform to scholars worldwide. of policy scenarios. This was followed by the discussion of future challenges, new insights in the data needed to determine the inputs into energy models, and methods to incorporate decisionmaking and policy in the models. Based on the discussion the workshop participants came to the following conclusions and recommendations: Current energy models are already complex, and it is already difficult to collect the model inputs. Hence, new approaches should be transparent and not lead to extremely complex models that try to "do everything". The model structure will be determined by the questions that need to be answered. A good understanding of the decision making framework of policymakers and clear communication on the needs are essential to make any future energy modeling effort successful. There is a need to better understand the effects of policy on future energy use, emissions and the economy. To allow the inclusion of policy instruments in models, evaluation of programs and instruments is essential, and need to be included in the policy instrument design. Increased efforts are needed to better understand the effects of innovative (non-monetary) policy instruments through evaluation and to develop approaches to model both conventional and innovative policies. The explicit modeling of barriers and decision making in the models seems a promising way to enable modeling of conventional and innovative policies. A modular modeling approach is essential to not only provide transparency, but also to use the available resources most effectively and efficiently. Many large models have been developed in the past, but have been abandoned after only brief periods of use. A development path based on modular building blocks needs the establishment of a flexible but uniform modeling framework. The leadership of international agencies and organizations is essential in the establishment of such a framework. A preference is given for "softlinks" between different modules and models, to increase transparency and reduce complexity. There is a strong need to improve the efficiency of data collection and interpretation efforts to produce reliable model inputs. The workshop participants support the need for the establishment of an (in-)formal exchanges of information, as well as modeling approaches. The development of an informal network of research institutes and universities to help build a common dataset and exchange ideas on specific areas is proposed. Starting with an exchange of students would be a relative low-cost way to start such collaboration. It would be essential to focus on specific topics. It is also essential to maintain means of regular exchange of ideas between researchers in the different focus points.