1. Introduction
Modern remote sensing, while providing radiance data, gradually tends to provide end users with a series of high-level standard data products [
1]. With the promotion of openness, sharing, interconnection and other services [
2], multi-source and multi-temporal quantitative remote-sensing products provide better data support for resources and environmental monitoring, global change and sustainable development [
3]. The quality of remote-sensing products is the key to restrict their application ability [
4]. The importance of accurately evaluating remote-sensing products has been generally recognized [
3,
4,
5,
6,
7,
8]. According to the definition of the Working Group on Calibration and Validation (WGCV) of the International Committee on Earth Observation Satellites (CEOS), validation refers to the process of independently evaluating the accuracy and uncertainty by the comparative analysis of remote-sensing products and reference data (relative truth values) that can represent ground targets [
5]. Therefore, validation is an important method used to solve the quality problems of remote-sensing products.
Obtaining land-surface parameter by means of in situ information from field observations is one of the key steps of validation [
1,
9]. The field observations are primarily based on deploying samples or constructing validation stations [
10]. Deploying sample points on site is more flexible with a larger observation area, which can facilitate the comprehensive application of multiple types of remote-sensing products. This method requires various measuring instruments and a large number of surveyors [
11]. Due to the limitations of time and cost, it is impossible to obtain long-term data. With the development of communication technology and the increasing demand for time-series data, long-term stable validation stations have emerged, and some newly advanced observation methods have developed, such as wireless sensor networks (WSN) [
12] and footprint observation [
13,
14,
15,
16]. Validation stations provide multi-spatial and multi-temporal data for the optimization of remote-sensing inversion product models and the research of validation algorithms, which have an important promotional significance for the development of validation [
17].
As early as 1999, National Aeronautics and Space Administration (NASA) constructed validation stations and flux towers to undertake ground validation of land cover (LC), leaf area indicator (LAI), photosynthetically active radiation (PAR), net primary productivity (NPP) and other products on the BigFoot Project [
18]. Subsequently, at the beginning of the 21st century, European Space Agency (ESA) also launched the Validation of European Remote Sensing Instruments (VALERI) project to validate land remote-sensing products made by Moderate-resolution Imaging Spectroradiometer (MODIS), VEGETATION, Medium Resolution Imaging Spectrometer (MERIS) and Advanced Very High Resolution Radiometer (AVHRR) sensors [
19]. From 2007 to 2015, China carried out the Heihe Watershed Allied Telemetry Experimental Research (WATER/HiWATER) in the Heihe River Basin for “atmosphere-hydrology-ecology” [
20,
21,
22,
23,
24] and constructed ChinaFLUX. Since 2015, with the implementation of the National Civil Space Infrastructure Medium and Long-term Development plan, Chinese Academy of Sciences (CAS) has constructed the first batch of 48 validation stations for different underlying surfaces to provide support for the establishment of a long-term, stable validation observation network. For the marine environment, Chinese State Oceanic Administration constructed the Hangzhou global ocean system field scientific observation and research station—Argo. In terms of international cooperation, the “Belt and Road Initiatives” is a scientific and technological innovation plan that promotes cooperation via the building capacity of BRICS countries; therefore, the trend of globalized joint validation is increasing, thereby providing a guarantee of quantitative, accurate and scientific information for multilateral sustainable development. The joint deployment of stations is the main method by which to obtain long-term stability data on the ground [
25], and it is also a key step to promote the joint validation of various countries. The validation stations have great application prospects and demands, which pose a challenge to current station-selection methods.
Location selection has been identified in many research fields as a planning problem [
26,
27,
28,
29,
30,
31], which is a multi-criteria decision-making problem that evaluates and selects schemes under the influence of multiple factors and criteria [
32,
33,
34,
35,
36,
37]. Whether it is a multi-criteria decision-making method of location selection or location search, decision makers, managers, and different stakeholders evaluate the schemes based on prior knowledge and interest preferences [
32]. Similarly, the location selection of validation stations involves many factors such as surface characteristics, atmospheric conditions, and the social environment. Long-term experiments with a reasonable cost-effectiveness ratio are required. Currently, the popular location-selection method is to designate candidate locations based on prior knowledge such as remote-sensing images and GIS data, then compare and “select” the optimal from several specific and precise candidate locations, such as the Analytic Hierarchy Process (AHP) [
38,
39,
40,
41,
42,
43,
44,
45], Gray Relational Analysis (GRA) [
46,
47,
48] and Fuzzy Comprehensive Evaluation (FCE) [
49,
50]. During the actual work of location selection for a validation station, there is usually no prior knowledge provided by experts to preselect candidate areas, which first requires a “search” for candidate areas from a large space, based on demand. In the process of building the scoring model for the candidate area, when the experts’ understanding and judgment are inconsistent or even contradictory, the results will be biased. Repeated field investigations and expert demonstrations make the station selection process very cumbersome. The evaluation process is affected by the prior knowledge of experts, which makes it difficult to reuse and promote the model. Therefore, it is necessary to establish the location-selection requirements, standards, and the principles of validation stations [
19,
25,
51]. With the rapid development of Machine Learning, it has been widely applied in many fields of remote sensing [
52,
53,
54,
55,
56,
57], but few studies have employed Machine Learning regression for the location selection of a land-product validation station. Machine learning is driven by existing data and a decision on the location selection performance for a validation station is provided by building a relatively objective model. The method of Machine Learning attempts to explain the inherent relationship between the location and multi-element evaluation indicators, and to simulate the knowledge-system model framework, which is a data-driven scientific-validation station quantitative evaluation system.
Overall, in accordance with the requirements of CEOS on the location selection of calibration and validation, this research puts forward a data-driven selection of a land-product validation station (DSS-LPV) based on Machine Learning. To achieve this, we (1) constructed an evaluation indicator system for location selection of validation station; (2) evaluated spatially based on multi-scale grid; and (3) constructed a data-driven scoring model based on Machine Learning.
2. Data
2.1. Evaluation Indicator
To support the proposed principles and requirements for the construction of a validation station, we selected the important available indicators from an evaluation indicator system via the example of the establishment of a Grassland Station, Forest Station and Agricultural Station in China. These three categories of station are land-surface vegetation stations and have similar requirements for surface features, atmospheric conditions and the social environment, so the same evaluation indicators are selected. The source and introduction of the evaluation indicator data is shown in
Table 1.
2.2. Machine Learning Dataset
The reliability of the training set affects the accuracy of the model. Under the problem of location selection for a validation station, it is necessary to find reliably constructed stations as a training sample. For instance, China has constructed a number of national-level field observation stations to better research the ecological environment and monitor the ecosystem. These stations are distributed in areas with typical ecological regions, climate types and surface cover. Moreover, a close cooperative relationship has been established with the global ecological research and monitoring network, which is an important part of the Global Environmental Monitoring System (GEMS). National field-observation stations provide comprehensive observation data for many research fields, including the validation of remote-sensing products. The location of the network has been repeatedly scrutinized on-site by many scientific researchers. After a lengthy period of inspection, a large number of existing research results have proved that the location is relatively reasonable and has a certain degree of universality. Therefore, national field-observation stations are undoubtedly the best choice as training samples for station selection s exampled by its establishment in China.
This research selects three categories of national-level field observation stations, including 16 Agricultural Stations, 9 Forest Stations, and 6 Grassland Stations, totaling 31 as is shown in
Figure 1. The blue points represent training stations, and the red points represent test stations. The location, category and the corresponding evaluation indicator parameters of each station constitute the Machine Learning dataset.
2.3. Data Preprocessing
The objective of preprocessing is to transform the evaluation indicator data of different types and formats into quantitatively calculated parameters. All data are converted to the unified geographic coordinate system and reference datum, and data divided into the same layer of the grid are resampled to the same resolution. The aim of quantification is to process basic data into parameters that can support evaluation through reasonable spatial analysis (e.g., buffer analysis, calculation of Euclidean distance) and data management (e.g., data normalization, piecewise assignment). However, basic data have different attributes and dimensions. According to the attribute characteristics, the indicators can be classified into target, spatial, temporal, and binary. The target indicators are determined by the goal and type of station. For example, suppose a forest station is designed to be constructed in Sichuan China, the target indicators’ designated values are “Sichuan China” and “Forest”. The spatial indicators represent spatial information of the station, such as “traffic accessibility”, “surface uniformity” and “average aerosol depth”. The temporal indicators represent the time information of station, such as “annual average number of sunny days” and “observing instrument running time”. The binary indicators dictate whether the station is feasible. There are only yes or no cases, so there are only two values of 0 and 1, such as “high-prone natural disaster areas”, “natural protection areas”.
Classification based on attributes contributes to quantifying the indicators better. For evaluation indicators that can be clearly judged via impact on the validation station, their attribute value can be directly substituted into the model, such as cloud cover, precipitation and road distance. Conversely, for the other evaluation indicators that cannot be clearly judged, indirect assignment methods are required. For example, the validation stations cannot be constructed around cities. However, if the distance from the urban area is too far, transportation may be inconvenient and the materials may be scarce. For such evaluation indicators, it is necessary to set up a buffer zone around the urban area. In the buffer, the area is ignored. In parallel to, and outside the buffer, the closer to the urban, the higher the value that is assigned. The data preprocessing-method for the evaluation indicators is shown in
Table 2.
5. Discussion
In this article, we proposed a multi-scale grid spatial evaluation and data-driven location-selection technique for the construction of validation stations. The main method was to divide the indicators that affect the location of the validation station into surface characteristics, atmospheric conditions and social environment and construct an evaluation-indicator system. We divided the three adaptive multi-scale grids and allocated the calculation indicators for each layer. Through Machine Learning of the indicator parameters of the stations that were built, a data-driven scoring model was established to characterize the internal relationship between the evaluation indicators and the location of the station.
The unique advantages of DSS-LPV have been verified by various analyses. Firstly, different from the traditional methods, DSS-LPV is an efficient and systematic method. It only needs to input the learning stations and evaluation indicators to automatically calculate the pixel score. Therefore, it saves the time spent in repeated expert argumentation and reduces the subjectivity introduced by expert knowledge. Secondly, DSS-LPV does not omit the traditional methods. When no candidate area is specified, DSS-LPV can search the candidate area according to the input conditions, and then combine the traditional method to select the optimal location. Therefore, DSS-LPV can provide support for traditional methods. Briefly, DSS-LPV explored more possibilities in the field of station selection and found a quantitative systematic evaluation method that can be effectively combined with traditional expert evaluation.
Additionally, DSS-LPV also has some application extensions. DSS-LPV is not only suitable for the station selection of a single observation element, but also for a comprehensive observation station or comprehensive experiment. When the observation task includes more than one type of surface object (such as vegetation, water body, and atmosphere), the station-selection results of a single observation element obtained by this method can be integrated into a comprehensive validation station to observe multiple features. According to different phenology, weather and land cover, it can also suggest a certain time period suitable for field observation. In particular, the established evaluation-indicator library can be dynamically adjusted and expanded according to actual requirements. For example, the article is mainly for optical satellite products, so atmospheric conditions are considered. For other types of targets, such as SAR satellite or luminous satellite products, there is no need to consider too many weather factors. For more data services and decision support, DSS-LPV can evaluate the rationality of the station location, use the correlation of model parameters to analyze the reasons for the unreasonable location, and provide decision-making support for project development and pre-evaluation.
Moreover, there are some limitations in this research. For example, the size of dataset limits the Machine Learning model. The reliability of the training dataset determines the reliability of the model. Therefore, taking China as an example, national-level stations were selected as the training samples. The limited number of national-level stations limits the construction of the model. With the continuous construction of stations of various types and industries, the training dataset will continue to increase, so the Machine Learning model can also be iteratively expanded. Additionally, since the location selection for validation is a rather complicated issue, the selection of evaluation indicators inevitably has certain limitations. To support the location-selection principle established in this study, the selection of evaluation indicators is based on papers with an in-depth literature review. Although there exist certain limitations, the focus of the study is to seek a quantitative method by which to explore, explain and deduce the location-selection process. In addition, theoretically, the applicability of the model established by Machine Learning depends on which country’s stations are studied. Then, in cases where countries have no or few validation stations, more international cooperation in science and technology is needed to provide support in finding a solution to the problem. Furthermore, whether the model of other countries can be applicable is an issue that we need to further research.