Understanding Integrity of Time Series IoT Datasets through Local Outlier Detection with Steep Peak and Valley

With substantial advances in emerging and enabling technologies in IoT sensors, a vast amount of IoT-based environmental data allows preparation for adverse impacts by providing helpful information for predictive and precise services. However, data acquired by IoT sensors can be corrupted by external environmental factors, which can negatively affect the integrity of data interpretation. To address this problem, a prior study proposed outlier detection techniques using transform-based sparse profiles. However, it would lose its worth without an evaluation methodology for data integrity after probing datasets by outlier detection. In addition, it did not consider data with steep peaks or data that is dependent on other data, which is common in real-world scenarios such as soil moisture data used in this paper. Therefore, we propose a process of preprocessing defective soil moisture sensor data using local pattern-based outlier detection (LPOD) and evaluating the integrity of data after outlier detection. Our paper specifically aims to: 1) detect outliers of original soil IoT datasets to eliminate fault data possibly giving wrong decisions using local and global outlier detection (OD); 2) exploit the results of statistical evaluation to determine whether the outliers have been well eliminated; and 3) find the ground truth pattern of soil IoT datasets considering precipitation. Experiments using real-world soil moisture datasets show that the LPOD method outperforms other statistical outlier detection methods, suggesting that the preprocessed data can improve the integrity of IoT datasets.