Abstract
The burgeoning growth of the Internet of Things (IoT) has unleashed a deluge of data from diverse sensors and devices, presenting both opportunities and challenges. Among these challenges is data redundancy in IoT datasets, which hinders efficient storage, processing, and analysis. This article explores the Matrix Profile technique as an innovative approach for addressing data redundancy in IoT applications. The Matrix Profile, comprising Distance Profile and Profile Index components, proves instrumental in identifying duplicate and near-duplicate data points. Leveraging the Stumpy library in Python, this study introduces a novel method that preserves time series integrity while reducing computational costs and optimizing memory usage. The proposed technique not only identifies redundant data but also streamlines its removal, thereby enhancing storage efficiency and reducing network bandwidth consumption. The Matrix Profile is facilitated by a robust distance measure and a sliding window approach, and marks a significant contribution to IoT data management.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Yu, W., Liu, Y., Dillon, T., Rahayu, W., Mostafa, F.: An integrated framework for health state monitoring in a smart factory employing IoT and big data techniques. IEEE Internet Things J. 9(3), 2443–2454 (2022). https://doi.org/10.1109/JIOT.2021.3096637
Fazea, Y., Mohammed, F., Al-Nahari, A.: A review on 5G technology in IoT-application based on light fidelity (Li-Fi) indoor communication. In: Saeed, F., Mohammed, F., Ghaleb, F. (eds.) IRICT 2021. LNDECT, vol. 127, pp. 371–384. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98741-1_31
Bora, A., Saiprasad, N., Venkat, A., Raja, T.A.: IoT And Python Based Smart Agriculture System For Effective Plant Growth, vol. 11, p. 1838 (2022)
Abdulzahra, S.A., Al-Qurabat, A.K.M.: Data aggregation mechanisms in wire-less sensor networks of IoT: a survey. IJCDS 13(1), 1–15 (2023). https://doi.org/10.12785/ijcds/130101
Alabadi, M., Habbal, A., Wei, X.: Industrial internet of things: requirements, architecture, challenges, and future research directions. IEEE Access 10, 66374–66400 (2022). https://doi.org/10.1109/ACCESS.2022.3185049
Benomar, Z., et al.: A fog-based architecture for latency-sensitive monitoring applications in industrial internet of things. IEEE Internet Things J. 10(3), 1908–1918 (2023). https://doi.org/10.1109/JIOT.2021.3138691
Vijayalakshmi, K., Jayalakshmi, V.: Analysis on data deduplication techniques of storage of big data in cloud. In: 2021 5th International Conference on Computing Methodologies and Communication (ICCMC), pp. 976–983 (2021). https://doi.org/10.1109/ICCMC51019.2021.9418445
Muthunagai, S.U., Anitha, R.: CTS-IIoT: computation of time series data during index based de-duplication of industrial IoT (IIoT) data in cloud environment. Wireless Pers. Commun. 129(1), 433–453 (2023). https://doi.org/10.1007/s11277-022-10105-5
Chhabra, N., Bala, M.: An optimized data duplication strategy for cloud computing: dedup with ABE and bloom filters. Int. J. Future Gener. Commun. Networking 13(1), 824–834 (2020)
Zhou, Y., Yu, Z., Gu, L., Feng, D.: An efficient encrypted deduplication scheme with security-enhanced proof of ownership in edge computing. BenchCouncil Trans. Benchmarks Standards Evaluations 2(2), 100062 (2022). https://doi.org/10.1016/j.tbench.2022.100062
Zhou, H., Pal, S., Jadidi, Z., Jolfaei, A.: A fog-based security framework for large-scale industrial internet of things environments. IEEE Internet Things Mag. 6, 1–7 (2022). https://doi.org/10.1109/IOTM.002.2200195
Pioli, L., Dorneles, C.F., de Macedo, D.D.J., Dantas, M.A.R.: An overview of data reduction solutions at the edge of IoT systems: a systematic mapping of the liter-ature. Computing 104(8), 1867–1889 (2022). https://doi.org/10.1007/s00607-022-01073-6
Mansour, E., Shahzad, F., Tekli, J., Chbeir, R.: Data redundancy management for leaf-edges in connected environments. Computing 104(7), 1565–1588 (2022). https://doi.org/10.1007/s00607-021-01051-4
Ismael, W.M., Gao, M., Yemeni, Z.: ESRRA-IoT: edge-based spatial redundancy reduction approach for internet of things. Internet Things 14(6), 100388 (2021). https://doi.org/10.1016/j.iot.2021.100388
Zhu, Y., Yeh, C.-C.M., Zimmerman, Z., Keogh, E.: Matrix profile XVII: indexing the matrix profile to allow arbitrary range queries. In: 2020 IEEE 36th International Conference on Data Engineering (ICDE), pp. 1846–1849 (2020). https://doi.org/10.1109/ICDE48307.2020.00185
Yeh, C.-C.M., et al.: Matrix profile I: all pairs similarity joins for time series: a unifying view that includes motifs, discords and shapelets. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 1317–1322 (2016). https://doi.org/10.1109/ICDM.2016.0179
Zymbler, M., Ivanova, E.: Matrix profile-based approach to industrial sensor data analysis inside RDBMS. Mathematics 9(17), 2146 (2021). https://doi.org/10.3390/math9172146
Li, H., Wu, X., Wan, X., Lin, W.: Time series clustering via matrix profile and community detection. Adv. Eng. Inform. 54, 101771 (2022). https://doi.org/10.1016/j.aei.2022.101771
Shahcheraghi, M., et al.: Matrix profile XXVI: mplots: scaling time series similarity matrices to massive data. In: 2022 IEEE International Conference on Data Mining (ICDM), pp. 1179–1184 (2022). https://doi.org/10.1109/ICDM54844.2022.00151
Alaee, S., Kamgar, K., Keogh, E.: Matrix profile XXII: exact discovery of time series motifs under DTW. In: 2020 IEEE International Conference on Data Mining (ICDM), pp. 900–905 (2020). https://doi.org/10.1109/ICDM50108.2020.00099
Zimmerman, Z., et al.: Matrix profile XVIII: time series mining in the face of fast moving streams using a learned approximate matrix profile. In: 2019 IEEE International Conference on Data Mining (ICDM), pp. 936–945 (2019). https://doi.org/10.1109/ICDM.2019.00104
Cartwright, E., Crane, M., Ruskin, H.J.: Financial time series: market analysis techniques based on matrix profiles †. Eng. Proc. 5(1), 45 (2021). https://doi.org/10.3390/engproc2021005045
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hussein, S.A.A., Ahmad, R.B., Yaakob, N., Mohammed, F. (2024). Matrix Profile Unleashed: A Solution to IoT Data Redundancy Challenges. In: Saeed, F., Mohammed, F., Fazea, Y. (eds) Advances in Intelligent Computing Techniques and Applications. IRICT 2023. Lecture Notes on Data Engineering and Communications Technologies, vol 211. Springer, Cham. https://doi.org/10.1007/978-3-031-59707-7_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-59707-7_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-59706-0
Online ISBN: 978-3-031-59707-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)