Hierarchical multi-scale network for cross-scale visual defect detection
Pages 1141 - 1157
Abstract
Nowadays, an increasing number of researchers apply deep-learning-based object detection methods to implement visual defect detection in industrial manufacturing. However, large-scale variation in visual defect detection impedes the improvement of detection accuracy to be further explored. Therefore, we propose a hierarchical multi-scale block (HMS-Block), equipped with hierarchical representation and multi-scale embedding, to afford scale-abundant features to facilitate multi-scale defect detection. Specially, the hierarchical representation is implemented by a cascade learning stage to extract features from local to global at the channel level. Based on this representation, a cross-branch shortcut is concisely embedded to relieve the large-scale variation problem. Ultimately, the hierarchical multi-scale network (HMSNet) is published elegantly via stacking a certain amount of HMS-Blocks. The proposed methods facilitate the defect detection at all scales and outperform the ResNet50 baseline by a large margin with minor time overhead and less parameter required, indicating that the proposed HMS-Block has a high practical utility in the field of industrial applications. Moreover, the proposed HMSNet can also be applied to other detection-based tasks and greatly surpasses existing methods. Concretely, the proposed HMSNets achieve 42.4/42.7 mAP on NEU and COCO datasets, surpassing the recent backbones (i.e., HRNetV2) by 2.6/1.2 mAP.
References
[1]
Bao Y, Song K, Liu J, Wang Y, Yan Y, Yu H, and Li X Triplet-graph reasoning network for few-shot metal generic surface defect segmentation IEEE Transactions on Instrumentation and Measurement 2021 70 1-11
[2]
Cai, Z., & Vasconcelos, N. (2018). Cascade r-cnn: Delving into high quality object detection, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6154–6162.
[3]
Cao, Y., Xu, J., Lin, S., Wei, F., & Hu, H. (2019). GCNet: Non-Local Networks Meet Squeeze-Excitation Networks and Beyond, in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), IEEE Computer Society, pp. 1971–1980. IEEE Computer Society.
[4]
Çelik A, Küçükmanisa A, Sümer A, Çelebi AT, and Urhan O A real-time defective pixel detection system for lcds using deep learning based object detectors Journal of Intelligent Manufacturing 2022 33 985-994
[5]
Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., & Xu, J. et al. (2019). Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 1–13.
[6]
Cheng KC-C, Chen LL-Y, Li J-W, Li KS-M, Tsai NC-Y, Wang S-J, Huang AY-A, Chou L, Lee C-S, Chen JE, et al. Machine learning-based detection method for wafer test induced defects IEEE Transactions on Semiconductor Manufacturing 2021 34 2 161-167
[7]
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database, in 2009 IEEE conference on computer vision and pattern recognition, Ieee, pp. 248–255. IEEE.
[8]
Everingham M, Van Gool L, Williams CK, Winn J, and Zisserman A The pascal visual object classes (voc) challenge International Journal of Computer Vision 2010 88 2 303-338
[9]
Gao S-H, Cheng M-M, Zhao K, Zhang X-Y, Yang M-H, and Torr P Res2net: A new multi-scale backbone architecture IEEE Transactions on Pattern Analysis and Machine Intelligence 2019 43 2 652-662
[10]
Gao Y, Lin J, Xie J, and Ning Z A real-time defect detection method for digital signal processing of industrial inspection applications IEEE Transactions on Industrial Informatics 2020 17 5 3450-3459
[11]
Guo, C., Fan, B., Zhang, Q., Xiang, S., & Pan, C. (2020). Augfpn: Improving multi-scale feature learning for object detection, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12595–12604.
[12]
Hao R, Lu B, Cheng Y, Li X, and Huang B A steel surface defect inspection approach towards smart industrial monitoring Journal of Intelligent Manufacturing 2021 32 1833-1843
[13]
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
[14]
He Y, Song K, Meng Q, and Yan Y An end-to-end steel surface defect detection approach via fusing multiple hierarchical features IEEE Transactions on Instrumentation and Measurement 2019 69 4 1493-1504
[15]
Hsu C-Y and Liu W-C Multiple time-series convolutional neural network for fault detection and diagnosis and empirical study in semiconductor manufacturing Journal of Intelligent Manufacturing 2021 32 823-836
[16]
Hu J, Shen L, Albanie S, Sun G, and Wu E Squeeze-and-excitation networks IEEE Transactions on Pattern Analysis and Machine Intelligence 2019 42 8 2011-2023
[17]
Jain S, Seth G, Paruthi A, Soni U, and Kumar G Synthetic data augmentation for surface defect detection and classification using deep learning Journal of Intelligent Manufacturing 2022 33 1007-1020
[18]
Kim Y, Cho D, and Lee J-H Wafer defect pattern classification with detecting out-of-distribution Microelectronics Reliability 2021 122
[19]
Kong, T., Yao, A., Chen, Y., & Sun, F. (2016). Hypernet: Towards accurate region proposal generation and joint object detection, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 845–853.
[20]
Li D, Li Y, Xie Q, Wu Y, Yu Z, and Wang J Tiny defect detection in high-resolution aero-engine blade images via a coarse-to-fine framework IEEE Transactions on Instrumentation and Measurement 2021 70 1-12
[21]
Li F and Xi Q Defectnet: Toward fast and effective defect detection IEEE Transactions on Instrumentation and Measurement 2021 70 1-9
[22]
Li, Y., Chen, Y., Wang, N., & Zhang, Z. (2019). Scale-aware trident networks for object detection, in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6054–6063.
[23]
Lin, T-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C.L. (2014). Microsoft coco: Common objects in context, in European conference on computer vision, Springer, 2014, pp. 740–755. Springer.
[24]
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017a) Feature pyramid networks for object detection, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017a, pp. 2117–2125.
[25]
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017b). Focal loss for dense object detection, in Proceedings of the IEEE international conference on computer vision, pp. 2980–2988.
[26]
Ling Z, Zhang A, Ma D, Shi Y, and Wen H Deep Siamese semantic segmentation network for pcb welding defect detection IEEE Transactions on Instrumentation and Measurement 2022 71 1-11
[27]
Liu, J.-J., Hou, Q., Cheng, M.-M., Wang, C., & Feng, J. (2020a). Improving convolutional networks with self-calibrated convolutions, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10096–10105.
[28]
Liu R, Sun Z, Wang A, Yang K, Wang Y, and Sun Q Real-time defect detection network for polarizer based on deep learning Journal of Intelligent Manufacturing 2020 31 1813-1823
[29]
Liu, S., Qi, L., Qin, H., Shi, J., & Jia, J. (2018). Path aggregation network for instance segmentation, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8759–8768.
[30]
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A.C. (2016). Ssd: Single shot multibox detector, in European conference on computer vision, Springer, pp. 21–37. Springer.
[31]
Liu Z, Yang B, Duan G, and Tan J Visual defect inspection of metal part surface via deformable convolution and concatenate feature pyramid neural networks IEEE Transactions on Instrumentation and Measurement 2020 69 12 9681-9694
[32]
Liu Z, Tang R, Duan G, and Tan J Truingdet: Towards high-quality visual automatic defect inspection for mental surface Optics and Lasers in Engineering 2021 138
[33]
Liu, Z., Song, Y., Tang, R., Duan, G., & Tan, J. (2022). Few-shot defect recognition of metal surfaces via attention-embedding and self-supervised learning. Journal of Intelligent Manufacturing, 1–15.
[34]
Meng S, Pan R, Gao W, Zhou J, Wang J, and He W A multi-task and multi-scale convolutional neural network for automatic recognition of woven fabric pattern Journal of Intelligent Manufacturing 2021 32 1147-1161
[35]
Pang, J., Chen, K., Shi, J., Feng, H., Ouyang, W., & Lin, D. (2019) Libra r-cnn: Towards balanced learning for object detection, in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 821–830.
[36]
Radosavovic, I., Kosaraju, R.P., Girshick, R., He, K., & Dollár, P. (2020). Designing network design spaces, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10428–10436.
[37]
Ren S, He K, Girshick R, and Sun J Faster r-cnn: Towards real-time object detection with region proposal networks IEEE Transactions on Pattern Analysis & Machine Intelligence 2017 39 06 1137-1149
[38]
Schlosser T, Friedrich M, Beuth F, and Kowerko D Improving automated visual fault inspection for semiconductor manufacturing using a hybrid multistage system of deep neural networks Journal of Intelligent Manufacturing 2022 33 4 1099-1123
[39]
Singh, B., & Davis, L.S. (2018) An analysis of scale invariance in object detection snip, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3578–3587.
[40]
Singh B, Najibi M, and Davis LS Sniper: Efficient multi-scale training Advances in Neural Information Processing Systems 2018 31 1-10
[41]
Song Y, Liu Z, Wang J, Tang R, Duan G, and Tan J Multiscale adversarial and weighted gradient domain adaptive network for data scarcity surface defect detection IEEE Transactions on Instrumentation and Measurement 2021 70 1-10
[42]
Stern ML and Schellenberger M Fully convolutional networks for chip-wise defect detection employing photoluminescence images: Efficient quality control in led manufacturing Journal of Intelligent Manufacturing 2021 32 113-126
[43]
Sun, K., Xiao, B., Liu, D., & Wang, J. (2019). Deep high-resolution representation learning for human pose estimation, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703.
[44]
Szegedy, C., Liu, W., Jia,Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9.
[45]
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826.
[46]
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017). Inception-v4, inception-resnet and the impact of residual connections on learning, in Proceedings of the AAAI conference on artificial intelligence, vol. 31, pp. 1–8.
[47]
Tang, R., Liu, Z., Li, Y., Song, Y., Liu,H., Wang, Q., Shao, J., Duan, G., & Tan, J. (2023). Task-balanced distillation for object detection. Pattern Recognition, 109320.
[48]
Tian, Z., Shen, C., Chen, H., & He, T. (2019). Fcos: Fully convolutional one-stage object detection, in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9627–9636.
[49]
Wang J, Sun K, Cheng T, Jiang B, Deng C, Zhao Y, Liu D, Mu Y, Tan M, Wang X, et al. Deep high-resolution representation learning for visual recognition IEEE Transactions on Pattern Analysis and Machine Intelligence 2020 43 10 3349-3364
[50]
Wang Y, Liu M, Zheng P, Yang H, and Zou J A smart surface inspection system using faster r-cnn in cloud-edge computing environment Advanced Engineering Informatics 2020 43
[51]
Woo, S., Park, J., Lee, J.-Y., & Kweon, I.S. (2018). Cbam: Convolutional block attention module, in Proceedings of the European conference on computer vision (ECCV), pp. 3–19.
[52]
Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492–1500.
[53]
Yang B, Liu Z, Duan G, and Tan J Mask2defect: A prior knowledge-based data augmentation method for metal surface defect inspection IEEE Transactions on Industrial Informatics 2021 18 10 6743-6755
[54]
Zhang, S., Chi, C., Yao, Y., Lei, Z., & Li, S.Z. (2020). Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9759–9768.
Recommendations
Multi-Scale Cascade Network for Salient Object Detection
MM '17: Proceedings of the 25th ACM international conference on MultimediaIn this paper we present a novel network architecture, called Multi-Scale Cascade Network (MSC-Net), to identify the most visually conspicuous objects in an image. Our network consists of several stages (sub-networks) for handling saliency detection ...
Normalizing Flow Based Defect Detection with Motion Detection
Smart MultimediaAbstractVisual defect detection is critical to ensure the quality of most products. However, the majority of small and medium-sized manufacturing enterprises still rely on tedious and error-prone human manual inspection. The main reasons include: 1) the ...
Multi-task Compositional Network for Visual Relationship Detection
AbstractPrevious methods treat visual relationship detection as a combination of object detection and predicate detection. However, natural images likely contain hundreds of objects and thousands of object pairs. Relying only on object detection and ...
Comments
Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 10 March 2023
Accepted: 13 February 2023
Received: 03 August 2021
Author Tags
Qualifiers
- Research-article
Funding Sources
- national key research and development program of china
- national natural science foundation of china
- high-level talent special support plan of zhejiang province
- natural science foundation of zhejiang province
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 06 Jan 2025