research-article

Hierarchical multi-scale network for cross-scale visual defect detection

Authors:

Jianrong TanAuthors Info & Claims

Journal of Intelligent Manufacturing, Volume 35, Issue 3

Pages 1141 - 1157

https://doi.org/10.1007/s10845-023-02097-1

Published: 10 March 2023 Publication History

Abstract

Nowadays, an increasing number of researchers apply deep-learning-based object detection methods to implement visual defect detection in industrial manufacturing. However, large-scale variation in visual defect detection impedes the improvement of detection accuracy to be further explored. Therefore, we propose a hierarchical multi-scale block (HMS-Block), equipped with hierarchical representation and multi-scale embedding, to afford scale-abundant features to facilitate multi-scale defect detection. Specially, the hierarchical representation is implemented by a cascade learning stage to extract features from local to global at the channel level. Based on this representation, a cross-branch shortcut is concisely embedded to relieve the large-scale variation problem. Ultimately, the hierarchical multi-scale network (HMSNet) is published elegantly via stacking a certain amount of HMS-Blocks. The proposed methods facilitate the defect detection at all scales and outperform the ResNet50 baseline by a large margin with minor time overhead and less parameter required, indicating that the proposed HMS-Block has a high practical utility in the field of industrial applications. Moreover, the proposed HMSNet can also be applied to other detection-based tasks and greatly surpasses existing methods. Concretely, the proposed HMSNets achieve 42.4/42.7 mAP on NEU and COCO datasets, surpassing the recent backbones (i.e., HRNetV2) by 2.6/1.2 mAP.

References

[1]

Bao Y, Song K, Liu J, Wang Y, Yan Y, Yu H, and Li X Triplet-graph reasoning network for few-shot metal generic surface defect segmentation IEEE Transactions on Instrumentation and Measurement 2021 70 1-11

[2]

Cai, Z., & Vasconcelos, N. (2018). Cascade r-cnn: Delving into high quality object detection, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6154–6162.

[3]

Cao, Y., Xu, J., Lin, S., Wei, F., & Hu, H. (2019). GCNet: Non-Local Networks Meet Squeeze-Excitation Networks and Beyond, in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), IEEE Computer Society, pp. 1971–1980. IEEE Computer Society.

[4]

Çelik A, Küçükmanisa A, Sümer A, Çelebi AT, and Urhan O A real-time defective pixel detection system for lcds using deep learning based object detectors Journal of Intelligent Manufacturing 2022 33 985-994

Digital Library

[5]

Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., & Xu, J. et al. (2019). Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 1–13.

[6]

Cheng KC-C, Chen LL-Y, Li J-W, Li KS-M, Tsai NC-Y, Wang S-J, Huang AY-A, Chou L, Lee C-S, Chen JE, et al. Machine learning-based detection method for wafer test induced defects IEEE Transactions on Semiconductor Manufacturing 2021 34 2 161-167

[7]

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database, in 2009 IEEE conference on computer vision and pattern recognition, Ieee, pp. 248–255. IEEE.

[8]

Everingham M, Van Gool L, Williams CK, Winn J, and Zisserman A The pascal visual object classes (voc) challenge International Journal of Computer Vision 2010 88 2 303-338

Digital Library

[9]

Gao S-H, Cheng M-M, Zhao K, Zhang X-Y, Yang M-H, and Torr P Res2net: A new multi-scale backbone architecture IEEE Transactions on Pattern Analysis and Machine Intelligence 2019 43 2 652-662

Digital Library

[10]

Gao Y, Lin J, Xie J, and Ning Z A real-time defect detection method for digital signal processing of industrial inspection applications IEEE Transactions on Industrial Informatics 2020 17 5 3450-3459

[11]

Guo, C., Fan, B., Zhang, Q., Xiang, S., & Pan, C. (2020). Augfpn: Improving multi-scale feature learning for object detection, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12595–12604.

[12]

Hao R, Lu B, Cheng Y, Li X, and Huang B A steel surface defect inspection approach towards smart industrial monitoring Journal of Intelligent Manufacturing 2021 32 1833-1843

Digital Library

[13]

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.

[14]

He Y, Song K, Meng Q, and Yan Y An end-to-end steel surface defect detection approach via fusing multiple hierarchical features IEEE Transactions on Instrumentation and Measurement 2019 69 4 1493-1504

[15]

Hsu C-Y and Liu W-C Multiple time-series convolutional neural network for fault detection and diagnosis and empirical study in semiconductor manufacturing Journal of Intelligent Manufacturing 2021 32 823-836

Digital Library

[16]

Hu J, Shen L, Albanie S, Sun G, and Wu E Squeeze-and-excitation networks IEEE Transactions on Pattern Analysis and Machine Intelligence 2019 42 8 2011-2023

Digital Library

[17]

Jain S, Seth G, Paruthi A, Soni U, and Kumar G Synthetic data augmentation for surface defect detection and classification using deep learning Journal of Intelligent Manufacturing 2022 33 1007-1020

Digital Library

[18]

Kim Y, Cho D, and Lee J-H Wafer defect pattern classification with detecting out-of-distribution Microelectronics Reliability 2021 122

[19]

Kong, T., Yao, A., Chen, Y., & Sun, F. (2016). Hypernet: Towards accurate region proposal generation and joint object detection, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 845–853.

[20]

Li D, Li Y, Xie Q, Wu Y, Yu Z, and Wang J Tiny defect detection in high-resolution aero-engine blade images via a coarse-to-fine framework IEEE Transactions on Instrumentation and Measurement 2021 70 1-12

[21]

Li F and Xi Q Defectnet: Toward fast and effective defect detection IEEE Transactions on Instrumentation and Measurement 2021 70 1-9

[22]

Li, Y., Chen, Y., Wang, N., & Zhang, Z. (2019). Scale-aware trident networks for object detection, in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6054–6063.

[23]

Lin, T-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C.L. (2014). Microsoft coco: Common objects in context, in European conference on computer vision, Springer, 2014, pp. 740–755. Springer.

[24]

Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017a) Feature pyramid networks for object detection, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017a, pp. 2117–2125.

[25]

Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017b). Focal loss for dense object detection, in Proceedings of the IEEE international conference on computer vision, pp. 2980–2988.

[26]

Ling Z, Zhang A, Ma D, Shi Y, and Wen H Deep Siamese semantic segmentation network for pcb welding defect detection IEEE Transactions on Instrumentation and Measurement 2022 71 1-11

[27]

Liu, J.-J., Hou, Q., Cheng, M.-M., Wang, C., & Feng, J. (2020a). Improving convolutional networks with self-calibrated convolutions, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10096–10105.

[28]

Liu R, Sun Z, Wang A, Yang K, Wang Y, and Sun Q Real-time defect detection network for polarizer based on deep learning Journal of Intelligent Manufacturing 2020 31 1813-1823

Digital Library

[29]

Liu, S., Qi, L., Qin, H., Shi, J., & Jia, J. (2018). Path aggregation network for instance segmentation, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8759–8768.

[30]

Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A.C. (2016). Ssd: Single shot multibox detector, in European conference on computer vision, Springer, pp. 21–37. Springer.

[31]

Liu Z, Yang B, Duan G, and Tan J Visual defect inspection of metal part surface via deformable convolution and concatenate feature pyramid neural networks IEEE Transactions on Instrumentation and Measurement 2020 69 12 9681-9694

[32]

Liu Z, Tang R, Duan G, and Tan J Truingdet: Towards high-quality visual automatic defect inspection for mental surface Optics and Lasers in Engineering 2021 138

[33]

Liu, Z., Song, Y., Tang, R., Duan, G., & Tan, J. (2022). Few-shot defect recognition of metal surfaces via attention-embedding and self-supervised learning. Journal of Intelligent Manufacturing, 1–15.

[34]

Meng S, Pan R, Gao W, Zhou J, Wang J, and He W A multi-task and multi-scale convolutional neural network for automatic recognition of woven fabric pattern Journal of Intelligent Manufacturing 2021 32 1147-1161

Digital Library

[35]

Pang, J., Chen, K., Shi, J., Feng, H., Ouyang, W., & Lin, D. (2019) Libra r-cnn: Towards balanced learning for object detection, in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 821–830.

[36]

Radosavovic, I., Kosaraju, R.P., Girshick, R., He, K., & Dollár, P. (2020). Designing network design spaces, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10428–10436.

[37]

Ren S, He K, Girshick R, and Sun J Faster r-cnn: Towards real-time object detection with region proposal networks IEEE Transactions on Pattern Analysis & Machine Intelligence 2017 39 06 1137-1149

Digital Library

[38]

Schlosser T, Friedrich M, Beuth F, and Kowerko D Improving automated visual fault inspection for semiconductor manufacturing using a hybrid multistage system of deep neural networks Journal of Intelligent Manufacturing 2022 33 4 1099-1123

Digital Library

[39]

Singh, B., & Davis, L.S. (2018) An analysis of scale invariance in object detection snip, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3578–3587.

[40]

Singh B, Najibi M, and Davis LS Sniper: Efficient multi-scale training Advances in Neural Information Processing Systems 2018 31 1-10

[41]

Song Y, Liu Z, Wang J, Tang R, Duan G, and Tan J Multiscale adversarial and weighted gradient domain adaptive network for data scarcity surface defect detection IEEE Transactions on Instrumentation and Measurement 2021 70 1-10

[42]

Stern ML and Schellenberger M Fully convolutional networks for chip-wise defect detection employing photoluminescence images: Efficient quality control in led manufacturing Journal of Intelligent Manufacturing 2021 32 113-126

Digital Library

[43]

Sun, K., Xiao, B., Liu, D., & Wang, J. (2019). Deep high-resolution representation learning for human pose estimation, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703.

[44]

Szegedy, C., Liu, W., Jia,Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9.

[45]

Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826.

[46]

Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017). Inception-v4, inception-resnet and the impact of residual connections on learning, in Proceedings of the AAAI conference on artificial intelligence, vol. 31, pp. 1–8.

[47]

Tang, R., Liu, Z., Li, Y., Song, Y., Liu,H., Wang, Q., Shao, J., Duan, G., & Tan, J. (2023). Task-balanced distillation for object detection. Pattern Recognition, 109320.

[48]

Tian, Z., Shen, C., Chen, H., & He, T. (2019). Fcos: Fully convolutional one-stage object detection, in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9627–9636.

[49]

Wang J, Sun K, Cheng T, Jiang B, Deng C, Zhao Y, Liu D, Mu Y, Tan M, Wang X, et al. Deep high-resolution representation learning for visual recognition IEEE Transactions on Pattern Analysis and Machine Intelligence 2020 43 10 3349-3364

[50]

Wang Y, Liu M, Zheng P, Yang H, and Zou J A smart surface inspection system using faster r-cnn in cloud-edge computing environment Advanced Engineering Informatics 2020 43

Digital Library

[51]

Woo, S., Park, J., Lee, J.-Y., & Kweon, I.S. (2018). Cbam: Convolutional block attention module, in Proceedings of the European conference on computer vision (ECCV), pp. 3–19.

[52]

Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492–1500.

[53]

Yang B, Liu Z, Duan G, and Tan J Mask2defect: A prior knowledge-based data augmentation method for metal surface defect inspection IEEE Transactions on Industrial Informatics 2021 18 10 6743-6755

[54]

Zhang, S., Chi, C., Yao, Y., Lei, Z., & Li, S.Z. (2020). Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9759–9768.

Cited By

Shao YSong Y(2025)CENet: improve counting performance of X-ray surface mounted chip counter via scale favor and cell extractionJournal of Intelligent Manufacturing10.1007/s10845-023-02223-z36:1(303-317)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1007/s10845-023-02223-z

Recommendations

Multi-Scale Cascade Network for Salient Object Detection
MM '17: Proceedings of the 25th ACM international conference on Multimedia

In this paper we present a novel network architecture, called Multi-Scale Cascade Network (MSC-Net), to identify the most visually conspicuous objects in an image. Our network consists of several stages (sub-networks) for handling saliency detection ...
Normalizing Flow Based Defect Detection with Motion Detection
Smart Multimedia
Abstract
Visual defect detection is critical to ensure the quality of most products. However, the majority of small and medium-sized manufacturing enterprises still rely on tedious and error-prone human manual inspection. The main reasons include: 1) the ... $^{}$ $^{}$
Multi-task Compositional Network for Visual Relationship Detection
Abstract
Previous methods treat visual relationship detection as a combination of object detection and predicate detection. However, natural images likely contain hundreds of objects and thousands of object pairs. Relying only on object detection and ...

Comments

Information & Contributors

Information

Published In

cover image Journal of Intelligent Manufacturing

Journal of Intelligent Manufacturing Volume 35, Issue 3

Mar 2024

458 pages

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 10 March 2023

Accepted: 13 February 2023

Received: 03 August 2021

Author Tags

Qualifiers

Research-article

Funding Sources

national key research and development program of china
national natural science foundation of china
high-level talent special support plan of zhejiang province
natural science foundation of zhejiang province

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Shao YSong Y(2025)CENet: improve counting performance of X-ray surface mounted chip counter via scale favor and cell extractionJournal of Intelligent Manufacturing10.1007/s10845-023-02223-z36:1(303-317)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1007/s10845-023-02223-z

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents