research-article

PatternNet: Visual Pattern Mining with Deep Neural Network

Authors:

Joseph G. Ellis,

Shih-Fu ChangAuthors Info & Claims

ICMR '18: Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval

Pages 291 - 299

https://doi.org/10.1145/3206025.3206039

Published: 05 June 2018 Publication History

Abstract

Visual patterns represent the discernible regularity in the visual world. They capture the essential nature of visual objects or scenes. Understanding and modeling visual patterns is a fundamental problem in visual recognition that has wide ranging applications. In this paper, we study the problem of visual pattern mining and propose a novel deep neural network architecture called PatternNet for discovering these patterns that are both discriminative and representative. The proposed PatternNet leverages the filters in the last convolution layer of a convolutional neural network to find locally consistent visual patches, and by combining these filters we can effectively discover unique visual patterns. In addition, PatternNet can discover visual patterns efficiently without performing expensive image patch sampling, and this advantage provides an order of magnitude speedup compared to most other approaches. We evaluate the proposed PatternNet subjectively by showing randomly selected visual patterns which are discovered by our method and quantitatively by performing image classification with the identified visual patterns and comparing our performance with the current state-of-the-art. We also directly evaluate the quality of the discovered visual patterns by leveraging the identified patterns as proposed objects in an image and compare with other relevant methods. Our proposed network and procedure, PatterNet, is able to outperform competing methods for the tasks described.

References

[1]

R. Agrawal, T. Imieli'nski, and A. Swami. Mining association rules between sets of items in large databases. In ACM SIGMOD Record, volume 22, pages 207--216. ACM, 1993.

Digital Library

[2]

B. Alexe, T. Deselaers, and V. Ferrari. Measuring the objectness of image windows. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(11):2189--2202, 2012.

Digital Library

[3]

T. Berg and P. N. Belhumeur. Poof: Part-based one-vs.-one features for fine-grained categorization, face verification, and attribute estimation. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 955--962. IEEE, 2013.

Digital Library

[4]

J. Carreira and C. Sminchisescu. Constrained parametric min-cuts for automatic object segmentation. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 3241--3248. IEEE, 2010.

[5]

Y. Chai, V. Lempitsky, and A. Zisserman. Symbiotic segmentation and part localization for fine-grained categorization. In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 321--328. IEEE, 2013.

Digital Library

[6]

G. Chen, J. Yang, H. Jin, E. Shechtman, J. Brandt, and T. X. Han. Selective pooling vector for fine-grained recognition. In Applications of Computer Vision (WACV), 2015 IEEE Winter Conference on, pages 860--867. IEEE, 2015.

Digital Library

[7]

C. Doersch, A. Gupta, and A. A. Efros. Mid-level visual element discovery as discriminative mode seeking. In Advances in Neural Information Processing Systems, pages 494--502, 2013.

Digital Library

[8]

I. Endres and D. Hoiem. Category independent object proposals. In Computer Vision--ECCV 2010, pages 575--588. Springer, 2010.

Digital Library

[9]

E. Gavves, B. Fernando, C. G. Snoek, A. W. Smeulders, and T. Tuytelaars. Fine-grained categorization by alignments. In Proceedings of the IEEE International Conference on Computer Vision, pages 1713--1720, 2013.

Digital Library

[10]

E. Gavves, B. Fernando, C. G. Snoek, A. W. Smeulders, and T. Tuytelaars. Local alignments for fine-grained categorization. International Journal of Computer Vision, 111(2):191--212, 2014.

Digital Library

[11]

R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 580--587. IEEE, 2014.

Digital Library

[12]

B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and fine-grained localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 447--456, 2015.

[13]

H. Harzallah, F. Jurie, and C. Schmid. Combining efficient object localization and image classification. In Computer Vision, 2009 IEEE 12th International Conference on, pages 237--244. IEEE, 2009.

[14]

M. Juneja, A. Vedaldi, C. Jawahar, and A. Zisserman. Blocks that shout: Distinctive parts for scene classification. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 923--930. IEEE, 2013.

Digital Library

[15]

J. Krause, H. Jin, J. Yang, and L. Fei-Fei. Fine-grained recognition without part annotations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5546--5555, 2015.

[16]

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097--1105, 2012.

Digital Library

[17]

L.-J. Li, H. Su, L. Fei-Fei, and E. P. Xing. Object bank: A high-level image representation for scene classification & semantic feature sparsification. In Advances in neural information processing systems, pages 1378--1386, 2010.

Digital Library

[18]

Q. Li, J. Wu, and Z. Tu. Harvesting mid-level visual concepts from large-scale internet images. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 851--858. IEEE, 2013.

Digital Library

[19]

Y. Li, L. Liu, C. Shen, and A. van den Hengel. Mid-level deep pattern mining. In CVPR, 2015.

[20]

D. G. Lowe. Object recognition from local scale-invariant features. In Computer vision, 1999. The proceedings of the seventh IEEE international conference on, volume 2, pages 1150--1157. Ieee, 1999.

Digital Library

[21]

S. N. Parizi, A. Vedaldi, A. Zisserman, and P. Felzenszwalb. Automatic discovery and optimization of parts for image classification. arXiv preprint arXiv:1412.6598, 2014.

[22]

J. Pu, Y.-G. Jiang, J. Wang, and X. Xue. Which looks like which: Exploring inter-class relationships in fine-grained visual categorization. In Computer Vision--ECCV 2014, pages 425--440. Springer, 2014.

[23]

A. Quattoni and A. Torralba. Recognizing indoor scenes. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 413--420. IEEE, 2009.

[24]

P. Sermanet, A. Frome, and E. Real. Attention for fine-grained categorization. arXiv preprint arXiv:1412.7054, 2014.

[25]

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.

[26]

S. Singh, A. Gupta, and A. Efros. Unsupervised discovery of mid-level discriminative patches. Computer Vision--ECCV 2012, pages 73--86, 2012.

[27]

J. Sun and J. Ponce. Learning discriminative part detectors for image classification and cosegmentation. In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 3400--3407. IEEE, 2013.

Digital Library

[28]

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1--9, 2015.

[29]

J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W. Smeulders. Selective search for object recognition. International journal of computer vision, 104(2):154--171, 2013.

Digital Library

[30]

A. Vedaldi, V. Gulshan, M. Varma, and A. Zisserman. Multiple kernels for object detection. In Computer Vision, 2009 IEEE 12th International Conference on, pages 606--613. IEEE, 2009.

[31]

M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Computer Vision--ECCV 2014, pages 818--833. Springer, 2014.

[32]

N. Zhang, J. Donahue, R. Girshick, and T. Darrell. Part-based r-cnns for fine-grained category detection. In Computer Vision--ECCV 2014, pages 834--849. Springer, 2014.

[33]

W. Zhang, H. Li, C.-W. Ngo, and S.-F. Chang. Scalable visual instance mining with threads of features. In Proceedings of the ACM International Conference on Multimedia, pages 297--306. ACM, 2014.

Digital Library

Cited By

Liang XLiang ZShi HZhang XZhou YMa Y(2024)Multipattern Mining Using Pattern-Level Contrastive Learning and Multipattern Activation MapIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.321807335:7(9080-9094)Online publication date: Jul-2024
https://doi.org/10.1109/TNNLS.2022.3218073
Mazini Rodrigues CBoutry NNajman L(2024)Unsupervised discovery of Interpretable Visual ConceptsInformation Sciences10.1016/j.ins.2024.120159(120159)Online publication date: Jan-2024
https://doi.org/10.1016/j.ins.2024.120159
Bhattacharya ASingha MJha ABanerjee B(2023)C-SAW: Self-Supervised Prompt Learning for Image Generalization in Remote SensingProceedings of the Fourteenth Indian Conference on Computer Vision, Graphics and Image Processing10.1145/3627631.3627669(1-10)Online publication date: 15-Dec-2023
https://dl.acm.org/doi/10.1145/3627631.3627669
Show More Cited By

Index Terms

PatternNet: Visual Pattern Mining with Deep Neural Network
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision

Recommendations

Stable Visual Pattern Mining via Pattern Probability Distribution
Pattern Recognition and Computer Vision
Abstract
Visual patterns are the fundamental elements that compose an image and often convey higher-level semantics. Visual pattern mining can be widely applied to real-world applications and various downstream tasks, such as tourist destination ...
Ada-Sal Network

Convolutional neural networks (CNNs) have become state-of-the-art for image classification. Inspired by the physiological mechanism of saliency in real human visual system (HVS), we had previously proposed the Sal-Mask Connection. As HVS tends to select ...
Integrating object proposal with attention networks for video saliency detection
Abstract
Video saliency detection is an active research issue in both information science and visual psychology. In this paper, we propose an efficient video saliency-detection model, based on integrating object-proposal with attention networks, for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMR '18: Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval

June 2018

550 pages

ISBN:9781450350464

DOI:10.1145/3206025

Conference Chairs:
Kiyoharu Aizawa
The Univ. of Tokyo, Japan
,
Michael Lew
Leiden Univ., Netherlands
,
Shin'ichi Satoh
National Inst. of Informatics, Japan

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Poster

Author Tags

Qualifiers

Research-article

Conference

ICMR '18

Sponsor:

SIGMM

ICMR '18: International Conference on Multimedia Retrieval

June 11 - 14, 2018

Yokohama, Japan

Acceptance Rates

ICMR '18 Paper Acceptance Rate 44 of 136 submissions, 32%;

Overall Acceptance Rate 254 of 830 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
352
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)4

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liang XLiang ZShi HZhang XZhou YMa Y(2024)Multipattern Mining Using Pattern-Level Contrastive Learning and Multipattern Activation MapIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.321807335:7(9080-9094)Online publication date: Jul-2024
https://doi.org/10.1109/TNNLS.2022.3218073
Mazini Rodrigues CBoutry NNajman L(2024)Unsupervised discovery of Interpretable Visual ConceptsInformation Sciences10.1016/j.ins.2024.120159(120159)Online publication date: Jan-2024
https://doi.org/10.1016/j.ins.2024.120159
Bhattacharya ASingha MJha ABanerjee B(2023)C-SAW: Self-Supervised Prompt Learning for Image Generalization in Remote SensingProceedings of the Fourteenth Indian Conference on Computer Vision, Graphics and Image Processing10.1145/3627631.3627669(1-10)Online publication date: 15-Dec-2023
https://dl.acm.org/doi/10.1145/3627631.3627669
Singha MJha ASolanki BBose SBanerjee B(2023)APPLeNet: Visual Attention Parameterized Prompt Learning for Few-Shot Remote Sensing Image Generalization using CLIP2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW59228.2023.00196(2024-2034)Online publication date: Jun-2023
https://doi.org/10.1109/CVPRW59228.2023.00196
Ma YLiang XLin XShi G(2023)Stable Visual Pattern Mining via Pattern Probability DistributionPattern Recognition and Computer Vision10.1007/978-981-99-8543-2_23(280-292)Online publication date: 29-Dec-2023
https://doi.org/10.1007/978-981-99-8543-2_23
Dong JSitler KScalia JGe YBireta PSihota NHoelen TLowry G(2022)Application of Transfer Learning and Convolutional Neural Networks for Autonomous Oil Sheen MonitoringApplied Sciences10.3390/app1217886512:17(8865)Online publication date: 3-Sep-2022
https://doi.org/10.3390/app12178865
Urruchi CCervantes-Chauca DHuamanchahua D(2022)Proposal of a Swimming Pool Drowning Detection System using Cameras and Raspberry Pi based on Machine Learning2022 2nd International Conference on Robotics, Automation and Artificial Intelligence (RAAI)10.1109/RAAI56146.2022.10092956(178-181)Online publication date: 9-Dec-2022
https://doi.org/10.1109/RAAI56146.2022.10092956
Iida TKomatsu TKaneda KHirakawa TYamashita TFujiyoshi HSugiura K(2022)Visual Explanation Generation Based on Lambda Attention Branch NetworksComputer Vision – ACCV 202210.1007/978-3-031-26284-5_29(475-490)Online publication date: 4-Dec-2022
https://dl.acm.org/doi/10.1007/978-3-031-26284-5_29
Zhao GZhang PShen YJiang X(2022)Passive User Authentication Utilizing Consecutive Touch Action Features for IIoT SystemsScience of Cyber Security10.1007/978-3-031-17551-0_18(276-284)Online publication date: 10-Aug-2022
https://dl.acm.org/doi/10.1007/978-3-031-17551-0_18
Malik KRobertson C(2021)Landscape Similarity Analysis Using Texture Encoded Deep-Learning Features on Unclassified Remote Sensing ImageryRemote Sensing10.3390/rs1303049213:3(492)Online publication date: 30-Jan-2021
https://doi.org/10.3390/rs13030492
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents