Abstract
Deep learning has paved the way for strong recognition systems which are often both trained on and applied to natural images. In this paper, we examine the give-and-take relationship between such visual recognition systems and the rich information available in the fine arts. First, we find that visual recognition systems designed for natural images can work surprisingly well on paintings. In particular, we find that interactive segmentation tools can be used to cleanly annotate polygonal segments within paintings, a task which is time consuming to undertake by hand. We also find that FasterRCNN, a model which has been designed for object recognition in natural scenes, can be quickly repurposed for detection of materials in paintings. Second, we show that learning from paintings can be beneficial for neural networks that are intended to be used on natural images. We find that training on paintings instead of natural images can improve the quality of learned features and we further find that a large number of paintings can be a valuable source of test data for evaluating domain adaptation algorithms. Our experiments are based on a novel large-scale annotated database of material depictions in paintings which we detail in a separate manuscript.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Augart, I., Saß, M., Wenderholm, I.: Steinformen. De Gruyter, Berlin, Boston (2018). https://doi.org/10.1515/9783110583618. https://www.degruyter.com/view/title/535173
Bell, S., Upchurch, P., Snavely, N., Bala, K.: OpenSurfaces: a richly annotated catalog of surface appearance. ACM Trans. Graph. 32(4), 1 (2013). https://doi.org/10.1145/2461912.2462002. http://dl.acm.org/citation.cfm?doid=2461912.2462002
Bell, S., Upchurch, P., Snavely, N., Bala, K.: Material recognition in the wild with the materials in context database. In: Computer Vision and Pattern Recognition (CVPR) (2015)
Benenson, R., Popov, S., Ferrari, V.: Large-scale interactive object segmentation with human annotators. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11700–11709 (2019)
Beurs, W.: “De groote waereld in’t kleen geschildert, of Schilderagtig tafereel van’s weerelds schilderyen. Kortelijk vervat in ses boeken: Verklarende de hooftverwen, haare verscheide mengelinge in oly, en der zelver gebruik...”. By Johannes en Gillis Janssonius van Waesberge (1692)
Bol, M., Lehmann, A.S.: Painting skin and water: towards a material iconography of translucent motifs in early Netherlandish painting. In: Rogier van der Weyden in context: papers presented at the Seventeenth Symposium for the Study of Underdrawing and Technology in Painting held in Leuven, 22–24 October, pp. 215–228. Peeters (2012)
Caesar, H., Uijlings, J., Ferrari, V.: Coco-stuff: Thing and stuff classes in context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1209–1218 (2018)
Cavanagh, P.: The artist as neuroscientist. Nature 434(7031), 301–307 (2005). https://doi.org/10.1038/434301a
Chang, W.G., You, T., Seo, S., Kwak, S., Han, B.: Domain-specific batch normalization for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7354–7362 (2019)
Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3223 (2016)
Crowley, E.J., Zisserman, A.: The state of the art: object retrieval in paintings using discriminative regions. In: British Machine Vision Conference (2014)
Di Cicco, F., Wijntjes, M.W., Pont, S.C.: Understanding gloss perception through the lens of art: combining perception, image analysis, and painting recipes of 17th century painted grapes. J. Vis. 19(3), 1–15 (2019). https://doi.org/10.1167/19.3.7
Dietrich, R.: Rocks depicted in painting & sculpture. Rocks Miner. 65(3), 224–236 (1990)
Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vision 88(2), 303–338 (2010)
Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Martinez-Gonzalez, P., Garcia-Rodriguez, J.: A survey on deep learning techniques for image and video semantic segmentation. Appl. Soft. Comput. 70, 41–65 (2018)
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel, W.: Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231 (2018)
Hafiz, A.M., Bhat, G.M.: A survey on instance segmentation: state of the art. International Journal of Multimedia Information Retrieval pp. 1–19 (2020)
Jing, Y., Yang, Y., Feng, Z., Ye, J., Yu, Y., Song, M.: Neural style transfer: a review. IEEE Trans. Vis. Comput. Graph. 26, 3365–3385 (2019)
Kang, G., Jiang, L., Yang, Y., Hauptmann, A.G.: Contrastive adaptation network for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4893–4902 (2019)
Kemp, M., et al.: The science of art: optical themes in western art from brunelleschi to seurat (1990)
Kuznetsova, A., et al.: The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv:1811.00982 (2018)
Lehmann, A.S.: Fleshing out the body: The’colours of the naked’in workshop practice and art theory, 1400–1600. Nederlands Kunsthistorisch Jaarboek 59, 86 (2008)
Li, D., Yang, Y., Song, Y.Z., Hospedales, T.M.: Deeper, broader and artier domain generalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5542–5550 (2017)
Lin, H., Upchurch, P., Bala, K.: Block annotation: Better image annotation with sub-image decomposition. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5290–5300 (2019)
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Lin, Z., Sun, J., Davis, A., Snavely, N.: Visual chirality. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12295–12303 (2020)
Ling, H., Gao, J., Kar, A., Chen, W., Fidler, S.: Fast interactive object annotation with curve-gcn. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5257–5266 (2019)
Long, M., Cao, Y., Wang, J., Jordan, M.: Learning transferable features with deep adaptation networks. In: International Conference on Machine Learning, pp. 97–105. PMLR (2015)
Mall, U., Matzen, K., Hariharan, B., Snavely, N., Bala, K.: Geostyle: discovering fashion trends and events. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 411–420 (2019)
Mamassian, P.: Ambiguities and conventions in the perception of visual art. Vision. Res. 48(20), 2143–2153 (2008). https://doi.org/10.1016/j.visres.2008.06.010
Maninis, K.K., Caelles, S., Pont-Tuset, J., Van Gool, L.: Deep extreme cut: from extreme points to object segmentation (2017). http://arxiv.org/abs/1711.09081
Matzen, K., Bala, K., Snavely, N.: Streetstyle: exploring world-wide clothing styles from millions of photos. arXiv preprint arXiv:1706.01869 (2017)
van Eikema, M.H.: Hommes: The contours in the paintings of the oranjezaal, huis ten bosch’ (2005)
Panofsky, E.: Perspective as Symbolic Form. Princeton University Press, Princeton (1927/2020)
Papadopoulos, D.P., Uijlings, J.R.R., Keller, F., Ferrari, V.: Extreme clicking for efficient object annotation. Int. J. Comput. Vis. (2017). https://doi.org/10.1109/ICCV.2017.528. http://arxiv.org/abs/1708.02750
Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., Wang, B.: Moment matching for multi-source domain adaptation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1406–1415 (2019)
Pirenne, M.H.: Optics, painting & photography. Cambridge University Press
Pottasch, C.: Frans van mieris’s painting technique as one of the possible sources for willem beurs’s treatise on painting. Art & Perception, pp. 1–17, 13 June 2020. https://doi.org/10.1163/22134913-bja10013. https://brill.com/view/journals/artp/aop/article-10.1163-22134913-bja10013/article-10.1163-22134913-bja10013.xml
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)
Rother, C., Kolmogorov, V., Blake, A.: “grabcut” interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. (TOG) 23(3), 309–314 (2004)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014)
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The inaturalist species classification and detection dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8769–8778 (2018)
Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset (2011)
Wei, X.S., Wu, J., Cui, Q.: Deep learning for fine-grained image analysis: a survey. arXiv preprint arXiv:1907.03069 (2019)
White, J.: The birth and rebirth of pictorial space, Cambridge MA (1957)
Wiersma, L.: Colouring – material depiction in flemish and dutch baroque art theory. Art & Perception, pp. 1–23, 22 April 2020. https://doi.org/10.1163/22134913-bja10005. https://brill.com/view/journals/artp/aop/article-10.1163-22134913-bja10005/article-10.1163-22134913-bja10005.xml
Wijntjes, M.W.A., Spoiala, C., de Ridder, H.: Thurstonian scaling and the perception of painterly translucency. Art Perception, 1–24, 04 Sep 2020. https://doi.org/10.1163/22134913-bja10021. https://brill.com/view/journals/artp/aop/article-10.1163-22134913-bja10021/article-10.1163-22134913-bja10021.xml
Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2 (2019)
Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L., Hoi, S.C.: Deep learning for person re-identification: a survey and outlook. arXiv preprint arXiv:2001.04193 (2020)
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 633–641 (2017)
van Zuijlen, M.J., Pont, S.C., Wijntjes, M.W.: Painterly depiction of material properties. J. Vision 20(7), 7–7 (2020)
Acknowledgements
This work was funded in part by Google, NSF (CHS-1617861 and CHS-1513967), NSERC (PGS-D 516803 2018), and the Netherlands Organization for Scientific Research (NWO) project 276-54-001.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Lin, H., Van Zuijlen, M., Wijntjes, M.W.A., Pont, S.C., Bala, K. (2021). Insights from a Large-Scale Database of Material Depictions in Paintings. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12663. Springer, Cham. https://doi.org/10.1007/978-3-030-68796-0_38
Download citation
DOI: https://doi.org/10.1007/978-3-030-68796-0_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68795-3
Online ISBN: 978-3-030-68796-0
eBook Packages: Computer ScienceComputer Science (R0)
