Abstract
Deep convolution neural network (CNN) models with millions of parameters trained in large-scale datasets make domain adaptation difficult to be realized. In order to be applied for different application scenarios, various light weight network models have been proposed. These models perform well in large-scale datasets but are hard to train from randomly initialized weights when lack of data. Our framework is proposed to connect a pre-trained deep model with a light weight model by enforcing feature distributions of the two models being identical. It is proved in our work that knowledge in source model can be transferred to target light weight model by identical distribution loss. Meanwhile, distribution loss allows training dataset to utilize sparse labeled data in semi-supervised classification task. Moreover, distribution loss can be applied to large amount of unlabeled data from target domain. In the experiments, several standard benchmarks on domain adaptation are evaluated and our work gets state-of-the-art performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aytar, Y., Zisserman, A.: Tabula rasa: model transfer for object category detection. In: International Conference on Computer Vision, pp. 2252–2259 (2011)
Ba, L.J., Caruana, R.: Do deep nets really need to be deep? In: Advances in Neural Information Processing Systems, pp. 2654–2662 (2013)
Bergamo, A., Torresani, L.: Exploiting weakly-labeled web images to improve object classification: a domain adaptation approach. In: Neural Information Processing Systems, pp. 181–189 (2010)
Chopra, S.: DLID: deep learning for domain adaptation by interpolating between domains. In: ICML Workshop on Challenges in Representation Learning (2013)
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (almost) from scratch. Mach. Learn. Res. 12(Aug), 2493–2537 (2011)
Donahue, J., et al.: Decaf: a deep convolutional activation feature for generic visual recognition 50(1), I-647 (2013)
Duan, L., Tsang, I.W., Xu, D.: Domain transfer multiple kernel learning. Pattern Anal. Mach. Intell. 34(3), 465–479 (2012)
Fernando, B., Habrard, A., Sebban, M., Tuytelaars, T.: Unsupervised visual domain adaptation using subspace alignment. In: International Conference on Computer Vision, pp. 2960–2967 (2013)
Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: International Conference on Machine Learning, pp. 1180–1189 (2015)
Girshick, R.B., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Computer Vision and Pattern Recognition, pp. 580–587 (2014)
Glorot, X., Bordes, A., Bengio, Y.: Domain adaptation for large-scale sentiment classification: a deep learning approach. In: Machine Learning, pp. 513–520 (2011)
Gong, Y., Liu, L., Yang, M., Bourdev, L.: Compressing deep convolutional networks using vector quantization. Computer Science (2014)
Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re-identification. In: Computer Vision and Pattern Recognition (2017)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. Comput. Sci. 14(7), 38–39 (2015)
Hoffman, J., et al.: LSDA: large scale detection through adaptation. In: Advances in Neural Information Processing Systems, pp. 3536–3544 (2014)
Hoffman, J., Rodner, E., Donahue, J., Darrell, T., Saenko, K.: Efficient learning of domain-invariant image representations. Computer Science (2013)
Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. In: Computer Vision and Pattern Recognition (2017)
Jia, Y., et al.: Caffe: convolutional architecture for fast feature embedding. In: ACM Multimedia, pp. 675–678 (2014)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Neural Information Processing Systems, pp. 1097–1105 (2012)
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Mansour, Y., Mohri, M., Rostamizadeh, A.: Domain adaptation: learning bounds and algorithms. arXiv preprint arXiv:0902.3430 (2009)
Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)
Pan, S.J., Yang, Q., et al.: A survey on transfer learning. Knowl. Data Eng. 22(10), 1345–1359 (2010)
Parkhi, O.M., Vedaldi, A., Zisserman, A., et al.: Deep face recognition. In: The British Machine Vision Conference, vol. 1, p. 6 (2015)
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Saenko, K., Kulis, B., Fritz, M., Darrell, T.: Adapting visual category models to new domains. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 213–226. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_16
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., Lecun, Y.: Overfeat: integrated recognition, localization and detection using convolutional networks. Eprint Arxiv (2013)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Computer Science (2014)
Szegedy, C., Toshev, A., Erhan, D.: Deep neural networks for object detection. In: Advances in Neural Information Processing Systems, vol. 26, pp. 2553–2561 (2013)
Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Simultaneous deep transfer across domains and tasks. In: IEEE International Conference on Computer Vision, pp. 4068–4076 (2017)
Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: Maximizing for domain invariance. Computer Science (2014)
Yang, J., Yan, R., Hauptmann, A.G.: Adapting SVM classifiers to data with shifted distributions. In: International Conference Data Mining Workshops, pp. 69–76. IEEE (2007)
Yi, D., Lei, Z., Liao, S., Li, S.Z.: Learning face representation from scratch. Computer Science (2014)
Zhang, K., Schölkopf, B., Muandet, K., Wang, Z.: Domain adaptation under target and conditional shift. In: International Conference on Machine Learning, pp. 819–827 (2013)
Acknowledgment
This paper was partially financially supported by National Natural Science Foundation of China under grants 61533012, 91748120 and 61521063.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Wei, X., Chen, Y., Su, J. (2018). Domain Adaptation via Identical Distribution Across Models and Tasks. In: Cheng, L., Leung, A., Ozawa, S. (eds) Neural Information Processing. ICONIP 2018. Lecture Notes in Computer Science(), vol 11301. Springer, Cham. https://doi.org/10.1007/978-3-030-04167-0_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-04167-0_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04166-3
Online ISBN: 978-3-030-04167-0
eBook Packages: Computer ScienceComputer Science (R0)