Abstract
We present a multiview method for markerless motion capture of multiple people. The main challenge in this problem is to determine cross-view correspondences for the 2D joints in the presence of noise. We propose a 3D hypothesis clustering technique to solve this problem. The core idea is to transform joint matching in 2D space into a clustering problem in a 3D hypothesis space. In this way, evidence from photometric appearance, multiview geometry, and bone length can be integrated to solve the clustering problem efficiently and robustly. Each cluster encodes a set of matched 2D joints for the same person across different views, from which the 3D joints can be effectively inferred. We then assemble the inferred 3D joints to form full-body skeletons for all persons in a bottom–up way. Our experiments demonstrate the robustness of our approach even in challenging cases with heavy occlusion, closely interacting people, and few cameras. We have evaluated our method on many datasets, and our results show that it has significantly lower estimation errors than many state-of-the-art methods.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Toshev, A.; Szegedy, C. DeepPose: Human pose estimation via deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1653–1660, 2014.
Wei, S.-E.; Ramakrishna, V.; Kanade, T.; Sheikh, Y. Convolutional pose machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4724–4732, 2016.
Insafutdinov, E.; Pishchulin, L.; Andres, B.; Andriluka, M.; Schiele, B. DeeperCut: A deeper, stronger, and faster multi-person pose estimation model. In: Computer Vision–ECCV 2016. Lecture Notes in Computer Science, Vol. 9910. Leibe B.; Matas J.; Sebe N.; Welling M. Eds. Springer Cham, 34–50, 2016.
Cao, Z.; Simon, T.; Wei, S.-E.; Sheikh, Y. Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1302–1310, 2017.
Li, M.; Zhou, Z.; Li, J.; Liu, X. Bottom-up pose estimation of multiple person with bounding box constraint. In: Proceedings of the 24th International Conference on Pattern Recognition, 115–120, 2018.
Wang, H.; An, W. P.; Wang, X.; Fang, L.; Yuan, J. Magnify-net for multi-person 2D pose estimation. In: Proceedings of the IEEE International Conference on Multimedia and Expo, 1–6, 2018.
Amin, S.; Andriluka, M.; Rohrbach, M.; Schiele, B. Multi-view pictorial structures for 3D human pose estimation. In: Proceedings of the British Machine Vision Conference, 2013.
Belagiannis, V.; Amin, S.; Andriluka, M.; Schiele, B.; Navab, N.; Ilic, S. 3D pictorial structures for multiple human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1669–1676, 2014.
Belagiannis, V.; Amin, S.; Andriluka, M.; Schiele, B.; Navab, N.; Ilic, S. 3D pictorial structures revisited: Multiple human pose estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 38, No. 10, 1929–1942, 2016.
Ershadi-Nasab, S.; Noury, E.; Kasaei, S.; Sanaei, E. Multiple human 3D pose estimation from multiview images. Multimedia Tools and Applications Vol. 77, No. 12, 15573–15601, 2018.
Joo, H.; Simon, T.; Li, X.; Liu, H.; Tan, L.; Gui, L.; Banerjee, S.; Godisart, T. S.; Nabbe, B.; Matthews, I. et al. Panoptic studio: A massively multiview system for social interaction capture. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 41, No. 1, 190–204, 2019.
Joo, H.; Liu, H.; Tan, L.; Gui, L.; Nabbe, B.; Matthews, I.; Kanade, T.; Nobuhara, S.; Sheikh, Y. Panoptic studio: A massively multiview system for social motion capture. In: Proceedings of the IEEE International Conference on Computer Vision, 3334–3342, 2015.
Kadkhodamohammadi, A.; Padoy, N. A generalizable approach for multi-view 3D human pose regression. arXiv preprint arXiv:1804.10462, 2018.
Dong, J.; Jiang, W.; Huang, Q.; Bao, H.; Zhou X. Fast and robust multi-person 3D pose estimation from multiple views. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7792–7801, 2019.
Ren, S. Q.; He, K. M.; Girshick, R.; Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 39, No. 6, 1137–1149, 2017.
Chen, Y.; Wang, Z.; Peng, Y.; Zhang, Z.; Yu, G.; Sun, J. Cascaded pyramid network for multi-person pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7103–7112, 2018.
Zhong, Z.; Zheng, L.; Zheng, Z.; Li, S.; Yang, Y. Camera style adaptation for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5157–5166, 2018.
Li, M. P.; Zhou, Z. M.; Liu, X. G. Multi-person pose estimation using bounding box constraint and LSTM. IEEE Transactions on Multimedia Vol. 21, No. 10, 2653–2663, 2019.
Huang, Q. X.; Zhang, G. X.; Gao, L.; Hu, S. M.; Butscher, A.; Guibas, L. An optimization approach for extracting and encoding consistent maps in a shape collection. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 167, 2012.
Zhou, T.; Jae Lee, Y.; Yu, S. X.; Efros, A. A. FlowWeb: Joint image set alignment by weaving consistent, pixel-wise correspondences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1191–1200, 2015.
Zhou, X.; Zhu, M.; Daniilidis, K. Multi-image matching via fast alternating minimization. In: Proceedings of the IEEE International Conference on Computer Vision, 4032–4040, 2015.
Ester, M.; Kriegel, H.-P.; Sander, J.; Xu, X. A densitybased algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the KDD-96, 1996.
Levenberg, K. A method for the solution of certain nonlinear problems in least squares. Quarterly of Applied Mathematics Vol. 2, No. 2, 164–168, 1944.
Marquardt, D. W. An algorithm for least-squares estimation of nonlinear parameters. Journal of the Society for Industrial and Applied Mathematics Vol. 11, No. 2, 431–441, 1963.
Andriluka, M.; Pishchulin, L.; Gehler, P.; Schiele, B. 2D human pose estimation: New benchmark and state of the art analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3686–3693, 2014.
Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable comments. This work was partially supported by National Natural Science Foundation of China (No. 61872317) and FaceUnity Technology.
Author information
Authors and Affiliations
Corresponding author
Additional information
Miaopeng Li is a Ph.D. student in the State Key Lab of CAD&CG, Zhejiang University, China. She received her bachelor degree from Northwestern Polytechnic University in 2016. Her research interests include markerless human motion capture, human pose estimation, and 3D reconstruction, and their applications.
Zimeng Zhou is a master student in the State Key Lab of CAD&CG, Zhejiang University. His research interests are computer vision and computer graphics, with a particular focus on human pose estimation.
Xinguo Liu received his bachelor and Ph.D. degrees in applied mathematics from Zhejiang University, in 1995 and 2001, respectively. He is a professor of computer science in the State Key Lab of CAD&CG, Zhejiang University. His research interests include geometry processing, realistic and image based rendering, deformable objects, and 3D reconstruction.
Electronic supplementary material
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Li, M., Zhou, Z. & Liu, X. 3D hypothesis clustering for cross-view matching in multi-person motion capture. Comp. Visual Media 6, 147–156 (2020). https://doi.org/10.1007/s41095-020-0171-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-020-0171-y