VILL: Toward Efficient and Automatic Visual Landmark Labeling
Authors: 
Qun Niu, Kunxin Zhu, Suining He, Shaoqi Cen, S.-H. Gary Chan, Ning LiuAuthors Info & Claims
Qun Niu, Kunxin Zhu, Suining He, Shaoqi Cen, S.-H. Gary Chan, Ning LiuAuthors Info & ClaimsArticle No.: 74, Pages 1 - 25
Abstract
Of all indoor localization techniques, vision-based localization emerges as a promising one, mainly due to the ubiquity of rich visual features. Visual landmarks, which present distinguishing textures, play a fundamental role in visual indoor localization. However, few researches focus on visual landmark labeling. Preliminary arts usually designate a surveyor to select and record visual landmarks, which is tedious and time-consuming. Furthermore, due to structural changes (e.g., renovation), the visual landmark database may be outdated, leading to degraded localization accuracy.
To overcome these limitations, we propose VILL, a user-friendly, efficient, and accurate approach for visual landmark labeling. VILL asks a user to sweep the camera to take a video clip of his/her surroundings. In the construction stage, VILL identifies unlabeled visual landmarks from videos adaptively according to the graph-based visual correlation representation. Based on the spatial correlations with selected anchor landmarks, VILL estimates locations of unlabeled ones on the floorplan accurately. In the update stage, VILL formulates an alteration identification model based on the judgments from different users to identify altered landmarks accurately. Extensive experimental results in two different trial sites show that VILL reduces the site survey substantially (by at least 65.9%) and achieves comparable accuracy.
References
[1]
Mohamed Abdelaal, Daniel Reichelt, Frank Dürr, Kurt Rothermel, Lavinia Runceanu, Susanne Becker, and Dieter Fritsch. 2018. ComNSense: Grammar-driven crowd-sourcing of point clouds for automatic indoor mapping. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (March 2018), Article 1, 26 pages.
[2]
Yasin Almalioglu, Muhamad Risqi U. Saputra, Pedro P. B. de Gusmão, Andrew Markham, and Niki Trigoni. 2019. GANVO: Unsupervised deep monocular visual odometry and depth estimation with generative adversarial networks. In Proceedings of the 2019 International Conference on Robotics and Automation. IEEE, Los Alamitos, CA, 5474–5480.
[3]
Yasin Almalioglu, Mehmet Turan, Muhamad Risqi U. Saputra, Pedro P. B. de Gusmão, Andrew Markham, and Niki Trigoni. 2022. SelfVIO: Self-supervised deep monocular visual-inertial odometry and depth estimation. Neural Networks 150 (2022), 119–136.
[4]
Heba Aly, Anas Basalamah, and Moustafa Youssef. 2017. Automatic rich map semantics identification through smartphone-based crowd-sensing. IEEE Transactions on Mobile Computing 16, 10 (2017), 2712–2725.
[5]
Roshan Ayyalasomayajula, Aditya Arun, Chenfeng Wu, Sanatan Sharma, Abhishek Rajkumar Sethi, Deepak Vasisht, and Dinesh Bharadia. 2020. Deep learning based wireless localization for indoor navigation. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. ACM, New York, NY, Article 17, 14 pages.
[6]
Yuan Chen, Keiko Katsuragawa, and Edward Lank. 2020. Understanding viewport- and world-based pointing with everyday smart devices in immersive augmented reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1–13.
[7]
Junyoung Choi, Gyujin Lee, Sunghyun Choi, and Saewoong Bahk. 2022. Smartphone based indoor path estimation and localization without human intervention. IEEE Transactions on Mobile Computing 21, 2 (2022), 681–695.
[8]
Erqun Dong, Jingao Xu, Chenshu Wu, Yunhao Liu, and Zheng Yang. 2019. Pair-Navi: Peer-to-peer indoor navigation with mobile visual SLAM. In Proceedings of the IEEE Conference on Computer Communications. IEEE, Los Alamitos, CA, 1189–1197.
[9]
Jiang Dong, Marius Noreikis, Yu Xiao, and Antti Ylä-Jääski. 2019. ViNav: A vision-based indoor navigation system for smartphones. IEEE Transactions on Mobile Computing 18, 6 (2019), 1461–1475.
[10]
Liang Dong, Jingao Xu, Guoxuan Chi, Danyang Li, Xinglin Zhang, Jianbo Li, Qiang Ma, and Zheng Yang. 2021. Enabling surveillance cameras to navigate. ACM Transactions on Sensor Networks 17, 4 (Sept. 2021), Article 35, 20 pages.
[11]
Ruipeng Gao, Yang Tian, Fan Ye, Guojie Luo, Kaigui Bian, Yizhou Wang, Tao Wang, and Xiaoming Li. 2016. Sextant: Towards ubiquitous indoor localization service by photo-taking of the environment. IEEE Transactions on Mobile Computing 15, 2 (Feb.2016), 460–474.
[12]
Ruipeng Gao, Xuan Xiao, Weiwei Xing, Chi Li, and Lei Liu. 2022. Unsupervised learning of monocular depth and ego-motion in outdoor/indoor environments. IEEE Internet of Things Journal 9, 17 (2022), 16247–16258.
[13]
Ruipeng Gao, Mingmin Zhao, Tao Ye, Fan Ye, Guojie Luo, Yizhou Wang, Kaigui Bian, Tao Wang, and Xiaoming Li. 2016. Multi-story indoor floor plan reconstruction via mobile crowdsensing. IEEE Transactions on Mobile Computing 15, 6 (June2016), 1427–1442.
[14]
Ruipeng Gao, Bing Zhou, Fan Ye, and Yizhou Wang. 2019. Fast and resilient indoor floor plan construction with a single user. IEEE Transactions on Mobile Computing 18, 5 (May2019), 1083–1097.
[15]
Yongbin Gao, Fangzheng Tian, Jun Li, Zhijun Fang, Saba Al-Rubaye, Wei Song, and Yier Yan. 2022. Joint optimization of depth and ego-motion for intelligent autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems. Early access, March 24, 2022.
[16]
Fuqiang Gu, Xuke Hu, Milad Ramezani, Debaditya Acharya, Kourosh Khoshelham, Shahrokh Valaee, and Jianga Shang. 2019. Indoor localization improved by spatial context—A survey. ACM Computing Surveys 52, 3 (July 2019), Article 64, 35 pages.
[17]
Maya Gupta, Ali Abdolrahmani, Emory Edwards, Mayra Cortez, Andrew Tumang, Yasmin Majali, Marc Lazaga, et al. 2020. Towards more universal wayfinding technologies: Navigation preferences across disabilities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1–13.
[18]
Farzam Hejazi, Katarina Vuckovic, and Nazanin Rahnavard. 2021. DyLoc: Dynamic localization for massive MIMO using predictive recurrent neural networks. In Proceedings of the IEEE Conference on Computer Communications. IEEE, Los Alamitos, CA, 1–9.
[19]
Chao Huang, Haoran Yu, Jianwei Huang, and Randall A. Berry. 2021. Strategic information revelation in crowdsourcing systems without verification. In Proceedings of the IEEE Conference on Computer Communications. IEEE, Los Alamitos, CA, 1–10.
[20]
Gang Huang, Zhaozheng Hu, Jie Wu, Hanbiao Xiao, and Fan Zhang. 2020. WiFi and vision-integrated fingerprint for smartphone-based self-localization in public indoor scenes. IEEE Internet of Things Journal 7, 8 (2020), 6748–6761.
[21]
Shaocheng Jia, Xin Pei, Xiao Jing, and Danya Yao. 2022. Self-supervised 3D reconstruction and ego-motion estimation via on-board monocular video. IEEE Transactions on Intelligent Transportation Systems 23, 7 (2022), 7557–7569.
[22]
Hongbo Jiang, Wenping Liu, Guoyin Jiang, Yufu Jia, Xingjun Liu, Zhicheng Lui, Xiaofei Liao, Jing Xing, and Daibo Liu. 2021. Fly-Navi: A novel indoor navigation system with on-the-fly map generation. IEEE Transactions on Mobile Computing 20, 9 (2021), 2820–2834.
[23]
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. 2019. Few-shot object detection via feature reweighting. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, Los Alamitos, CA, 8420–8429.
[24]
Danyang Li, Jingao Xu, Zheng Yang, Yumeng Lu, Qian Zhang, and Xinglin Zhang. 2021. Train once, locate anytime for anyone: Adversarial learning based wireless localization. In Proceedings of the IEEE Conference on Computer Communications. IEEE, Los Alamitos, CA, 1–9.
[25]
Danyang Li, Jingao Xu, Zheng Yang, Chenshu Wu, Jianbo Li, and Nicholas D. Lane. 2022. Wireless localization with spatial-temporal robust fingerprints. ACM Transactions on Sensor Networks 18, 1 (Oct. 2022), Article 15, 23 pages.
[26]
Qing Li, Jiasong Zhu, Tao Liu, Jon Garibaldi, Qingquan Li, and Guoping Qiu. 2017. Visual landmark sequence-based indoor localization. In Proceedings of the 1st Workshop on Artificial Intelligence and Deep Learning for Geographic Knowledge Discovery (GeoAI’17). ACM, New York, NY, 14–23.
[27]
Tao Li, Dianqi Han, Yimin Chen, Rui Zhang, Yanchao Zhang, and Terri Hedgpeth. 2020. IndoorWaze: A crowdsourcing-based context-aware indoor navigation system. IEEE Transactions on Wireless Communications 19, 8 (2020), 5461–5472.
[28]
Manni Liu, Jialuo Du, Qing Zhou, Zhichao Cao, and Yunhao Liu. 2021. EyeLoc: Smartphone vision-enabled plug-n-play indoor localization in large shopping malls. IEEE Internet of Things Journal 8, 7 (2021), 5585–5598.
[29]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, Los Alamitos, CA, 10012–10022.
[30]
David G. Lowe. 2004. Image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2 (Nov.2004), 91–110.
[31]
Qun Niu, Mingkuan Li, Suining He, Chengying Gao, S. H. Gary Chan, and Xiaonan Luo. 2019. Resource-efficient and automated image-based indoor localization. ACM Transactions on Sensor Networks 15, 2 (Feb. 2019), Article 19, 31 pages.
[32]
Meng-Shiuan Pan and Kuan-Ying Li. 2021. ezNavi: An easy-to-operate indoor navigation system based on pedestrian dead reckoning and crowdsourced user trajectories. IEEE Transactions on Mobile Computing 20, 2 (2021), 488–501.
[33]
Shaifali Parashar, Mathieu Salzmann, and Pascal Fua. 2020. Local non-rigid structure-from-motion from diffeomorphic mappings. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 2059–2067.
[34]
Milan D. Redžić, Christos Laoudias, and Ioannis Kyriakides. 2020. Image and WLAN bimodal integration for indoor user localization. IEEE Transactions on Mobile Computing 19, 5 (2020), 1109–1122.
[35]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS’15). 91–99.
[36]
Kari Sentz and Scott Ferson2002. Combination of Evidence in Dempster-Shafer Theory. Vol. 4015. Sandia National Laboratories, Albuquerque, NM.
[37]
Xingfa Shen, Chuang Li, Weijie Chen, Yongcai Wang, and Quanbo Ge. 2022. Transition model-driven unsupervised localization framework based on crowd-sensed trajectory data. ACM Transactions on Sensor Networks 18, 2 (Jan. 2022), Article 26, 21 pages.
[38]
Xiaoqiang Teng, Deke Guo, Yulan Guo, Xiang Zhao, and Zhong Liu. 2018. SISE: Self-updating of indoor semantic floorplans for general entities. IEEE Transactions on Mobile Computing 17, 11 (2018), 2646–2659.
[39]
Liang Wang, Dingqi Yang, Zhiwen Yu, Qi Han, En Wang, Kuang Zhou, and Bin Guo. 2023. Acceptance-aware mobile crowdsourcing worker recruitment in social networks. IEEE Transactions on Mobile Computing 22, 2 (2023), 634–646.
[40]
Tao Wang, Xiaopeng Zhang, Li Yuan, and Jiashi Feng. 2019. Few-shot adaptive faster R-CNN. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 7173–7182.
[41]
Xingkui Wei, Yinda Zhang, Zhuwen Li, Yanwei Fu, and Xiangyang Xue. 2020. DeepSFM: Structure from motion via deep bundle adjustment. In Computer Vision—ECCV 2020. Lecture Notes in Computer Science, Vol. 12346. Springer, 230–247.
[42]
Hang Wu, Jiajie Tan, and S.-H. Gary Chan. 2022. Pedometer-free geomagnetic fingerprinting with casual walking speed. ACM Transactions on Sensor Networks 18, 1 (Oct. 2022), Article 8, 21 pages.
[43]
Han Xu, Zheng Yang, Zimu Zhou, Longfei Shangguan, Ke Yi, and Yunhao Liu. 2016. Indoor localization via multi-modal sensing on smartphones. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, New York, NY, 208–219.
[44]
Jingao Xu, Erqun Dong, Qiang Ma, Chenshu Wu, and Zheng Yang. 2021. Smartphone-based indoor visual navigation with leader-follower mode. ACM Transactions on Sensor Networks 17, 2 (May2021), 22 pages.
[45]
Yuri D. V. Yasuda, Luiz Eduardo G. Martins, and Fabio A. M. Cappabianco. 2020. Autonomous visual navigation for mobile robots: A systematic literature review. ACM Computing Surveys 53, 1 (Feb. 2020), Article 13, 34 pages.
[46]
Lotfi A. Zadeh. 1986. A simple view of the Dempster-Shafer theory of evidence and its implication for the rule of combination. AI Magazine 7, 2 (1986), 85.
[47]
Dian Zhang, Wen Xie, Zexiong Liao, Wenzhan Zhu, Landu Jiang, and Yongpan Zou. 2022. Beyond RSS: A PRR and SNR aided localization system for transceiver-free target in sparse wireless networks. IEEE Transactions on Mobile Computing 21, 11 (2022), 3866–3879.
[48]
Yifan Zhang and Xinglin Zhang. 2021. Price learning-based incentive mechanism for mobile crowd sensing. ACM Transactions on Sensor Networks 17, 2 (June 2021), Article 17, 24 pages.
[49]
Yanchao Zhao, Jing Xu, Jie Wu, Jie Hao, and Hongyan Qian. 2020. Enhancing camera-based multimodal indoor localization with device-free movement measurement using WiFi. IEEE Internet of Things Journal 7, 2 (2020), 1024–1038.
[50]
Siwang Zhou, Yi Lian, Daibo Liu, Hongbo Jiang, Yonghe Liu, and Keqin Li. 2022. Compressive sensing based distributed data storage for mobile crowdsensing. ACM Transactions on Sensor Networks 18, 2 (Feb. 2022), Article 25, 21 pages.
[51]
Tongqing Zhou, Zhiping Cai, and Fang Liu. 2021. The crowd wisdom for location privacy of crowdsensing photos: Spear or shield? Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2 (Sept. 2021), Article 142, 23 pages.
Index Terms
- VILL: Toward Efficient and Automatic Visual Landmark Labeling
Recommendations
Weakly supervised landmark labeling in searched data
ICIMCS '10: Proceedings of the Second International Conference on Internet Multimedia Computing and ServiceMillions of place-specified photos are uploaded on the internet. Modeling and representing the landmark is very important for landmark retrieval and auto-annotation. In this paper, we aim at collecting images with a specific landmark and labeling the ...
Landmark image retrieval using visual synonyms
MM '10: Proceedings of the 18th ACM international conference on MultimediaIn this paper, we consider the incoherence problem of the visual words in bag-of-words vocabularies. Different from existing work, which performs assignment of words based solely on closeness in descriptor space, we focus on identifying pairs of ...
A visual attention-based approach for automatic landmark selection and recognition
WAPCV'04: Proceedings of the Second international conference on Attention and Performance in Computational VisionVisual attention refers to the ability of a vision system to rapidly detect visually salient locations in a given scene. On the other hand, the selection of robust visual landmarks of an environment represents a cornerstone of reliable vision-based ...
Comments
Information & Contributors
Information
Published In
November 2023
622 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Journal Family
Publication History
Published: 21 April 2023
Online AM: 15 February 2023
Accepted: 27 December 2022
Revised: 30 October 2022
Received: 11 March 2022
Published in TOSN Volume 19, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Natural Science Foundation of China
- Guangdong Basic and Applied Research Foundation
- Hong Kong General Research Fund
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 425Total Downloads
- Downloads (Last 12 months)243
- Downloads (Last 6 weeks)6
Reflects downloads up to 30 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format