short-paper

Open access

Demo Abstract: VGGlass - Demonstrating Visual Grounding and Localization Synergy with a LiDAR-enabled Smart-Glass

Authors:

Darshana Rathnayake,

Dulanga Weerakoon,

Meera Radhakrishnan,

Vigneshwaran Subbaraju,

Inseok Hwang,

Archan MisraAuthors Info & Claims

SenSys '23: Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems

Pages 492 - 493

https://doi.org/10.1145/3625687.3628407

Published: 26 April 2024 Publication History

PDF eReader

Abstract

This work demonstrates the VGGlass system, which simultaneously interprets human instructions for a target acquisition task and determines the precise 3D positions of both user and the target object. This is achieved by utilizing LiDARs mounted in the infrastructure and a smart glass device worn by the user. Key to our system is the union of LiDAR-based localization termed LiLOC and a multi-modal visual grounding approach termed RealG(2)In-Lite. To demonstrate the system, we use Intel RealSense L515 cameras and a Microsoft HoloLens 2, as the user devices. VGGlass is able to: a) track the user in real-time in a global coordinate system, and b) locate target objects referred by natural language and pointing gestures.

References

[1]

Darshana Rathnayake, Meeralakshmi Radhakrishnan, Inseok Hwang, and Archan Misra. 2023. LILOC: Enabling Precise 3D Localization in Dynamic Indoor Environments using LiDARs. In Proceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation. 158--171.

Digital Library

Google Scholar

[2]

Dulanga Weerakoon, Vigneshwaran Subbaraju, Tuan Tran, and Archan Misra. 2022. COSM2IC: Optimizing real-time multi-modal instruction comprehension. 7, 4 (2022), 10697--10704.

Google Scholar

[3]

Yiyi Zhou, Rongrong Ji, Gen Luo, Xiaoshuai Sun, Jinsong Su, Xinghao Ding, Chia-Wen Lin, and Qi Tian. 2021. A real-time global inference network for one-stage referring expression comprehension. IEEE Transactions on Neural Networks and Learning Systems (2021).

Crossref

Google Scholar

Recommendations

LILOC: Leveraging LiDARs for Accurate 3D Localization in Dynamic Indoor Environments
We present LiLoc, a system for precise 3D localization and tracking of mobile IoT devices (e.g., robots) in indoor environments using multi-perspective LiDAR sensing. LiLoc stands out with two key differentiators. First, unlike traditional localization ...
On the hybrid aid-localization for outdoor augmented reality applications
VRST '08: Proceedings of the 2008 ACM symposium on Virtual reality software and technology

In mobile outdoor augmented reality applications, accurate localization is critical to register virtual augmentations over a real scene. Vision-based approaches provide accurate localization estimates but are still too sensitive to outdoor conditions (...
Accelerating vision-based 3D indoor localization by distributing image processing over space and time
VRST '14: Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology

In a vision-based 3D indoor localization system, conducting localization of user's device at a high frame rate is important to support real-time augment reality applications. However, vision-based 3D localization typically involves 2D keypoint detection ...

Comments

Information & Contributors

Information

Published In

SenSys '23: Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems

November 2023

574 pages

ISBN:9798400704147

DOI:10.1145/3625687

General Chair:
Rasit Eskicioglu,
Program Chair:
Polly Huang,
Program Co-chair:
Neal Patwari

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 April 2024

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

NRF Investigatorship Grant

Conference

SenSys '23

Sponsor:

SenSys '23: 21st ACM Conference on Embedded Networked Sensor Systems

November 12 - 17, 2023

Istanbul, Turkiye

Acceptance Rates

Overall Acceptance Rate 198 of 990 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
129
Total Downloads

Downloads (Last 12 months)129
Downloads (Last 6 weeks)21

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Recommendations

LILOC: Leveraging LiDARs for Accurate 3D Localization in Dynamic Indoor Environments

On the hybrid aid-localization for outdoor augmented reality applications

Accelerating vision-based 3D indoor localization by distributing image processing over space and time