research-article

Public Access

Hybrid Indexes for Spatial-Visual Search

Authors:

Abdullah Alfarrarjeh,

Seon Ho KimAuthors Info & Claims

Thematic Workshops '17: Proceedings of the on Thematic Workshops of ACM Multimedia 2017

Pages 75 - 83

https://doi.org/10.1145/3126686.3126763

Published: 23 October 2017 Publication History

Abstract

Due to the growth of geo-tagged images, recent web and mobile applications provide search capabilities for images that are similar to a given query image and simultaneously within a given geographical area. In this paper, we focus on designing index structures to expedite these spatial-visual searches. We start by baseline indexes that are straightforward extensions of the current popular spatial (R*-tree) and visual (LSH) index structures. Subsequently, we propose hybrid index structures that evaluate both spatial and visual features in tandem. A unique challenge of spatial-visual search is that there are inaccuracies in both spatial and visual features. Therefore, different traversals in the same index structures may produce different images as output, some of which are more relevant to the query than the others. We compare our hybrid structures with a set of baseline indexes in both performance and result accuracy using three real world datasets from Flickr, Google Street View, GeoUGV, and a large synthetic dataset. Our comprehensive experimental results demonstrate that our proposed hybrid indexes significantly outperform baselines.

References

[1]

J. Sivic, and A. Zisserman. "Video Google: A text retrieval approach to object matching in videos." In ICCV, pp. 1470--1477. IEEE, 2003.

Digital Library

[2]

D. Lowe. "Distinctive image features from scale-invariant keypoints." In IJCV, no. 2 (2004): 91--110.

Digital Library

[3]

M. Nixon and A. Aguado. "Feature extraction & image processing for computer vision." Academic Press 2012.

Digital Library

[4]

L. Chen, G. Cong, C. S. Jensen, and D. Wu. "Spatial keyword query processing: an experimental evaluation." In VLDB, vol. 6, pp. 217--228. IEEE, 2013.

Digital Library

[5]

A. Guttman. R-trees: "a dynamic index structure for spatial searching." In SIGMOD, Vol. 14, no. 2. ACM, 1984.

Digital Library

[6]

N. Beckmann, H. Kriegel, R. Schneider, and B. Seeger. "The R*-tree: an efficient and robust access method for points and rectangles." In SIGMOD, Vol. 19, no. 2. ACM, 1990.

Digital Library

[7]

How many public photos are uploaded to Flickr every day, month, year? https://www.flickr.com/photos/franckmichel/6855169886/

[8]

P. Zhao, X. Kuang, V. Sheng, J. Xu, J. Wu, and Z. Cui. "Scalable Top-k Spatial Image Search on Road Networks." In DASFAA, pp. 379--396. Springer, 2015.

[9]

L. Dechao, M. Scott, R. Ji, W. Jiang, H. Yao, and X. Xie. "Location sensitive indexing for image-based advertising." In ACM MM, pp. 793--796. ACM, 2009.

Digital Library

[10]

I. Felipe, V. Hristidis, and N. Rishe. "Keyword search on spatial databases." In ICDE, pp. 656--665. IEEE, 2008.

Digital Library

[11]

Y. Zhou, X. Xie, C. Wang, Y. Gong, and W. Ma. "Hybrid index structures for location-based web search." In CIKM, pp. 155--162. ACM, 2005.

Digital Library

[12]

Y. Lu, C. Shahabi, and S. H. Kim. "Efficient indexing and retrieval of large-scale geo-tagged video databases." In GeoInformatica, Vol. 20(4), pp. 829--857. Springer 2016.

Digital Library

[13]

A. Khodaei, C. Shahabi, and C. Li. "Hybrid indexing and seamless ranking of spatial and textual features of web documents." In DEXA pp. 450--466. Springer, 2010.

Digital Library

[14]

J. Wan, D. Wang, S. Hoi, P. Wu, J. Zhu, Y. Zhang, and J. Li. "Deep learning for content-based image retrieval: A comprehensive study." In MM, pp. 157--166. ACM, 2014.

Digital Library

[15]

P. Ciaccia, M. Patella, and P. Zezula. "M-Tree: an Efficient Access Method for Similarity Search in Metric Spaces." In VLDB, vol. 23, p. 426. IEEE, 1997.

Digital Library

[16]

A. Krizhevsky, I. Sutskever, and G. Hinton. "ImageNet classification with deep convolutional neural networks." In NIPS, pp. 1097--1105. 2012.

Digital Library

[17]

S. A. Ay, R. Zimmermann, and S. H. Kim. "Viewable scene modeling for geospatial video search." In MM, pp. 309--318. ACM, 2008.

Digital Library

[18]

A. Babenko, A. Slesarev, A. Chigorin, and V. Lempitsky. "Neural codes for image retrieval." In ECCV, pp. 584--599. Springer 2014.

[19]

A. Andoni and P. Indyk. "Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions." In ACM Comm., vol 51(1), pp. 117--122. ACM 2008.

Digital Library

[20]

100M geotagged photos (Plus). http://code.flickr.net/2009/02/04/100000000-geotagged-photos-plus/ .

[21]

S. Vaid, C. Jones, H. Joho, and M. Sanderson. "Spatio-textual indexing for geographical search on the web." In SSTD, pp. 218--235. Springer, 2005.

Digital Library

[22]

N. Morsillo, G. Mann, and C. Pal. "Youtube scale, large vocabulary video annotation." In Video Search and Mining, pp. 357--386. Springer, 2010.

[23]

D. Nister and H. Stewenius. "Scalable recognition with a vocabulary tree." In CVPR, vol. 2, pp. 2161--2168. IEEE, 2006.

Digital Library

[24]

M. Datar, N. Immorlica, P. Indyk, and V. Mirrokni. "Locality-sensitive hashing scheme based on p-stable." distributions. In SoCG, pp 253--262. ACM 2004.

Digital Library

[25]

R. Zamir and M. Shah. "Image geo-localization based on multiple nearest neighbor feature matching using generalized graphs." In TPAMI, 36(8), pp. 1546--1558. IEEE 2014.

Digital Library

[26]

Y. Lu, H. To, A. Alfarrarjeh, S. Kim, Y. Yin, R. Zimmermann, and C. Shahabi. "GeoUGV: user-generated mobile video dataset with fine granularity spatial metadata." In MMSys, p. 43. ACM, 2016.

Digital Library

[27]

B. Thomee, D. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L. Li. "YFCC100M: The new data in multimedia research." In ACM Comm., 59(2), pp. 64--73. 2016

Digital Library

[28]

G. Amato, F. Falchi, C. Gennaro, and F. Rabitti. "YFCC100M HybridNet fc6 Deep Features for Content-Based Image Retrieval." In ACM Workshop on Multimedia COMMONS, pp. 11--18. ACM, 2016.

Digital Library

[29]

Sample Size Table, http://www.research-advisors.com/tools/SampleSize.htm.

[30]

Google Image Search. https://images.google.com/.

[31]

S. Arya and D. Mount. "Approximate nearest neighbor queries in fixed dimensions." In SODA, vol. 93, pp. 271--280. 1993

Digital Library

[32]

P. Indyk and R. Motwani. "Approximate nearest neighbors: Towards removing the curse of dimensionality." In STOC, pp. 604--613. ACM 1998.

Digital Library

[33]

Flow Mobile App. http://www.a9.com/whatwedo/mobile-technology/flow-powered-by-amazon/

[34]

Flickr's Photo API. https://www.flickr.com/services/api/flickr.photos.search.html

[35]

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. "Caffe: Convolutional architecture for fast feature embedding." In MM, pp. 675--678. ACM, 2014.

Digital Library

[36]

Yandex. https://www.yandex.com/.

[37]

R. Fagin, A. Lotem, and M. Naor. "Optimal aggregation algorithms for middleware." In J COMPUT SYST SCI 66, no. 4 (2003): 614--656.

Digital Library

Cited By

Zhang ZZhou FHou R(2024)Privacy-preserving geo-tagged image search in edge–cloud computing for IoTJournal of Information Security and Applications10.1016/j.jisa.2024.10380884(103808)Online publication date: Aug-2024
https://doi.org/10.1016/j.jisa.2024.103808
Alfarrarjeh AYang XJabal AKim SShahabi C(2021)Exploring the Spatial-Visual Locality of Geo-tagged Urban Street Images2021 IEEE 4th International Conference on Multimedia Information Processing and Retrieval (MIPR)10.1109/MIPR51284.2021.00023(104-110)Online publication date: Sep-2021
https://doi.org/10.1109/MIPR51284.2021.00023
Omarov BKizdarbekova MOmarov BOmarov N(2021)Development of an Automatic Road Damage Detection System to Ensure the Safety of TouristsAdvanced Informatics for Computing Research10.1007/978-981-16-3660-8_38(404-413)Online publication date: 20-Jun-2021
https://doi.org/10.1007/978-981-16-3660-8_38
Show More Cited By

Index Terms

Hybrid Indexes for Spatial-Visual Search
1. Information systems
  1. Data management systems
    1. Data structures
      1. Data access methods
  2. Information systems applications
    1. Multimedia information systems
      1. Multimedia databases

Recommendations

Adaptive Hybrid Indexes
SIGMOD '22: Proceedings of the 2022 International Conference on Management of Data

While index structures are crucial components in high-performance query processing systems, they occupy a large fraction of the available memory. Recently-proposed compact indexes reduce this space overhead and thus speed up queries by allowing the ...
Reducing the Storage Overhead of Main-Memory OLTP Databases with Hybrid Indexes
SIGMOD '16: Proceedings of the 2016 International Conference on Management of Data

Using indexes for query execution is crucial for achieving high performance in modern on-line transaction processing databases. For a main-memory database, however, these indexes consume a large fraction of the total memory available and are thus a major ...
A Hybrid BitFunnel and Partitioned Elias-Fano Inverted Index
WWW '19: The World Wide Web Conference

Search engines encounter a time vs. space trade-off: search responsiveness (i.e., a short query response time) comes at the cost of increased index storage. We propose a hybrid method which uses both (a) the recently published mapping-matrix-style index ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

Thematic Workshops '17: Proceedings of the on Thematic Workshops of ACM Multimedia 2017

October 2017

558 pages

ISBN:9781450354165

DOI:10.1145/3126686

Program Chairs:
Wanmin Wu
Google, USA
,
Jianchao Yang
Snap Inc., USA
,
Qi Tian
The University of Texas at San Antonio, USA
,
Roger Zimmermann
National University of Singapore, Singapore

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

MM '17

Sponsor:

SIGMM

MM '17: ACM Multimedia Conference

October 23 - 27, 2017

California, Mountain View, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
284
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)8

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang ZZhou FHou R(2024)Privacy-preserving geo-tagged image search in edge–cloud computing for IoTJournal of Information Security and Applications10.1016/j.jisa.2024.10380884(103808)Online publication date: Aug-2024
https://doi.org/10.1016/j.jisa.2024.103808
Alfarrarjeh AYang XJabal AKim SShahabi C(2021)Exploring the Spatial-Visual Locality of Geo-tagged Urban Street Images2021 IEEE 4th International Conference on Multimedia Information Processing and Retrieval (MIPR)10.1109/MIPR51284.2021.00023(104-110)Online publication date: Sep-2021
https://doi.org/10.1109/MIPR51284.2021.00023
Omarov BKizdarbekova MOmarov BOmarov N(2021)Development of an Automatic Road Damage Detection System to Ensure the Safety of TouristsAdvanced Informatics for Computing Research10.1007/978-981-16-3660-8_38(404-413)Online publication date: 20-Jun-2021
https://doi.org/10.1007/978-981-16-3660-8_38
Alfarrarjeh AYoon JKim SAbu Jabal ANagaraj ASiddaramaiah C(2021)An Interactive Video Search Tool: A Case Study Using the V3C1 DatasetMultiMedia Modeling10.1007/978-3-030-67835-7_43(448-454)Online publication date: 21-Jan-2021
https://doi.org/10.1007/978-3-030-67835-7_43
Alfarrarjeh AKim SHegde VAkshansh Shahabi CXie QRavada S(2020)A Class of R*-tree Indexes for Spatial-Visual Search of Geo-tagged Street Images2020 IEEE 36th International Conference on Data Engineering (ICDE)10.1109/ICDE48307.2020.00221(1990-1993)Online publication date: Apr-2020
https://doi.org/10.1109/ICDE48307.2020.00221
Hegde VTrivedi DAlfarrarjeh ADeepak AHo Kim SShahabi C(2020)Yet Another Deep Learning Approach for Road Damage Detection using Ensemble Learning2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9377833(5553-5558)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9377833
Alfarrarjeh ATrivedi DKim SPark HHuang CShahabi C(2019)Recognizing Material of a Covered Object: A Case Study With Graffiti2019 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP.2019.8803286(2491-2495)Online publication date: Sep-2019
https://doi.org/10.1109/ICIP.2019.8803286
Kim SAlfarrarjeh AConstantinou GShahabi C(2019)TVDP: Translational Visual Data Platform for Smart Cities2019 IEEE 35th International Conference on Data Engineering Workshops (ICDEW)10.1109/ICDEW.2019.00-36(45-52)Online publication date: Apr-2019
https://doi.org/10.1109/ICDEW.2019.00-36
Constantinou GSankar Ramachandran GAlfarrarjeh AKim SKrishnamachari BShahabi C(2019)A Crowd-Based Image Learning Framework using Edge Computing for Smart City Applications2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM)10.1109/BigMM.2019.00-47(11-20)Online publication date: Sep-2019
https://doi.org/10.1109/BigMM.2019.00-47
Alfarrarjeh AKim SBright AHegde VAkshansh AShahabi C(2019)Spatial Aggregation of Visual Features for Image Data Search in a Large Geo-Tagged Image Dataset2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM)10.1109/BigMM.2019.00-43(48-57)Online publication date: Sep-2019
https://doi.org/10.1109/BigMM.2019.00-43
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents