Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Gesellschaft für Informatik e.V.

Lecture Notes in Informatics


Datenbanksysteme für Business, Technologie und Web (BTW) P-214, 165-184 (2013).

Gesellschaft für Informatik, Bonn
2013


Copyright © Gesellschaft für Informatik, Bonn

Contents

Duplicate detection on gpus

Benedikt Forchhammer , Thorsten Papenbrock , Thomas Stening , Sven Viehmeier , Uwe Draisbach and Felix Naumann

Abstract


With the ever increasing volume of data and the ability to integrate different data sources, data quality problems abound. Duplicate detection, as an integral part of data cleansing, is essential in modern information systems. We present a complete duplicate detection workflow that utilizes the capabilities of modern graphics processing units (GPUs) to increase the efficiency of finding duplicates in very large datasets. Our solution covers several well-known algorithms for pair selection, attribute-wise similarity comparison, record-wise similarity aggregation, and clustering. We redesigned these algorithms to run memory-efficiently and in parallel on the GPU. Our experiments demonstrate that the GPU-based workflow is able to outperform a CPU-based implementation on large, real-world datasets. For instance, the GPU-based algorithm deduplicates a dataset with 1.8m entities 10 times faster than a common CPU-based algorithm using comparably priced hardware.


Full Text: PDF

Gesellschaft für Informatik, Bonn
ISBN 978-3-88579-608-4


Last changed 04.10.2013 18:38:50