Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
72 views

BigData Tutorial Part3

The document discusses large-scale linked data management using Apache Cassandra. It presents CumulusRDF, which uses Cassandra for distributed storage and indexing of RDF triples. CumulusRDF supports triple pattern lookups and linked data lookups through different index layouts in Cassandra. An evaluation loads 120M DBpedia triples into CumulusRDF and tests triple pattern and linked data queries, finding the flat index layout performs best overall. The results suggest Cassandra is viable for linked data lookups compared to triple stores.

Uploaded by

srk_uk
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
72 views

BigData Tutorial Part3

The document discusses large-scale linked data management using Apache Cassandra. It presents CumulusRDF, which uses Cassandra for distributed storage and indexing of RDF triples. CumulusRDF supports triple pattern lookups and linked data lookups through different index layouts in Cassandra. An evaluation loads 120M DBpedia triples into CumulusRDF and tests triple pattern and linked data queries, finding the flat index layout performs best overall. The results suggest Cassandra is viable for linked data lookups compared to triple stores.

Uploaded by

srk_uk
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 23

Large-scale Linked Data Management

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman


Big Linked Data Tutorial Semantic Days 2012

Large-Scale Linked Data Management (Andreas)


Motivation
Preliminaries
Apache Cassandra
CumulusRDF

Storage Layouts
Storage Model
Hierarchical Layout
Flat Layout

Evaluation
Conclusion

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

MOTIVATION

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Linked Data Storage and Retrieval at Scale


Size

Batch (Map-Reduce)
TB

CumulusRDF
(Apache Cassandra)

BigData,
4Store,
YARS2
Jena TBD
Sesame

Distributed
GB
Single
machine
MB

CloudSPARQL

Runtime

Index Lookups

Redland
Jena Mem

SPARQL

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

SAOR

OWLIM
Pellet, HermiT

Reasoning

Algorithmic
complexity

Linked Data Management


RDF data accessible via HTTP lookups
Many datasets cover descriptions of millions of entities
Publishers often use full-fledged triple stores
Complex query processing capabilities not necessary for Linked
Data lookups

Trend towards specialized data management systems


tailored for specific use cases
Distributed key-value stores
Simple (often nested) data model
No (expensive) joins
High availability and scalability

We investigate applicability of key-value stores for


managing and publishing Linked Data
Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Linked Data Lookups


Dereferencing URI t should return RDF graph describing t
Exact content is only lightly specified

Common practice (e.g. DBpedia) is to return


all triples with the given URI as subject and
some triples with the given URI as object

Other options
Only triples with the given URI as subject
Concise Bounded Descriptions
User Agent
http://www.bbc.co.uk/music/
artists/191cba6a-b83f-49ca883c-02b20c7a9dd5#artist

G
E
T

R
D
F

Server

http://www.bbc.co.uk/music/artists/191c
ba6a-b83f-49ca-883c-02b20c7a9dd5.rdf

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Triple Patterns
A triple pattern is an RDF triple that may contain variables
instead of RDF terms in any position
?s dbpprop:birthPlace dbpedia:Karlsruhe .

or
?s foaf:name ?o .

Linked Data Lookup on t translates into


two triple patterns lookups
(t ? ?)
(? ? t)

At least three indexes to cover all possible


triple patterns (with prefix lookups)

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Patterns Index
???

Any

s??

SPO

?p?

POS

??o

OSP

sp?

SPO

?po

POS

s?o

OSP

spo

Any

Apache Cassandra
Open source data
management system
Distributed key-value store (DHT-based)
Nested key-value data model
Schema-less

Decentralized
Every node in the cluster has the same role
No single point of failure

Elastic
Throughput increases linearly
as machines are added with no downtime

Fault-tolerant
Data can be replicated
Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

CumulusRDF

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

CumulusRDF Functionality
Distributed deployment to enable scale (more data and
also more clients) by adding more machines (via
Cassandra)
Geographical replication (via Cassandra)
Write-optimised indices with eventual consistency (via
Cassandra)
Triple pattern lookups (via CumulusRDF index structures)
Linked Data Lookups (via CumulusRDF index structures)
Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

STORAGE LAYOUTS

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Nested Key-Value Storage Model


Columns

Column-only { row-key : { column : value } }

ro

c00

v00

c01

v01

...

...

r1

c10

v10

c11

v11

...

...

Column key

...

Column value

Row
Super column key
sc00
r2

sc01

c000

v000

c010

v010

...

...

sc00
r3

sc01

c000

v000

c010

v010

...

...

Super columns { row-key : { supercolumn : { column : value } } }


Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Nested Key-Value Storage Model


Secondary indexes map column values to rows
{ value : row-key }

Cassandra limitations
Entire rows always stored on a single node
No range queries on row keys
Columns are stored in specified order and allow for range queries

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Hierarchical Layout
Uses super columns
RDF terms occupy row, supercolumn and column positions
Value is empty

Three indexes SPO, POS, OSP cover all possible triple


pattern
Example: SPO index
SPO: { s : { p : { o : - } } }
foaf:name
dbp:Jaws

Row key

rdf:type
Super column key

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Jaws

dbp:Film

dbp:Work

Column key

Value

Flat Layout
Uses columns only
Range queries on column keys allow prefix lookups

Concatenate second & third position to form column key


SPO { s : { po : - } }
po is the concatenation of predicate and object
For (sp?) we perform a prefix lookup on p in row with key s

dbp:Jaws

Row key

foaf:name Jaws

rdf:type dbp:Film

rdf:type dbp:Work

Column key

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Value

POS Index
RDF data is skewed: many triples may share the same predicate
(rdf:type is a prime example)
p as row key will result in a very uneven distribution
Cassandra cannot split rows among several nodes

We take advantage of Cassandras secondary indexes


Use po as row key
{ po : { s : - } }
Smaller rows, better distribution
No range queries on rows key: no prefix lookup!

In each row we add a special column p which has p as its value


{ po : { p : p } }

Secondary index on column p allows retrieval of all po row keys for a


given p

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

EVALUATION

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Evaluation
Clients

System: 4 node cluster on virtualized


infrastructure

CumulusRDF

2 CPUs, 4GB RAM, 80GB disk per node

Dataset: DBpedia 3.6 subset


120M triples (all w/o multilingual labels)

Triple pattern queries

C0

C1

C2

1M sampled S, SP, SPO, SO, and O patterns from dataset


Output: all matching triples

C3
C0-C3:
Cassandra
nodes

Linked Data lookup queries


2M resource lookups from DBpedia logs (1.2M unique)
Output: all triples with URI as subject and 10k triples with URI as
object
10k
Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

all

Results Storage Layout


Index

Node 1

Node 2

Node 3

Node 4

Std. Dev.

Max.
Row

SPO Hier

4.41

4.40

4.41

4.41

0.01

0.0002

SPO Flat

4.36

4.36

4.36

4.36

0.00

0.0004

OSP Hier

5.86

6.00

5.75

6.96

0.56

1.16

OSP Flat

5.66

5.77

5.54

6.61

0.49

0.96

POS Hier

4.43

3.68

4.69

1.08

1.65

2.40

POS Sec

7.35

7.43

7.38

8.05

0.33

0.56

Values in GB
SPO Flat: { s : { po : - } }, OSP
POS Sec: { po : { p : p } }
SPO Hier: { s : { p : { o : - } } }, OSP, POS
Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Results Pattern Lookups

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Results Pattern Lookups

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Results Linked Data Lookups

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

Conclusion
We evaluated two index schemes for RDF on nested keyvalue stores to support Linked Data lookups
Flat indexing gives best overall results
Output format impacts performance (N-Triples v RDF/XML)

Apache Cassandra is a viable alternative to full-fledged


triple stores for Linked Data lookups
Future work
Automatic generation and maintenance of dataset statistics
Evaluate insert and update performance

Get CumulusRDF at http://code.google.com/p/cumulusrdf

Marko Grobelnik, Andreas Harth (Gnter Ladwig), Dumitru Roman, Big Linked Data

You might also like