Workshop Dubai Cut

Storage and Management of Large
Scale Point Cloud Data

Oracle Workshop
9th International 3DGeoInfo 2014
Hans Viehmann1, Mike Horhammer1 ,
Peter van Oosterom2 and Oscar Martinez Rubi3
1 Oracle 2 TU Delft 3 Netherlands eScience Center
November 11, 2014
Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Image by Fugro
Program Agenda
1
Introduction
File formats and tested datasets
Point cloud data management systems
Loading
Querying
Research and others
Conclusions
Program Agenda
1
Introduction
Loading
Querying
Research and others
Conclusions
Introduction
2-year Netherlands eScience research project on Massive Point Clouds
1.
2.
3.
4.
5.
Image courtesy of: PDOK, NL
TU Delft:
GIS technology
TU Delft Library
3TU.Datacentrum
TU Delft Shared Service Center ICT
Cooperation with CWI/MonetDB
NLeSC (Netherlands eScience Center)

Oracle Corporation (NEDC)
Rijkswaterstaat
Fugro
Introduction
Collection of point cloud data
grows rapidly
Many new applications now
economically viable
Relevant use cases in GIS:

Digital elevation model of terrains
3D models of urban environment
Flood risk management
Dike monitoring (NL)
Forest mapping
Requiring new techniques for

Managing massive datasets
Integrating various kinds of data
Other potential applications:

3D models of manufactured parts
Visualization, animation, rendering
Medical imaging
Industrial metrology
Introduction
Evaluation of user requirements
Based on structured interviews with:

Government community: RWS (Ministry)
Commercial community: Fugro (company)
Scientific community: TU Delft Library
Report at MPC public website http://pointclouds.nl

Basis for conceptual benchmark:
Tests for functionality
Classified by importance
Testing single-user environments to avoid load-generator complexities

Introduction
Point cloud data management/processing systems
Why use a database here?

Requires a spatially enabled database
Data integration with other sources

Online availability
Geo-referenced imagery, existing 3D structures, attributes,...
Fast access to arbitrary part of data set

For processing or visualization
General benefits of mature DBMS

Information lifecycle management data administration, tuning
Scaleability multi-user environment, multi-processor support, clustering, ...
Executing data-intense logic where the data resides
Possible 3D processing flows in Oracle 12c

LiDAR Files
Convert to
Geometries
Georaster
LiDAR loader
Query and Clip
SDO_PC tables
or flat table
Load point
cloud
Flat files
Point Tables
Generate TIN
Generate DEM
TIN tables
Query and Clip

Query and Clip
Calculate
Contour Lines
Convert to
Geometries
BB-Infrastruktur AG, R&D

Oracle Spatial and Graph: Railway Network Management
Objectives
Optimize railway planning, construction and
maintenance
Enable efficient routing of railway lines
Solution
[This technology] is indispensable to process

geospatial data with high efficiency at low
cost,
Dr. Michaela Haberler-Weber
BB-Infrastruktur AG, R & D
Stores and processes more than 8 billion
points of objects along railway tracks

Enables LiDAR data to be viewed with existing
infrastructure vector data
Provides comprehensive metadata about
railway tracks through CSW
Delivers data through open WebServices
Image courtesy of: IQsoft, Austria
Program Agenda
1
Introduction
Loading
Querying
Research and others
Conclusions

LAS
De facto standard
Current version 1.4
http://www.asprs.org/a/society/com
mittees/standards/LAS_1_4_r13.pdf
Header 375 bytes
12+ attributes
11 Record (point) formats
Binary (20+ bytes per point)
Limited to 4.2B pts (< 1.4)
ASPRS Standard LIDAR Point Classes

(Point Data Record Format 0-5)
LIDAR data schema evolution

LAS 1.0 (2003)
Value
Meaning
Created, Never Classified
Unclassified 1
Ground
Low Vegetation
Medium Vegetation
High Vegetation
Format 0, 1
Building
Format 2, 3: , R, G, B
Low Point (noise)
Model Key Point (mass point)
Water
Format 0 3
10
Reserved for ASPRS Def
Format 4, 5: (1, 3) , waveform packet info
11
12
Overlap Points 2
13-31
Format 0: X, Y, Z, Intensity, Return #, , classification,
Format 1: , GPS time
LAS 1.1 (2005)

Formats 0, 1: similar
LAS 1.2 (2008)
LAS 1.3 (2009)
LAS 1.4 (2011)

Formats 6 10: , near infrared channel, 64-Bit

LAS: compression and in-file indexing
Highly compressible
LAZ (LAStools)
zLAS (ESRI)
Compression rate 10-15
Indexable
LAX (LAStools)
LASX (ESRI)

Actual Height Model of the Netherlands (AHN2)
Covering surface of the entire country
6 -10 pts/m2 640 billion pts
60,185 LAZ files, 987 GB/11.64 TB
(X, Y, Z) only
Future
AHN3 higher resolution
Cyclorama-based photogrammetric
datasets (50x AHN2, and with RGB)

Tested datasets
Name
Points
LAS files
Disk size [GB]
Area [km2]
Description
20M
20,165,862
0.4
1.25
TU Delft campus
210M
210,631,597
16
4.0
11.25
Major part of Delft city
2201M
2,201,135,689
153
42.0
125
City of Delft and surroundings
23090M
23,090,482,455
1,492
440.4
2,000
Major part of Zuid-Holland province
639478M
639,478,217,460
60,185
11,644.4
40,000
The Netherlands

Benchmarks
Mini-benchmark
1 dataset: 20M
X, Y, Z
Management systems:
PointCloud (blocks) solutions in Oracle and
PostgreSQL
Flat tables in Oracle and PostgreSQL
LAStools (not only X,Y,Z)
7 queries

Benchmarks
Medium benchmark
Four datasets: 20M, 210M,
2201M, 23090M
X, Y, Z
Management system:
Same as mini-benchmark
Use compression
Add MonetDB column-store
(flat table)
Add Exadata (diff. Hardware)
20 queries

Benchmarks queries

Benchmarks
Full benchmark
AHN2 (640bn points)
X, Y, Z
Management systems:
Oracle EXADATA
Oracle blocks
PostgreSQL blocks
LAStools
MonetDB
30 queries

Hardware & Software
Server HP Proliant DL380p Generation8 (2U rack)
2 x 8-core Intel Xeon processors E5-2690, 2.9 GHz
128 GB RAM (DDR3)
Disk storage:
400 GB SATA SD device
300 GB SAS 10K rpm in RAID-1 configuration (internal, 2 disks)
6 TB SAS 15K rpm in RAID-5 configuration (internal, 22 disks)

2 x 41 TB SATA 7200 rpm in RAID-5 configuration (external in 4U rack 'Yotta-III' box, 24 disks)
Network: 4 x 1 Gbit Ethernet

Software:
Operating system: Red Hat Enterprise Linux, version 6.5
DBMS: Oracle, PostgreSQL/PostGIS, MonetDB
'Geo' tools and libraries: CGAL, GDAL, GEOS, LAStools, PROJ.4, GeoTIFF, NetCDF, etc., etc.

Hardware & Software
Oracle Exadata X4-2: Oracle SUN hardware for Oracle database software
DBMS counterpart of
GPU for graphics
Characteristics of Oracle Exadata Database Machine

Standard Database Servers
8x 2-socket servers 192 cores, 2TB DRAM
or
2x 8-socket servers 160 cores, 4TB DRAM
Unified Ultra-Fast Network
40 Gb InfiniBand internal connectivity
10 Gb or 1 Gb Ethernet data center connectivity
Scale-out Intelligent Storage Servers

14x 2-socket servers 168 cores in storage
Fully Redundant
168 SAS disk drives
672 TB HC or 200 TB HP
56 Flash PCI cards
44 TB Flash + compression
Oracle Exadata Key innovations

Database offload in storage
Data intensive query operations offloaded to
storage CPUs
100 GB/sec SQL data throughput
Storage Index data skipping
Database optimized compression
Hybrid Columnar for 10x DB size reduction
and faster analytics
Database optimized PCI Flash
Smart caching of database data
2.66 Million Database IOs/sec
Smart Flash log speeds transactions
Database optimized QoS

End-to-end prioritization from
application to DB and storage
Database optimized availability
Fastest recovery of failed database,
server, storage or switch
Fastest backup. Incremental offload
Exachk top-to-bottom validation of
hardware, software, settings
Database optimized messaging
SQL optimized InfiniBand protocol for
high throughput low latency SQL
Oracle Exadata for Spatial Workloads

Oracle Spatial and Graph is particularly suitable for the
Exadata architecture
CPU intensive geospatial calculations
excellent use of parallelism
load distribution between server nodes and storage heads
index operations benefiting from parallelism and large memory
Breaks new boundaries for ingesting spatial data

using the fast I/O and high network bandwidth
Specific optimizations for point cloud data

Excellent use of hybrid columnar compression (HCC)
Scalable from Eighth-Rack to Multi-Rack
Eighth
Quarter
Half
Full
Field Upgradeable - Supports

Multiple Generations of Hardware
Multi-Rack
Additional work package

Standardization
ISO/OGC spatial data:
at abstract/generic level, 2 types of spatial representations: features and coverages
at next level (ADT level), 2 types: vector and raster, but perhaps points clouds should be added
at implementation/ encoding level, many different formats
(for all three data types)
nD point cloud:
points in nD space and not per se limited to x,y,z
(n ordinates of point which may also have m attributes)
make fit in future ISO 19107 (as ISO 19107 is under revision).
note: nD point clouds are very generic;
e.g. also cover moving object point data: x,y,z,t (id) series.
Program Agenda
1
Introduction
Loading
Querying
Research and others
Conclusions
Point cloud data management systems (PC DMS)

Large scale point cloud data
Oracle
Blocks
Flat table
PostgreSQL
Blocks
Flat table
LASTools
MonetDB
PC DMS Oracle Blocks

Oracle blocks
Loading:
Querying:
Global loading:
CLIP_PC function
Incremental loading:
SDO_PC_NN (12.1.0.1)
Blocking methods:
R-Tree
CLIP_PC_GET_MIN_HEIGHT (12.1.0.1)
CREATE_CONTOUR_GEOMETRIES (12.1.0.1)
Hilbert R-Tree

Oracle blocks Schema
TABLE of SDO_PC_BLK type

OBJ_ID
Table Column:
SDO_POINT_CLOUD
PC BLOB:
n points
(x, y, z, ): 8-byte IEEE doubles (<= 8)
BLK_ID
BLK_EXTENT:
SDO_GEOMETRY
Storage:
Up to 30 Mpts/GB
Compressed LOB: 100 Mpts/GB
Blk_id: 4-byte big-endian integer

Pt_id: 4-byte big-endian integer
POINTS:
BLOB

Oracle blocks

Oracle blocks Querying
Query

Oracle blocks Overlapping Blocks, Example Hilbert R-tree
Query
PC DMS PostgreSQL Blocks

PostgreSQL blocks
Group points in blocks
Defined in 2D (x,y) space
Block type: pcpatch
PointCloud extension by P. Ramsey

https://github.com/pramsey/pointcloud
Requires PostGIS
Storage
Compression: 100 Mpts/GB
Compression+: 200 Mpts/GB
Loading:
Incremental loading
Require good input data spatial organization
Ordered loading (avoid de-clustering)
External tool PDAL (pipeline)

http://www.pdal.io
No parallel (doable with scripting, e.g. Python)
GIST (R-Tree) index for the blocks

Querying:
PC_Intersects and PC_Intersection
No native parallel

PostgreSQL blocks: PDAL chipper blocks
Images from http://www.pdal.io/stages/filters.chipper.html

PC DMS LAStools

LAStools
File-based solution by M. Isenburg
http://rapidlasso.com/lastools/
Partly commercial:
Some tools free open source
Some tools require license (Windows)

No license: noise, limit parallel
Directly on LAS/LAZ
Storage:
LAS: 53 Mpts/GB
LAZ: 540 Mpts/GB
No data loading, but data preparation

lastile, lassort, lasindex
Require good input data spatial organization

Multi-core support (not really in queries)
Wine in Linux for Windows-only tools
Hybrid solutions (DB) for massive PCs
Querying:
lasmerge for rectangles and circles
lasclip for polygons
Polygon holes ignored
PC DMS Oracle Flat

Oracle flat
Pro
Indexing
No blocking required fast creation
None on Exadata
Dynamic updates fast updates
B-tree on (x, y)
Flexible schema
IOT works well
Con
Queries less scalable on non-Exadata
(< 1 billion pts)
Storage: 23 Mpts/GB (200 Mpts/GB on Exadata)
Querying:
Plain old SQL
Range query for rectangles
Distance method for circles
SDO_PointInPolygon for polygons
PC DMS PostgreSQL Flat

PostgreSQL flat
Points in flat table
B-Tree index on X,Y
Row per point
GIST too expensive
Data type: Double precission
No parallel
Storage: 13 Mpts/GB
Querying:
No native parallel
Loading:
Las2txt + COPY FROM
Range query for rectangles
Faster NLeSC binary loader (convert LAS to PostgreSQL

binary dump format)
Distance method for circles
No parallel (doable with scripting, e.g. Python)
_ST_Contains,ST_MakePoint for polygons
Bad scaling: Only use for small datasets (< 0.5 . 109)
PC DMS MonetDB

MonetDB
Column store
Imprints indexing
Points in flat table
Querying:
Row per point

Data type: Double precission
Parallel
Active development
Storage: 40 Mpts/GB
Not released yet
Loading:
Bad scaling: Only use for small datasets (< 1 . 109)
Las2txt + COPY FROM

Faster NLeSC binary loader (convert LAS to MonetDB
columnar format)
Parallel

Storage models
Storage model
Pro
Con
DB blocks
Storage (compression)
Scaling
Indexing
DB functionalities
Complex queries
Loading (make blocks)

Block overhead in queries
(noticeable in simple queries)
DB flat
Faster loading/updating
DB functionalities
Dynamic schema
Simple queries
Storage (except Exadata)

Not scaling (except Exadata)
Indexing (except Exadata)
File-based
Storage (LAZ)
Data preparation
Simple queries (if not LAZ)
Limited functionalities
Fixed schema (LAS)
Scaling requires DB help
Program Agenda
1
Introduction
Loading
Querying
Research and others
Conclusions
Loading Oracle Blocks
Loading
Oracle Blocks Loading Process
CSV
LAS
LASzip
LAZ
SqlLoader & preprocessor (4)
Flat table
CREATE_PC (1)
SDO_PC
Loading
CSV
LAS
LASzip
LAZ
SqlLoader & preprocessor
Flat table
CREATE_PC
SDO_PC
Loading
Oracle Blocks (Hilbert blocking) Create tables
CREATE TABLE AHN_BLCK
TABLESPACE USERS
PCTFREE 0
NOLOGGING
LOB(POINTS)
STORE AS SECUREFILE (TABLESPACE USERS COMPRESS HIGH) AS
SELECT * FROM MDSYS.SDO_PC_BLK_TABLE WHERE 0 = 1;
CREATE TABLE AHN_BASE (
id number,
PC SDO_PC)
TABLESPACE USERS
PCTFREE 0
NOLOGGING;
Loading
Oracle Blocks (Hilbert blocking) Create PC
declare
pc sdo_pc;
begin
pc :=
sdo_pc_pkg.init(
basetable
basecol
blktable
ptn_params
pc_extent
sdo_pc_pkg.create_pc(
inp
=> pc,
inptable
=> 'INPTAB',
clstpcdatatbl => null);
=>
=>
=>
=>
=>
'PCS',
'PC',
'BLOCKS',
'blk_capacity=10000',
mdsys.sdo_geometry(2003, null, null,
mdsys.sdo_elem_info_array(1, 1003, 3),
mdsys.sdo_ordinate_array(289020.90, 4320942.61,
290106.02, 4323641.57)),
0.05,
3,
null,
null,
xmltype(
pc_tol
=>
pc_tot_dimensions =>
pc_domain
=>
pc_val_attr_tables =>
pc_other_attrs
=>
'<opc:sdoPcObjectMetadata
xmlns:opc="http://xmlns.oracle.com/spatial/vis3d/2011/sdovis3d.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
blockingMethod="Hilbert R-tree">
</opc:sdoPcObjectMetadata>'));
insert into pcs values (1, pc);

end;
/
Loading
Oracle Blocks (Hilbert blocking) Metadata, Index
Update the SRID of the blocks, drop the staging table and add the blocks index (SQL).
UPDATE AHN_BLCK B SET B.BLK_EXTENT.SDO_SRID = 28992;
DROP TABLE AHN_STAGING;
INSERT INTO USER_SDO_GEOM_METADATA VALUES (
AHN_BLCK,
BLK_EXTENT,
SDO_DIM_ARRAY(
SDO_DIM_ELEMENT(X, [min. X], [max. X],0.0001),
SDO_DIM_ELEMENT(Y, [min. Y], [max. Y],0.0001)),
28992);
CREATE INDEX AHN_BLCK_SIDX
ON AHN_BLCK (BLK_EXTENT)
INDEXTYPE IS MDSYS.SPATIAL_INDEX
PARAMETERS (TABLESPACE=INDX WORK_TABLESPACE=PCWORKLAYER_GTYPE=POLYGON SDO_INDX_DIMS=2
SDO_RTR_PCTFREE=0)
PARALLEL 16;
Loading
CSV
LAS
LASzip
LAZ
Flat table
CREATE_PC
SDO_PC
Loading
Oracle Blocks (Hilbert blocking) Incremental Loading
$JDKHOME/bin/java -Xms2048m -classpath //ojdbc6.jar://sdoutl.jar oracle.spatial.util.Las2SqlLdrIndep 1 BLOCKS BLOCK_ID_SEQ
//ahn_bench023090_01.las 10000 jdbc:oracle:thin:@//host:port/sid largepc largepc 100000000
(10K block capacity, 100M pts in memory)
Connecting to jdbc:oracle:thin:@//host:port/sid...
Reading
100,000,000 points...
23.324 s. Sorting:
65.362 s. Writing: 164.608 s
Reading
99,995,084 points...
28.793 s. Sorting:
63.231 s. Writing: 187.621 s
Reading
99,997,274 points...
38.275 s. Sorting: 138.074 s. Writing: 216.841 s
Reading
99,991,996 points...
Reading
99,990,636 points...
Reading
99,990,636 points...
Reading
99,990,637 points...
Reading
99,990,638 points...
Reading
99,990,639 points...
Reading
99,990,640 points...
Reading
99,990,641 points...
Reading
99,999,834 points...
Reading
99,990,124 points...
Reading
99,999,835 points...
Reading
99,990,145 points...
Reading
99,997,246 points...
Reading
99,992,320 points...
Reading
99,990,606 points...
Reading
99,990,607 points...
Reading
99,990,608 points...
Reading
139,854 points...
0.110 s. Sorting:
0.097 s. Writing:
0.375 s
-----------------------------------------------------------------------------------------Read
2,000,000,000 points... 1,078.165 s. Sorting: 2,854.376 s. Writing: 4,752.362 s
(~ sequential I/O networked drive)
Time elapsed......................... 8,708.131 s
(~ 2h)
Loading PostgreSQL Blocks
Loading
PostgreSQL blocks
Load extensions and create blocks table (SQL):
CREATE
CREATE
CREATE
CREATE
EXTENSION postgis;
EXTENSION pointcloud;
EXTENSION pointcloud_postgis;
TABLE patches (id SERIAL PRIMARY KEY, pa PCPATCH);
Requires XML with data format inserted in

pointcloud_formats
Attributes
Scales and offsets
Compression: none or dimensional
Multiple formats possible
INSERT INTO pointcloud_formats (pcid, srid, schema) VALUES (1,

SRID,
<?xml version="1.0" encoding="UTF-8"?>
<pc:PointCloudSchema xmlns:pc=
"http://pointcloud.org/schemas/PC/1.1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<pc:dimension>
<pc:position>1</pc:position>
<pc:size>4</pc:size>
<pc:description>X coordinate </pc:description>
<pc:name>X</pc:name>
<pc:interpretation>int32_t </pc:interpretation>
<pc:scale> 0.01 </pc:scale>
<pc:offset> 0 </pc:offset>
</pc:dimension>
<pc:dimension>
<pc:description>Y coordinate</pc:description>
<pc:name>Y</pc:name>
<pc:interpretation>int32_t</pc:interpretation>
</pc:dimension>
<pc:dimension>
<pc:description>Z coordinate</pc:description>
<pc:name>Z</pc:name>
<pc:interpretation>int32_t</pc:interpretation>
</pc:dimension>
<pc:metadata>
<Metadata name="compression">dimensional</Metadata>
</pc:metadata>
</pc:PointCloudSchema>);
Loading
PostgreSQL blocks
<?xml version="1.0" encoding="utf-8"?>
<Pipeline version="1.0">
<Writer type="drivers.pgpointcloud.writer">
<Option name="connection"> host=dbhost dbname=dbname password=dbpass
user=dbuser </Option>
<Option name="table">blocks</Option>
<Option name="srid">SRID</Option>
<Option name="overwrite">false</Option>
<Option name="pcid">1</Option>
<Filter type="filters.chipper">
<Option name="capacity">BlockSize</Option>
<Filter type="filters.cache">
<Option name="max_cache_blocks">1</Option>
<Filter type="filters.selector">
<Option name="keep">
<Options>
<Option name="dimension">X</Option>
<Option name="dimension">Y</Option>
<Option name="dimension">Z</Option>
</Options>
</Option>
<Option name="overwrite_existing_dimensions">false</Option>
GIST(geometry(pa));
<Reader type="drivers.las.reader">
<Option name="filename"> [input LAS file]</Option>
<Option name="spatialreference"> EPSG:SRID</Option>
</Reader>
</Filter>
</Filter>
</Filter>
</Writer>
</Pipeline>
PDAL (Command-line) with configuration XML:

pcid as in pointclouds_format
Block size: Chipper filter capacity (smaller than Oracle)

Selector filter to select attributes
Multiple PDAL processes possible
pdal pipeline [xml File]
Create a PostGIS GIST index (SQL):

CREATE INDEX pa_gix ON blocks USING
VACUUM and ANALYZE (SQL)

optional
VACUUM FULL ANALYZE blocks;
Loading LAStools
Loading
LAStools
Optional tiling (if bad spatial organization)
lastile i *.las o tile.las
Resort and index (Command-line):

lassort.exe *las
lasindex *las
Multi-core:
option cores
lassort.exe *las cores 8
lasindex *las cores 8
Scripting
lassort.exe -i [input LASfile] -o [output LASfile]
lasindex -i [output LASfile]
Loading
LAStools (Hybrid: PostgreSQL-PostGIS)
Create DB and table for LAS files extents (SQL).
CREATE EXTENSION postgis;

CREATE TABLE lasindex (filepath text, geom public.geometry(Geometry,28992));
Extract header information (lasinfo) and insert in DB:

Multiple processes possible
lasinfo [LAS file] -nc -nv -nco

INSERT INTO lasindex (filepath,geom) VALUES ([LAS file], ST_MakeEnvelope(minx,miny,maxx,maxy, SRID));
Create the GIST index on the extents (SQL).

CREATE INDEX ON lasindex USING GIST (geom);
VACUUM and ANALYZE (SQL)

optional
VACUUM FULL ANALYZE lasindex;
Loading Oracle Flat (Exadata)
Loading
Oracle flat (Exadata) Loading Process
CSV
LAS
LASzip
LAZ
Flat table
CREATE_PC
SDO_PC
Loading
Oracle flat (Exadata) SqlLoader from CSV / ASCII
connect sys as sysdba
create or replace directory las_dir as
'/scratch/';
CREATE TABLE lidar_ext (
val_d1 NUMBER,
val_d2 NUMBER,
val_d3 NUMBER,
d
NUMBER)
ORGANIZATION EXTERNAL (
TYPE ORACLE_LOADER
DEFAULT DIRECTORY las_dir
ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
FIELDS (
val_d1 CHAR(8),
val_d2 CHAR(8),
val_d3 CHAR(8),
d
CHAR(8)))
LOCATION (
'lidar1.txt',
'lidar2.txt',
'lidar3.txt'))
PARALLEL;
more lidar*txt
::::::::::::::
lidar1.txt
::::::::::::::
1
2
3
::::::::::::::
lidar2.txt
::::::::::::::
4
5
6
::::::::::::::
lidar3.txt
::::::::::::::
7
8
9
1
2
3
1
2
3
123
124
125
4
5
6
4
5
6
123
124
125
7
8
9
7
8
9
123
124
125
create table lidar as select * from

lidar_ext;
Loading
Oracle flat (Exadata) Loading Process
CSV
LAS
LASzip
LAZ
Flat table
CREATE_PC
SDO_PC
Loading
Oracle flat (Exadata) SqlLoader from LAS
create directory las_dir as ';
create directory exe_dir as ';
create directory log_dir as ';
grant read on directory las_dir to mdsys;
grant read, execute on directory exe_dir to
mdsys;
grant write on directory log_dir to mdsys;
preprocessor.sh
#!/bin/bash
$JDKHOME/bin/java -classpath /scratch/mhorhamm/sdoutl.jar
oracle.spatial.util.Las2SqlLdr $1
create table inptab_ext(

val_d1 number,
val_d2 number,
val_d3 number,
d
number)
organization external (
type oracle_loader
default directory las_dir
access parameters (
records delimited by newline
preprocessor exe_dir:'preprocessor.sh'
BADFILE log_dir:'preprocess.bad'
LOGFILE log_dir:'preprocess.log'
fields terminated by ",")
location (
'Serpent_Mound_Model_LAS_Data.las',
'Serpent_Mound_Model_LAS_Data2.las',
'Serpent_Mound_Model_LAS_Data3.las',
'Serpent_Mound_Model_LAS_Data4.las'))
parallel 16
reject limit unlimited;
Loading
Oracle flat (Exadata) Materialize table & Create Index (no for Exadata)
CREATE TABLE AHN_FLAT
TABLESPACE USERS
PCTFREE 0
NOLOGGING
PARALLEL 16 AS
SELECT
VAL_D1,
VAL_D2,
VAL_D3
FROM
INPTAB_EXT;
CREATE INDEX AHN_FLAT_IDX
ON AHN_FLAT (
VAL_D1,
VAL_D2)
TABLESPACE INDX
PCTFREE 0
NOLOGGING
PARALLEL 16;
Loading
Tuning
Tune DB server parameters (working memory, )
Use different file-system for DB and input data (faster IO)
If available use SSD disks for indexes (different table space)

> 40bn points consider partitions
Block sizes Oracle > PostgreSQL (also depend on size typical query regions)
Input data not spatial overlap Incremental loading
Compression: Better storage, slower queries and loading
Not native parallel easy do-it-your-own (be careful with de-clustering)
Loading
Medium benchmark comparison
PostgreSQL flat: Binary loader, not native parallel
16 by script, index not parallel
Oracle flat: SQL loader from LAS (ext. table, preprocessor script) , parallel 16
MonetDB flat: Binary loader, parallel
PostgreSQL blocks: Incremental loading with PDAL,

no native parallel 12 by script, blocksize 3000
Oracle blocks: Global loading, SQL loader from LAS,
Hilbert blocking, parallel 16, blocksize 10000
LAStools: (unlicensed), LAS files, no native parallel
16 by script , hybrid DB solution (PostgreSQL)
Oracle EXADATA X4-2 half rack: 96 cores, 5
compression levels, SQL loader from LAS, flat table,
no index
Loading
Medium benchmark comparison: Loading (inc. indexing) and storage of 23090M points
Loading [s]
Oracle blocks global
70000,00
Oracle blocks global compress
192612,07
Oracle blocks increm.
14000,00
Oracle blocks increm. compress
45000,00
PostgreSQL blocks increm. *
7000,00
PostgreSQL blocks increm. compress*
9079,57
Oracle flat
15825,53
PostgreSQL flat
95014,05
MonetDB
8490,90
LAStools**
1639,30
LAStools LAZ**
2377,30
Oracle Exadata (oe-qh)
928,84
Loading
Storage Storage
[Mpts/s]
[GB] [Mpts/GB]
0,33 700,00
32,99
0,12 220,10
104,91
1,65 700,00
32,99
0,51 220,10
104,91
3,30 200,00
115,45
2,54 106,80
216,20
1,46 997,10
23,16
0,24 1780,90
12,97
2,72 529,40
43,62
14,09 440,50
52,42
9,71
35,67
647,28
22,71
94,24
223,80
* Ordered loading
** Licensed version
Loading
Medium benchmark comparison: Loading (inc. indexing) of 23090M points
Loading [Mpts/s]
25,00
22,71
20,00
15,00
14,09
9,71
10,00
5,00
3,30
0,33
0,12
Oracle blocks
global
Oracle blocks
global compress
2,72
2,54
1,65
1,46
0,51
0,24
0,00
Oracle blocks
increm.
Oracle blocks PostgreSQL blocks PostgreSQL blocks

increm. compress
increm.
increm. compress
Oracle flat
PostgreSQL flat
MonetDB
LAStools
LAStools LAZ
Oracle Exadata
Loading
Medium benchmark comparison: Loading (inc. indexing) of 23090M points
Loading [Mpts/s]
4,00
3,50
3,30
3,00
2,72
2,54
2,50
2,00
1,65
1,46
1,50
1,00
0,51
0,50
0,33
0,24
0,12
0,00
Oracle blocks
global
Oracle blocks
global compress
Oracle blocks
increm.
Oracle blocks
PostgreSQL blocks PostgreSQL blocks
increm. compress
increm.
increm. compress
Oracle flat
PostgreSQL flat
MonetDB
LAStools
LAStools LAZ
Oracle Exadata
Loading
Medium benchmark comparison: Storage of 23090M points
Storage [Mpts/GB]
700,00
647,28
600,00
500,00
400,00
300,00
223,80
216,20
200,00
104,91
104,91
115,45
100,00
32,99
32,99
23,16
12,97
Oracle flat
PostgreSQL flat
43,62
52,42
MonetDB
LAStools
0,00
Oracle blocks
global
Oracle blocks
global compress
Oracle blocks
increm.
Oracle blocks PostgreSQL blocks PostgreSQL blocks

increm. compress
increm.
increm. compress
LAStools LAZ
Oracle Exadata
Program Agenda
1
Introduction
Loading
Querying
Research and others
Conclusions
Querying Oracle Blocks
Querying
Oracle blocks CLIP_PC (return blocks)
SQL> select
blk_id,
num_points
from
table(
sdo_pc_pkg.clip_pc(
inp
=> (select pc from pcs where id = 1),
ind_dim_qry
=> sdo_geometry(
2003,
null,
null,
sdo_elem_info_array(1, 1003, 3),
sdo_ordinate_array(10, 10, 14, 14)),
other_dim_qry => null,
qry_min_res
=> 1,
qry_max_res
=> 1));
BLK_ID NUM_POINTS
---------- ---------53
25
Querying
Oracle blocks CLIP_PC (return individual point records)
select
query_points.x,
query_points.y,
query_points.z
from
table(
sdo_pc_pkg.clip_pc(
inp
ind_dim_qry
=> sdo_geometry(
2003, null, null,
qry_min_res
=> 1,
qry_max_res
=> 1)) query_blocks,
table(
sdo_util.getvertices(
geometry => sdo_pc_pkg.to_geometry(
pts
=> query_blocks.points,
num_pts
=> query_blocks.num_points,
pc_tot_dim => 3,
srid
=> null))) query_points;
Querying
Oracle blocks CLIP_PC (query by non-indexed dim)
select
query_points.x,
query_points.y,
query_points.z,
query_points.w Intensity -- followed by v5 - v11
from
table(
sdo_pc_pkg.clip_pc(
inp
ind_dim_qry
=> sdo_geometry(
2003, null, null,
other_dim_qry => sdo_mbr(
lower_left => sdo_vpoint_type(123,
1),
upper_right => sdo_vpoint_type(123, 1000)),
qry_min_res
=> 1,
qry_max_res
table(
pts
num_pts
pc_tot_dim => 4,
srid
X
Y
Z INTENSITY
---------- ---------- ---------- ---------10
13
123
456
Querying
Oracle blocks CLIP_PC (support non-blocked dimensions)
select
query_points.x,
query_points.y,
query_points.z,
query_points.w Intensity, -- followed by v5 - v11
out.non_blocked_dim
from
table(
sdo_pc_pkg.clip_pc(
inp
ind_dim_qry
=> sdo_geometry(2003, null, null, sdo_elem_info_array(1, 1003, 3), sdo_ordinate_array(10, 10, 14, 14)),
other_dim_qry => sdo_mbr(
lower_left => sdo_vpoint_type(123,
1),
upper_right => sdo_vpoint_type(123, 1000)),
qry_min_res
=> 1,
qry_max_res
table(
pts
num_pts
pc_tot_dim => 4,
srid
=> null ,
get_ids
=> 1))) query_points ,
pcs_out out
where
out.ptn_id
= query_points.v5 and
-- v5 is blk_id, v6 is pt_id
out.point_id = query_points.v6;
X
Y
Z INTENSITY NON_BLOCKED_DIM
---------- ---------- ---------- ---------- -------------------------------11
10
123
456 # 21
Querying
Oracle blocks SDO_PC_NN
SQL> select
2
rownum pt_pos,
3
sdo_geometry(
4
3001,
5
null,
6
sdo_point_type(x, y, z),
10
null,
11
null) pts
12 from
13
table(
14
15
16
pts => sdo_pc_pkg.sdo_pc_nn(
17
pc
=> (select pc from pcs where pc_id = 1),
18
center => sdo_geometry(3001, null, sdo_point_type(15, 15, 30), null, null),
24
n
=> 3200),
25
num_pts
=> 3200,
26
pc_tot_dim => 3,
27
srid
=> null,
28
blk_domain => null,
29
get_ids
=> 1)));
PT_POS
PTS(SDO_GTYPE, SDO_SRID, SDO_POINT(X, Y, Z), SDO_ELEM_INFO, SDO_ORDINATES)
-----------------------------------------------------------------------------------------1
SDO_GEOMETRY(3001, NULL, SDO_POINT_TYPE(15, 15, 30), NULL, NULL)
2
3200
Querying PostgreSQL Blocks
Querying
PostgreSQL blocks PC_Intersects / PC_Intersection (return blocks)
CREATE TABLE query_polygons (id integer, geom public.geometry(Geometry, SRID));

INSERT INTO query_polygons VALUES (1,ST_GeomFromEWKT(WKT));
SELECT
PC_Intersection(pa,geom)
FROM patches,
query_polygons
WHERE PC_Intersects(pa,geom) and query_polygons.id = 1;
Querying
PostgreSQL blocks PC_Intersects / PC_Intersection (return individual point records)
SELECT
PC_Get(qpoint, 'x') AS x,PC_Get(qpoint, 'y') AS y,
PC_Get(qpoint, 'z') AS z
FROM
(
SELECT
PC_Explode(PC_Intersection(pa,geom)) AS qpoint
FROM patches, query_polygons
WHERE PC_Intersects(pa,geom) and query_polygons.id = 1
) AS qtable;
Querying
PostgreSQL blocks Nearest Neighbor
SELECT
PC_Get(qpoint, 'x') as x, PC_Get(qpoint, 'y') as y, PC_Get(qpoint, 'z') as z
FROM
(SELECT
PC_explode(pa) as qpoint
FROM
(SELECT
pa
FROM patches ORDER BY geometry(pa) <#> ST_SetSRID(ST_MakePoint(px, py), SRID)
LIMIT 9
) as A
) as B
ORDER BY ((PC_Get(qpoint, 'x') - px)^2 +(PC_Get(qpoint, 'y') - py)^2)
LIMIT 1000;
Querying LAStools
Querying
LAStools lasmerge
If hybrid approach with PostgreSQL-PostGIS:
CREATE TABLE query_polygons (id integer, geom public.geometry(Geometry, SRID));
INSERT INTO query_polygons VALUES (1,ST_GeomFromEWKT(WKT));
psql dbname -U dbuser -h dbhost -p dbport -t -A -c \
SELECT l.filepath FROM lasindex l,query_polygons qp WHERE ST_Intersects(qp.geom, l.geom) AND qp.id=1"
\
> input1.list
Rectangle:
lasmerge -lof input.list
-inside minx miny maxx maxy -o output.las
Circle:
lasmerge -lof input.list
-inside_circle cx cy r -o output.las
Querying
LAStools lasclip (query region in ShapeFile)
pgsql2shp -f query5.shp -h dbhost -p dbport -u dbuser -P dbpass dbname \
select ST_SetSRID(geom, 28992) from query_polygons where id = 5;
If hybrid approach with PostgreSQL-PostGIS:

psql dbname -U dbuser -h dbhost -p dbport -t -A -c \
SELECT l.filepath FROM lasindex l,query_polygons qp WHERE
ST_Intersects(qp.geom, l.geom) AND qp.id=5" \
> input5.list
lasclip.exe -lof input5.list -poly query5.shp -o output5.las -merged
Querying
LAStools
Tune LAZ chunksize (default 50000) depending on your type of queries
lassort of large files done with sub-tiling
lasclip/lasmerge option io_buffer (default 262144 bytes) for granular accesses

Ask in LAStools forum
Querying Oracle Flat
Querying
Oracle flat Plain old SQL
Rectangle:
SELECT
*
FROM
FLAT
WHERE
(VAL_D1 BETWEEN minx AND maxx) AND
(VAL_D2 BETWEEN miny AND maxy);
Circle:
SELECT
*
FROM
(
SELECT
*
FROM
FLAT
WHERE
(VAL_D1 BETWEEN cx - r AND cx + r) AND
(VAL_D2 BETWEEN cy - r AND cy + r)
) B
WHERE
POWER((VAL_D1 - cx), 2) +
POWER((VAL_D2 - cy), 2) <= POWER(r, 2);
Querying
Oracle flat SDO_PointInPolygon
Other regions (example circle):
SQL> SELECT
x,y,z
FROM
TABLE(
mdsys.sdo_PointInPolygon(
CURSOR(
select
val_d1 x,
val_d2 y,
val_d3 z
from
flat),
MDSYS.SDO_GEOMETRY(2003, NULL, NULL,
MDSYS.SDO_ELEM_INFO_ARRAY(1, 1003, 4), -- circle with
MBR(10,10, 14,14)
MDSYS.SDO_ORDINATE_ARRAY(10, 12, 12, 10, 14, 12)),
0.05));
X
Y
Z
---------- ---------- ---------10
12
123
13 rows selected.
Elapsed: 00:00:13.16
Other regions (with pre-filtering):
SQL> SELECT
x, y, z
FROM
TABLE(
mdsys.sdo_PointInPolygon(
CURSOR(
select
val_d1 x,
val_d2 y,
val_d3 z
from
flat
where
val_d1 between 10 and 14 and
val_d2 between 10 and 14),
MDSYS.SDO_GEOMETRY(2003, NULL, NULL,
MDSYS.SDO_ELEM_INFO_ARRAY(1, 1003, 4), -- circle with MBR
(10, 10, 14, 14)
MDSYS.SDO_ORDINATE_ARRAY(10, 12, 12, 10, 14, 12)), 0.05));
X
Y
Z
---------- ---------- ---------10
12
123
13 rows selected.
Elapsed: 00:00:00.00
Querying
Oracle flat
Nearest neighbor - Approximation:
SELECT
*
FROM
(
SELECT
*
FROM
flat
WHERE
(
(x BETWEEN (px r) AND (px + r)) AND
(y BETWEEN (py r) AND (py + r))
)
ORDER BY
POWER(x - px, 2) + POWER(y - py, 2)
) a
WHERE
ROWNUM <= 1000;
Querying
Medium benchmark comparison
PostgreSQL flat: parallel
querying with PQGR algorithm
Oracle flat: native parallel
MonetDB flat: native parallel
PostgreSQL blocks: : parallel

Oracle blocks: : parallel
LAStools: single-process
Oracle EXADATA X4-2 half rack:

only 23090M
Querying
Medium benchmark comparison: 20M and 23090M; Queries #1 and #6
Oracle blocks global*

Oracle blocks global compress*
Oracle blocks increm.*
Oracle blocks increm. compress*
PostgreSQL blocks increm.*
PostgreSQL blocks increm. compress*
Oracle flat
PostgreSQL flat*
MonetDB
LAStools**
LAStools LAZ**
Oracle Exadata
Query 1 Query 1 Query 6 Query 6

20M
23090M
20M
23090M
0,35
0,35
1,20
1,20
0,41
0,41
1,38
1,40
0,35
0,35
1,20
1,20
0,41
0,41
1,38
1,40
0,25
0,25
1,50
1,50
0,32
0,32
1,72
1,68
0,24
1,25
1,32
27,18
0,35
5,17
1,72
15,46
0,06
7,21 187,16
198,51
0,02
0,03
1,46
1,48
0,14
0,12
1,90
1,91
0,18
0,67
* Parallelized with
algorithmn (not native)
** Single process
Querying
Medium benchmark comparison: Queries #1
Rectangle 51x53 m, 0.0027 km2, ~74872 points
Query #1 response time

0,50
Query 1 20M
Query 1 23090M
0,45
0,40
0,35
Time [s]
0,30
0,25
0,20
0,15
0,10
0,05
0,00
Oracle blocks
global
Oracle blocks
global compress
Oracle blocks
Oracle blocks
PostgreSQL
increm.
increm. compress blocks increm.
PostgreSQL
blocks increm.
compress
Oracle flat
PostgreSQL flat
MonetDB
LAStools
LAStools LAZ
Oracle Exadata
Querying
Complex Polygon, 792 pts, 0.025 km2, 1 hole,

~387135 points
Medium benchmark comparison: Queries #6

Query #6 response time
2,00
1,80
Query 6 20M
1,60
Query 6 23090M
1,40
Time [s]
1,20
1,00
0,80
0,60
0,40
0,20
0,00
Oracle blocks
global
Oracle blocks
global compress
Oracle blocks
increm.
Oracle blocks
increm.
compress
PostgreSQL
blocks increm.
PostgreSQL
blocks increm.
compress
Oracle flat
PostgreSQL flat
MonetDB
LAStools
LAStools LAZ
Oracle Exadata
Program Agenda
1
Introduction
Loading
Querying
Research and others
Conclusions
Research and others

Level of detail
Currently available in Oracle
SDO_PCR-tree, SDO_TIN, not yet
SDO_PCHilbert
Makes visualization of point cloud
scale very well
Resulting blocks in pyramid
SQL> select
2
pcblk_min_res
"Pyramid Level",
3
sum(num_points) "# Points",
4
count(*)
"# Blocks"
5 from
6
blocks
7 group by
8
pcblk_min_res
9 order by
10
pcblk_min_res;
Pyramid Level
# Points
# Blocks
------------- ---------- ---------1
1048576
105
2
524288
41
8
27166
1
Create PC with pyramid

declare
pc sdo_pc;
begin
pc :=
sdo_pc_pkg.init(
basetable
basecol
blktable
ptn_params
pc_extent
pc_tol
pc_tot_dimensions
pc_domain
pc_val_attr_tables
pc_other_attrs
=>
=>
=>
=>
=>
=>
=>
=>
=>
=>
'PCS',
--(base table)
'PC',
--(base column)
'BLOCKS',
--(data table)
'blk_capacity=10000',
--(block capacity, tablespace)
mdsys.sdo_geometry(
--(spatial extent)
2003,
null,
null,
mdsys.sdo_elem_info_array(1, 1003, 3),
mdsys.sdo_ordinate_array(1, 1, power(2, 13), power(2, 13))),
0.5,
--(tolerance)
3,
--(total dimensions)
null,
null,
xmltype(
--(object metadata pyramiding)
'<opc:sdoPcObjectMetadata
xmlns:opc="http://xmlns.oracle.com/spatial/vis3d/2011/sdovis3d.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<opc:sdoPcPyramid preserveLevel1="true"/>
</opc:sdoPcObjectMetadata>'));
sdo_pc_pkg.create_pc(
pc,
'INPTAB');
--(input table)
insert into pcs values (1, pc);

end;
/
8 rows selected.
6
5
4
Research and others

Level of detail
Currently available in Oracle
SDO_PCR-tree, SDO_TIN, not yet
SDO_PCHilbert
Thinning of points
Currently single simple approach
Others planned, maybe based on slope, category,
intensity, ...
Maybe customizable with Java function...
2
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
1 2 1 3 1
Query with LoD
select
query_points.x,
query_points.y,
query_points.z
from
table(
sdo_pc_pkg.clip_pc(
inp
ind_dim_qry
=> sdo_geometry(
2003,
null,
null,
qry_min_res
=> 1,
qry_max_res
table(
pts
num_pts
pc_tot_dim => 3,
srid
Research and others

Level of detail
Research and others

Parallel querying algorithms
Parallel Querying with Candidate blocks (PQCB)
Parallel Querying with Gridded Region (PQGR)
Research and others

Parallel Querying with Candidate blocks: Example Oracle blocks (Python cx_Oracle)
-- Get IDs of blocks touching the query region
-- The 8 processes insert data in parallel (example for process 1)
SELECT BLK_ID
INSERT INTO query_results_1
FROM blocks, query_polygons
SELECT x, y, z
WHERE SDO_FILTER(BLK_EXTENT,GEOM) = 'TRUE' AND id = 1;
FROM table (
-- Split the list of IDs in 8 chunks (in python)

-- Create table to store the results
sdo_PointInPolygon (
cursor (
CREATE TABLE query_results_1 (x NUMERIC, y NUMERIC, z NUMERIC);
SELECT pnt.x x, pnt.y y, pnt.z z

FROM (
SELECT points, num_points
FROM blocks
WHERE (BLK_ID = 1 OR BLK_ID = 2 )
) pcblob,
TABLE (
sdo_util.getvertices( sdo_pc_pkg.to_geometry(pcblob.points,
pcblob.num_points, 3,NULL))
) pnt
),
(select geom from query_polygons where id = 1), 0.0001, NULL));
Research and others

Parallel Querying with Gridded Region: Example PostgreSQL blocks (Python psycopg2)
-- Create table with 8 sub-regions of query region
-- The 8 processes insert data in parallel (example for process 1-1)
CREATE TABLE query_grid_1 AS
INSERT INTO
WITH A AS (
SELECT geom, st_xmin(geom) as minx, st_xmax(geom) as maxx,
query_results_1 ( x,y,z )
SELECT PC_Get(qpoint, x) as x,
st_ymin(geom) as miny, st_ymax(geom) as maxy
PC_Get(qpoint, y) as y,
FROM query_polygons where id = 1)

SELECT B.row, B.col,
ST_Intersection(A.geom, ST_SetSRID(B.geom, 28992)) as geom
FROM
PC_Get(qpoint, z) as z
FROM
(SELECT PC_explode(pc_intersection(pa,geom)) as qpoint
FROM patches, query_grid_1
A,
WHERE pc_intersects(pa,geom) AND row = 1 AND col = 1) as D;
(SELECT F.row, F.col, F.geom

FROM A,
ST_CreateFishnet(2, 4, (maxx-minx) / 4,
-- Remove possible duplicated points

CREATE TABLE cleaned_query_results_1 AS SELECT DISTINCT * FROM query_results_1;
(maxy-miny) / 2, minx, miny) F

) AS B ;
CREATE INDEX query_grid_1_rowcol ON query_grid_1 ( row, col );
-- Create table to store the results
CREATE TABLE query_results_1 (x NUMERIC, y NUMERIC, z NUMERIC);
http://trac.osgeo.org/postgis/wiki/UsersWikiCreateFishnet
Research and others

Faster flat tables: Query comparison
Query #1 (rectangle 51 x 53 m, ~74872 points)
8
7
6
Time[s]
PostgreSQL flat
5
Oracle flat
MonetDB flat
PostgreSQL blocks
Oracle blocks
2
LAStools
1
0
20M
210M
2201M
23090M
Research and others

Faster flat tables
Flat tables fast when small dataset but not scaling
Blocks not so fast but perfectly scaling (better storage but need blocks processing in queries)
What if we want DBMS with speed of flat tables but good scaling? (we do not care about storage)
Research and others

Faster flat tables: spatial clustering
Reorder data based on spatial dimensions
Efficiency in storage and queries
Already used in:
LAStools (lassort)
Oracle blocks (Data preparation in Hilbert R-Tree blocking)
Space filling curves: Hilbert/Morton

Useful for flat model directly
X, Y -> Code (position in the curve) No need to store X,Y!
Can also be used in cases where point dimension is not per se spatial (x, y, z), but of different nature (t or varioLoD/imp)
Queries need to be re-written to use spatial clustering and be more efficient
Research and others

Faster flat tables: Current approach flat table
Example input data
(sorted as acquired)
x
3
Flat table
15
2
1
0
0
0
x y
0 0
1 0
0 1
0 2
1 1
....
B-Tree XY
index
x
0
0
0
0
1
1
y
0
1
2
3
0
1
pos
0
2
3
9
1
4
Research and others

Faster flat tables: Current approach flat table with clustering (on B-Tree)
Flat table
x
3
15
2
1
0
0
0
x y
0 0
1 0
0 1
0 2
1 1
....
B-Tree XY
x
0
0
0
0
1
1
y
0
1
2
3
0
1
pos
0
2
3
9
1
4
Flat table
x
3
15
clustering
1
0
0
0
x
0
0
0
0
1
1
y
0
1
2
3
0
1
pos
0
1
2
3
4
5
Research and others

Faster flat tables: Space filling curves
(first y, then x)
x
3
x
3
15
x
3
2
1
0
Morton / Peano
Hilbert
0
1
Bad spatial clustering
15
15
0
0
Better spatial clustering
Research and others

Faster flat tables: Constructing Morton curve
Peano or Morton or N-order (or Z-order)
Recursively replace each vertex of basic curve
with the previous order curve
Alternative: bitwise interleaving
row
7
63
Also works in 3D (or nD)
5
row
3
2
row
01
00
15
3
0
00
01
1
0
4
3
2
1
0
0
col
0
0
col
Research and others

Faster flat tables: 3D Morton curve
2x2x2
4x4x4
8x8x8
Illustrations from http://asgerhoedt.dk

Research and others

Faster flat tables: Constructing Hilbert curve
Hilbert
rotate previous order curve
at vertex 0 (-) and vertex 3 (+)
Also in 3D
row
3
2
row
01
00
0
00
01
15
3
0
0
col
63
Research and others

Faster flat tables: 3D Hilbert curve
2x2x2
4x4x4
8x8x8
Illustrations from Wikimedia Commons

Research and others

Faster flat tables: Average number of clusters for all possible range queries
N=8, number of Clusters for a given range query:
Peano (3 ranges)
63
N*N
GRID
HILBERT
PEANO
2*2
1.11
1.22
0.11
4*4
1.64
2.16
0.52
8*8
2.93
4.41
1.48
16*16
5.60
9.29
3.69
Faloutsos and Roseman, 1989
Hilbert (2 ranges)
Research and others

LAStools lassort+lasindex
Oracle Hilbert blocking
PostgreSQL Morton/QuadTree blocking (NLeSC/TU Delft test)
SFC flat table model (NLeSC/TU Delft test)
compute Hilbert / Morton codes for all points

create b-tree index on Hilbert / Morton code (position in the curve)
cluster flat table on Hilbert / Morton curve
re-write queries to select ranges of codes
Research and others

Better SFC flat table model:
Not use the default heap-table, but an indexed organized table (not PostgreSQL)
No separate index structure needed -> more compact, faster (not PostgreSQL)
No need to store encoded dimensions (if 2D: x, y not necessary -> only code, z)
Research and others

Faster flat tables: SQL DDL (2D morton codes, still using X, Y)
Oracle
CREATE TABLE flat_temp (x,y,z,morton);
[ LOAD ]
CREATE TABLE flat (x,y,z,morton), CONSTRAINT PRIMARY KEY (morton, z) ORGANIZATION INDEX;
DROP TABLE flat_temp;
PostgreSQL
CREATE TABLE flat (x,y,z,morton);
[ BINARY LOAD ]
CREATE INDEX morton_index ON flat (morton);
CLUSTER flat using morton_index;
MonetDB
CREATE TABLE flat_temp (x,y,z,morton);
[ BINARY LOAD ]
CREATE TABLE flat (x,y,z,morton);
INSERT INTO flat SELECT * FROM flat_temp ORDER BY morton;
DROP TABLE flat_temp;
Research and others

Faster flat tables: Computing Morton code
111
7
110
6
101
5
100
4
011
3
010
2
001
1
000
000
0
0
61
63
62
Bitwise interleaving x-y coordinates

Also works in higher dimensions (nD)
Two example of Morton code:
x= 110, y=111 xy= 111101 (decimal 61)

x= 001, y=010 xy= 000110 (decimal 6)
6
3
010
2
011
3
001
1
100 101
4
5
110
6
111
7
X
Research and others

Faster flat tables: Morton queries
Based on concepts of Region Quadtree & Quadcodes
Y
111
7
110
6
101
5
100
4
011
3
010
2
001
1
000
0
Works for any type of query geometry (point, polyline,

polygon)
10
12
Also works in 3D (Octree) and higher dimensions
300
000
0
001
1
010
2
011
3
100 101
4
5
110
6
111
7
Quadcode 0:
Morton range 0-15
Quadcode 10:
Morton range 16-19
Quadcode 12:
Morton range 24-27
Quadcode 300:
Morton range 48-48
(Morton code gaps resp. 0, 4, 20)
query_geometry, polygon
Note : SW=0, NW=1, SE=2, NE=3
Research and others

CREATE TABLE query_results_1 AS (
SELECT x,y,z FROM (
SELECT x,y,z FROM ahn_flat WHERE (
(morton2D between 1341720113446912 and 1341720117641215)
OR (morton2D between 1341720126029824 and 1341720134418431)
OR (morton2D between 1341720679677952 and 1341720683872255))
) a WHERE (x between 85670.0 and 85721.0)
and (y between 446416.0 and 446469.0));
Research and others

Query #1 (rectangle 51 x 53 m, 74872 points)
6
Time[s]
5
4
PostgreSQL
PostgreSQL Morton
2
1
0
20M
210M
2201M
23090M
Time[s]
5
4
MonetDB
MonetDB Morton
2
1
0
20M
210M
2201M
23090M
Research and others

Full AHN2 benchmark: loading
system
Total load
time [hours]
Total
size [TB]
#points
LAStools unlic.
22:54
12.181
638,609,393,087
LAStools lic
16:47
11.617
638,609,393,101
LAStools lic LAZ
15:48
0.973
638,609,393,101
Oracle Exadata
4:39
2.240
639,478,217,460
MonetDB*
17:21
15.00
639,478,217,460
* Run in system with 256 GB

RAM (instead of 128 GB)
Research and others

Full AHN2 benchmark: querying
#points
QUERY
01_S_RECT
02_L_RECT
03_S_CIRC
04_L_CIRC
10_S_RECT_MAXZ
11_S_RECT_MINZ
12_L_RECT_MINZ
16_XL_RECT_EMPTY
18_NN_1000
19_NN_5000
20_NN_1000_river
05_S_SIMP_POLY
06_L_COMP_POLY_HOLE
07_DIAG_RECT
08_XL_POLYGON_2_HOLES
09_S_POLYLINE_BUFFER
15_XL_RECT
17_XL_CIRC
21_L_NARROW_RECT
27_STREET_DIAG_RECT
13_L_POLYLINE_BUFFER
14_DIAG_POLYLINE_BUFFER
22_L_NARROW_DIAG_RECT
23_XL_NARROW_DIAG_RECT
24_L_NARROW_DIAG_RECT_2
25_PROV_DIAG_RECT
26_MUNI_RECT
28_VAALS
29_MONTFERLAND
30_WESTERVELD
Time[s]
LAStools unlic. LAStools Exadata MonetDB LAStools unlic. LAS DB LAStools unlic. LAS No DB
LAStools LAS DB LAStools LAS No DB LAStools LAZ DB LAStools LAZ No DB Exadata MonetDB
74861
74863
74863
74855
0,07
0,79
0,06
0,77
0,2
0,84
0,48
108,99
718057
718070
718070
718132
0,16
0,85
0,16
0,78
0,49
1,09
0,79
111,33
34700
34699
34675
34699
0,07
0,78
0,06
0,7
0,27
0,89
1,22
109,36
563119
563105
563082
563105
0,16
0,85
0,12
0,78
0,44
1,15
1,69
112,9
2413
1734
2434
2434
0,08
0,82
0,08
0,69
0,43
1,02
0,4
112,43
591
591
591
591
0,05
0,84
0,05
0,72
0,13
0,73
0,44
111,31
343168
343168
343171
343171
0,26
1
0,26
0,98
1,32
1,95
0,6
114,98
0
0
0
0
0,04
0,75
0,04
0,66
0,03
0,69
0
113,51
1000
112,9
5000
107,94
0
109,2
182871
182875
182875
182875
0,7
31,05
0,39
28,2
0,63
29,46
1,43
110,42
460140
460158
387201
387193
1,52
32
1,57
30,2
1,97
30,31
1,29
112,17
45831
45813
45815
45815
0,55
31,43
0,55
28,76
0,99
29,23
1,71
112,14
2365925 2365896 2273469 2273380
3,72
34,72
3,87
33,88
5,67
35,88
2,86
120,93
620568
620527
620719
620529
2,34
33,44
2,27
32,4
3,94
33,43
1,58
118,71
3992330 3992377 3992290 3992525
0,49
1,37
0,57
1,23
1,94
2,63
2,23
120,18
2203066 2203055 2201280 2203055
0,32
1,16
0,32
1,01
1,1
1,71
2,51
114,31
382395
382335
382335
386278
2,23
4,43
2,38
4,11
16,94
18,81
0,95
109,2
27443
27453
27459
27456
1,61
914,44
0,25
29,18
0,35
29,06
1,23
107,92
897042
896927
897359
897126
412,29
804,29
266,13
958,88
317,91
1039,09
23,34
723,58
765989
766039
766029
766029
102,19
419,78
104,09
428,51
151,45
600,93
15,05
559,73
12148049 12147863 121478022246.9
154,97
1115,2
141,95
1800,41
284,58
2478,47
113,37
2246,9
691422551 6,91E+08 6,91E+08151.16
379,2
886,35
318,93
1051,76
552,01
1154,7
330,85
151,16
2468239 2468268 2468367 2468367
251,13
4106,54
294,99
13454
551,13
16510,9
393,21 6308,53
2,03E+09 3,53E+1412228.76
1193,1 12228,76
2124162497 2,12E+09 2,12E+09622.73
417,28
1127,56
214,98
405,26
536,1
560,95
25,79
622,73
809097723 8,09E+08 8,67E+081469.47
1635,3
1883,19
2240,1
2525,98
2426,43
2718,26
67,67 1469,47
1509662615 1,51E+09 1,51E+095577.69
3579,73
5746,61
4809,9
6419,05
3855,93
6396,54
120,02 5577,69
2,9E+09 1,34E+1427683.62
3569,54 27683,62
Research and others

Easy queries
1
0,9
0,8
0,7
LAStools LAS DB
Time [s]
0,6
LAStools LAS No DB
0,5
LAStools LAZ DB
LAStools LAZ No DB
0,4
Exadata
0,3
MonetDB
0,2
0,1
0
Research and others

Medium queries
5
4,5
4
3,5
LAStools LAS DB
Time [s]
LAStools LAS No DB
2,5
LAStools LAZ DB
LAStools LAZ No DB
Exadata
1,5
MonetDB
1
0,5
0
Research and others

Difficult queries
4000
3500
3000
Time [s]
2500
LAStools LAS DB
LAStools LAS No DB
2000
LAStools LAZ DB
LAStools LAZ No DB
1500
Exadata
1000
MonetDB
500
0
Program Agenda
1
Introduction
Loading
Querying
Research and others
Conclusions
Conclusions
For simple queries LAStools is the fastest but has not DBMS functionalities.
Careful with multi-user environments
LAStools with LAZ offers best storage (but slower queries)
For complex queries DBs with blocks offer good performance
For small datasets (million points) flat tables are suitable
For Exadata, a lot more
Blocks require packing/unpacking which adds overhead (noticeable in small queries)
DB loading should be improved

Parallel algorithms improve query response times
Parallelism can be pushed further
SFC-based flat tables improve scaling issue and faster queries (being investigated)
Oracle, MonetDB and LAStools actively improving PointCloud support

Workshop Dubai Cut

Uploaded by

Copyright:

Available Formats

Workshop Dubai Cut

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Workshop Dubai Cut

Uploaded by

Copyright:

Available Formats

Storage and Management of Large

Scale Point Cloud Data

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

File formats and tested datasets

Point cloud data management systems

Research and others

File formats and tested datasets

Point cloud data management systems

Research and others

Image courtesy of: PDOK, NL

NLeSC (Netherlands eScience Center)

Relevant use cases in GIS:

Requiring new techniques for

Other potential applications:

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Based on structured interviews with:

Report at MPC public website http://pointclouds.nl

Testing single-user environments to avoid load-generator complexities

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Why use a database here?

Data integration with other sources

Fast access to arbitrary part of data set

General benefits of mature DBMS

Possible 3D processing flows in Oracle 12c

Query and Clip

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

BB-Infrastruktur AG, R&D

[This technology] is indispensable to process

Stores and processes more than 8 billion

points of objects along railway tracks

Image courtesy of: IQsoft, Austria

File formats and tested datasets

Point cloud data management systems

Research and others

File formats and tested datasets

Current version 1.4

Header 375 bytes

11 Record (point) formats

Binary (20+ bytes per point)

Limited to 4.2B pts (< 1.4)

ASPRS Standard LIDAR Point Classes

LIDAR data schema evolution

Created, Never Classified

Low Point (noise)

Model Key Point (mass point)

Reserved for ASPRS Def

Format 4, 5: (1, 3) , waveform packet info

Reserved for ASPRS Def

Reserved for ASPRS Def

Format 0: X, Y, Z, Intensity, Return #, , classification,

Format 1: , GPS time

LAS 1.1 (2005)

LAS 1.2 (2008)

LAS 1.3 (2009)

LAS 1.4 (2011)

File formats and tested datasets

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

File formats and tested datasets

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

File formats and tested datasets