Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

Commit 32c27c6

Browse files
zilderzilder
committed
Merge branch 'master' of github.com:postgrespro/pg_pathman
2 parents b696a7e + 728c468 commit 32c27c6

11 files changed

+236
-18
lines changed

‎LICENSE

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
1-
Copyright (c) 2015-2016, Postgres Professional
2-
3-
Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
1+
pg_pathman is released under the PostgreSQL License, a liberal Open Source license, similar to the BSD or MIT licenses.
42

3+
Copyright (c) 2015-2017, Postgres Professional
4+
Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
55
Portions Copyright (c) 1994, The Regents of the University of California
66

77
Permission to use, copy, modify, and distribute this software and its documentation for any purpose, without fee, and without a written agreement is hereby granted, provided that the above copyright notice and this paragraph and the following two paragraphs appear in all copies.
88

9-
IN NO EVENT SHALL THE UNIVERSITY OF CALIFORNIA BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF THE UNIVERSITY OF CALIFORNIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
9+
IN NO EVENT SHALL POSTGRES PROFESSIONAL BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF POSTGRES PROFESSIONAL HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
1010

11-
THE UNIVERSITY OF CALIFORNIA SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, AND THE UNIVERSITY OF CALIFORNIA HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
11+
POSTGRES PROFESSIONAL SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, AND POSTGRES PROFESSIONAL HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.

‎Makefile

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,8 @@ REGRESS = pathman_basic \
3434
pathman_rowmarks \
3535
pathman_runtime_nodes \
3636
pathman_utility_stmt_hooking \
37-
pathman_calamity
37+
pathman_calamity \
38+
pathman_join_clause
3839

3940
EXTRA_REGRESS_OPTS=--temp-config=$(top_srcdir)/$(subdir)/conf.add
4041

‎README.md

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
[![Build Status](https://travis-ci.org/postgrespro/pg_pathman.svg?branch=master)](https://travis-ci.org/postgrespro/pg_pathman)
22
[![PGXN version](https://badge.fury.io/pg/pg_pathman.svg)](https://badge.fury.io/pg/pg_pathman)
33
[![codecov](https://codecov.io/gh/postgrespro/pg_pathman/branch/master/graph/badge.svg)](https://codecov.io/gh/postgrespro/pg_pathman)
4+
[![GitHub license](https://img.shields.io/badge/license-PostgreSQL-blue.svg)](https://raw.githubusercontent.com/postgrespro/pg_pathman/master/LICENSE)
45

56
# pg_pathman
67

@@ -67,7 +68,7 @@ Modify the **`shared_preload_libraries`** parameter in `postgresql.conf` as foll
6768
```
6869
shared_preload_libraries = 'pg_pathman'
6970
```
70-
> **Important:** `pg_pathman` may have conflicts with some other extensions which uses the same hook functions. For example, `pg_pathman` uses `ProcessUtility_hook` hook to handle COPY queries for partitioned tables. And it could sometimes interfere with `pg_stat_statements` extension which uses the same hook. In this case try to list libraries in certain order: `shared_preload_libraries = 'pg_pathman, pg_stat_statements'`
71+
> **Important:** `pg_pathman` may cause conflicts with some other extensions that use the same hook functions. For example, `pg_pathman` uses `ProcessUtility_hook` to handle COPY queries for partitioned tables, which means it may interfere with `pg_stat_statements` from time to time. In this case, try listing libraries in certain order: `shared_preload_libraries = 'pg_stat_statements, pg_pathman'`.
7172
7273
It is essential to restart the PostgreSQL instance. After that, execute the following query in psql:
7374
```plpgsql
@@ -99,8 +100,8 @@ create_hash_partitions(relation REGCLASS,
99100
attribute TEXT,
100101
partitions_count INTEGER,
101102
partition_data BOOLEAN DEFAULT TRUE,
102-
partition_names TEXT[] DEFAULT NULL,
103-
tablespaces TEXT[] DEFAULT NULL)
103+
partition_names TEXT[] DEFAULT NULL,
104+
tablespaces TEXT[] DEFAULT NULL)
104105
```
105106
Performs HASH partitioning for `relation` by integer key `attribute`. The `partitions_count` parameter specifies the number of partitions to create; it cannot be changed afterwards. If `partition_data` is `true` then all the data will be automatically copied from the parent table to partitions. Note that data migration may took a while to finish and the table will be locked until transaction commits. See `partition_table_concurrently()` for a lock-free way to migrate data. Partition creation callback is invoked for each partition if set beforehand (see `set_init_callback()`).
106107

@@ -661,11 +662,11 @@ SELECT disable_pathman_for('range_rel');
661662
```
662663
All sections and data will remain unchanged and will be handled by the standard PostgreSQL inheritance mechanism.
663664

664-
##Feedback
665+
## Feedback
665666
Do not hesitate to post your issues, questions and new ideas at the [issues](https://github.com/postgrespro/pg_pathman/issues) page.
666667

667668
## Authors
668-
Ildar Musin <i.musin@postgrespro.ru> Postgres Professional Ltd., Russia
669-
Alexander Korotkov <a.korotkov@postgrespro.ru> Postgres Professional Ltd., Russia
670-
Dmitry Ivanov <d.ivanov@postgrespro.ru> Postgres Professional Ltd., Russia
669+
Ildar Musin <i.musin@postgrespro.ru> Postgres Professional Ltd., Russia
670+
Alexander Korotkov <a.korotkov@postgrespro.ru> Postgres Professional Ltd., Russia
671+
Dmitry Ivanov <d.ivanov@postgrespro.ru> Postgres Professional Ltd., Russia
671672

‎README.rus.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -488,6 +488,6 @@ SELECT disable_pathman_for('range_rel');
488488
Если у вас есть вопросы или предложения, а также если вы обнаружили ошибки, напишите нам в разделе [issues](https://github.com/postgrespro/pg_pathman/issues).
489489

490490
## Авторы
491-
Ильдар Мусин <i.musin@postgrespro.ru> Postgres Professional, Россия
492-
Александр Коротков <a.korotkov@postgrespro.ru> Postgres Professional, Россия
493-
Дмитрий Иванов <d.ivanov@postgrespro.ru> Postgres Professional, Россия
491+
Ильдар Мусин <i.musin@postgrespro.ru> Postgres Professional, Россия
492+
Александр Коротков <a.korotkov@postgrespro.ru> Postgres Professional, Россия
493+
Дмитрий Иванов <d.ivanov@postgrespro.ru> Postgres Professional, Россия

‎expected/pathman_join_clause.out

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
\set VERBOSITY terse
2+
CREATE SCHEMA pathman;
3+
CREATE EXTENSION pg_pathman SCHEMA pathman;
4+
CREATE SCHEMA test;
5+
/*
6+
* Test push down a join clause into child nodes of append
7+
*/
8+
/* create test tables */
9+
CREATE TABLE test.fk (
10+
id1 INT NOT NULL,
11+
id2 INT NOT NULL,
12+
start_key INT,
13+
end_key INT,
14+
PRIMARY KEY (id1, id2));
15+
CREATE TABLE test.mytbl (
16+
id1 INT NOT NULL,
17+
id2 INT NOT NULL,
18+
key INT NOT NULL,
19+
CONSTRAINT fk_fk FOREIGN KEY (id1, id2) REFERENCES test.fk(id1, id2),
20+
PRIMARY KEY (id1, key));
21+
SELECT pathman.create_hash_partitions('test.mytbl', 'id1', 8);
22+
create_hash_partitions
23+
------------------------
24+
8
25+
(1 row)
26+
27+
/* ...fill out with test data */
28+
INSERT INTO test.fk VALUES (1, 1);
29+
INSERT INTO test.mytbl VALUES (1, 1, 5), (1,1,6);
30+
/* gather statistics on test tables to have deterministic plans */
31+
ANALYZE test.fk;
32+
ANALYZE test.mytbl;
33+
/* run test queries */
34+
EXPLAIN (COSTS OFF) /* test plan */
35+
SELECT m.tableoid::regclass, id1, id2, key, start_key, end_key
36+
FROM test.mytbl m JOIN test.fk USING(id1, id2)
37+
WHERE NOT key <@ int4range(6, end_key);
38+
QUERY PLAN
39+
------------------------------------------------------------------------------------
40+
Nested Loop
41+
-> Seq Scan on fk
42+
-> Custom Scan (RuntimeAppend)
43+
-> Bitmap Heap Scan on mytbl_0 m
44+
Recheck Cond: (id1 = fk.id1)
45+
Filter: ((fk.id2 = id2) AND (NOT (key <@ int4range(6, fk.end_key))))
46+
-> Bitmap Index Scan on mytbl_0_pkey
47+
Index Cond: (id1 = fk.id1)
48+
-> Bitmap Heap Scan on mytbl_1 m
49+
Recheck Cond: (id1 = fk.id1)
50+
Filter: ((fk.id2 = id2) AND (NOT (key <@ int4range(6, fk.end_key))))
51+
-> Bitmap Index Scan on mytbl_1_pkey
52+
Index Cond: (id1 = fk.id1)
53+
-> Bitmap Heap Scan on mytbl_2 m
54+
Recheck Cond: (id1 = fk.id1)
55+
Filter: ((fk.id2 = id2) AND (NOT (key <@ int4range(6, fk.end_key))))
56+
-> Bitmap Index Scan on mytbl_2_pkey
57+
Index Cond: (id1 = fk.id1)
58+
-> Bitmap Heap Scan on mytbl_3 m
59+
Recheck Cond: (id1 = fk.id1)
60+
Filter: ((fk.id2 = id2) AND (NOT (key <@ int4range(6, fk.end_key))))
61+
-> Bitmap Index Scan on mytbl_3_pkey
62+
Index Cond: (id1 = fk.id1)
63+
-> Bitmap Heap Scan on mytbl_4 m
64+
Recheck Cond: (id1 = fk.id1)
65+
Filter: ((fk.id2 = id2) AND (NOT (key <@ int4range(6, fk.end_key))))
66+
-> Bitmap Index Scan on mytbl_4_pkey
67+
Index Cond: (id1 = fk.id1)
68+
-> Bitmap Heap Scan on mytbl_5 m
69+
Recheck Cond: (id1 = fk.id1)
70+
Filter: ((fk.id2 = id2) AND (NOT (key <@ int4range(6, fk.end_key))))
71+
-> Bitmap Index Scan on mytbl_5_pkey
72+
Index Cond: (id1 = fk.id1)
73+
-> Bitmap Heap Scan on mytbl_6 m
74+
Recheck Cond: (id1 = fk.id1)
75+
Filter: ((fk.id2 = id2) AND (NOT (key <@ int4range(6, fk.end_key))))
76+
-> Bitmap Index Scan on mytbl_6_pkey
77+
Index Cond: (id1 = fk.id1)
78+
-> Bitmap Heap Scan on mytbl_7 m
79+
Recheck Cond: (id1 = fk.id1)
80+
Filter: ((fk.id2 = id2) AND (NOT (key <@ int4range(6, fk.end_key))))
81+
-> Bitmap Index Scan on mytbl_7_pkey
82+
Index Cond: (id1 = fk.id1)
83+
(43 rows)
84+
85+
/* test joint data */
86+
SELECT m.tableoid::regclass, id1, id2, key, start_key, end_key
87+
FROM test.mytbl m JOIN test.fk USING(id1, id2)
88+
WHERE NOT key <@ int4range(6, end_key);
89+
tableoid | id1 | id2 | key | start_key | end_key
90+
--------------+-----+-----+-----+-----------+---------
91+
test.mytbl_6 | 1 | 1 | 5 | |
92+
(1 row)
93+
94+
DROP SCHEMA test CASCADE;
95+
NOTICE: drop cascades to 10 other objects
96+
DROP EXTENSION pg_pathman CASCADE;
97+
DROP SCHEMA pathman CASCADE;

‎expected/pathman_utility_stmt_hooking.out

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -256,8 +256,33 @@ SELECT * FROM copy_stmt_hooking.test ORDER BY val;
256256
6 | hash_2 | 0 | 0
257257
(2 rows)
258258

259+
/* Check dropped colums before partitioning */
260+
CREATE TABLE copy_stmt_hooking.test2 (
261+
a varchar(50),
262+
b varchar(50),
263+
t timestamp without time zone not null
264+
);
265+
ALTER TABLE copy_stmt_hooking.test2 DROP COLUMN a;
266+
SELECT create_range_partitions('copy_stmt_hooking.test2',
267+
't',
268+
'2017-01-01 00:00:00'::timestamp,
269+
interval '1 hour', 5, false
270+
);
271+
NOTICE: sequence "test2_seq" does not exist, skipping
272+
create_range_partitions
273+
-------------------------
274+
5
275+
(1 row)
276+
277+
COPY copy_stmt_hooking.test2(t) FROM stdin;
278+
SELECT COUNT(*) FROM copy_stmt_hooking.test2;
279+
count
280+
-------
281+
1
282+
(1 row)
283+
259284
DROP SCHEMA copy_stmt_hooking CASCADE;
260-
NOTICE: drop cascades to 7 other objects
285+
NOTICE: drop cascades to 798 other objects
261286
/*
262287
* Test auto check constraint renaming
263288
*/

‎sql/pathman_join_clause.sql

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
\set VERBOSITY terse
2+
3+
CREATE SCHEMA pathman;
4+
CREATE EXTENSION pg_pathman SCHEMA pathman;
5+
CREATE SCHEMA test;
6+
7+
8+
/*
9+
* Test push down a join clause into child nodes of append
10+
*/
11+
12+
/* create test tables */
13+
CREATE TABLE test.fk (
14+
id1 INT NOT NULL,
15+
id2 INT NOT NULL,
16+
start_key INT,
17+
end_key INT,
18+
PRIMARY KEY (id1, id2));
19+
CREATE TABLE test.mytbl (
20+
id1 INT NOT NULL,
21+
id2 INT NOT NULL,
22+
key INT NOT NULL,
23+
CONSTRAINT fk_fk FOREIGN KEY (id1, id2) REFERENCES test.fk(id1, id2),
24+
PRIMARY KEY (id1, key));
25+
SELECT pathman.create_hash_partitions('test.mytbl', 'id1', 8);
26+
27+
/* ...fill out with test data */
28+
INSERT INTO test.fk VALUES (1, 1);
29+
INSERT INTO test.mytbl VALUES (1, 1, 5), (1,1,6);
30+
31+
/* gather statistics on test tables to have deterministic plans */
32+
ANALYZE test.fk;
33+
ANALYZE test.mytbl;
34+
35+
/* run test queries */
36+
EXPLAIN (COSTS OFF) /* test plan */
37+
SELECT m.tableoid::regclass, id1, id2, key, start_key, end_key
38+
FROM test.mytbl m JOIN test.fk USING(id1, id2)
39+
WHERE NOT key <@ int4range(6, end_key);
40+
/* test joint data */
41+
SELECT m.tableoid::regclass, id1, id2, key, start_key, end_key
42+
FROM test.mytbl m JOIN test.fk USING(id1, id2)
43+
WHERE NOT key <@ int4range(6, end_key);
44+
45+
46+
DROP SCHEMA test CASCADE;
47+
DROP EXTENSION pg_pathman CASCADE;
48+
DROP SCHEMA pathman CASCADE;
49+

‎sql/pathman_utility_stmt_hooking.sql

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,23 @@ COPY copy_stmt_hooking.test FROM stdin;
139139
SELECT count(*) FROM ONLY copy_stmt_hooking.test;
140140
SELECT * FROM copy_stmt_hooking.test ORDER BY val;
141141

142+
/* Check dropped colums before partitioning */
143+
CREATE TABLE copy_stmt_hooking.test2 (
144+
a varchar(50),
145+
b varchar(50),
146+
t timestamp without time zone not null
147+
);
148+
ALTER TABLE copy_stmt_hooking.test2 DROP COLUMN a;
149+
SELECT create_range_partitions('copy_stmt_hooking.test2',
150+
't',
151+
'2017-01-01 00:00:00'::timestamp,
152+
interval '1 hour', 5, false
153+
);
154+
COPY copy_stmt_hooking.test2(t) FROM stdin;
155+
2017-02-02 20:00:00
156+
\.
157+
SELECT COUNT(*) FROM copy_stmt_hooking.test2;
158+
142159
DROP SCHEMA copy_stmt_hooking CASCADE;
143160

144161

‎src/hooks.c

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
#include "xact_handling.h"
2121

2222
#include "access/transam.h"
23+
#include "catalog/pg_authid.h"
2324
#include "miscadmin.h"
2425
#include "optimizer/cost.h"
2526
#include "optimizer/restrictinfo.h"
@@ -556,14 +557,35 @@ pathman_post_parse_analysis_hook(ParseState *pstate, Query *query)
556557
/* Check that pg_pathman is the last extension loaded */
557558
if (post_parse_analyze_hook != pathman_post_parse_analysis_hook)
558559
{
559-
char *spl_value; /* value of "shared_preload_libraries" GUC */
560+
Oid save_userid;
561+
int save_sec_context;
562+
bool need_priv_escalation = !superuser(); /* we might be a SU */
563+
char *spl_value; /* value of "shared_preload_libraries" GUC */
560564

565+
/* Do we have to escalate privileges? */
566+
if (need_priv_escalation)
567+
{
568+
/* Get current user's Oid and security context */
569+
GetUserIdAndSecContext(&save_userid, &save_sec_context);
570+
571+
/* Become superuser in order to bypass sequence ACL checks */
572+
SetUserIdAndSecContext(BOOTSTRAP_SUPERUSERID,
573+
save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
574+
}
575+
576+
/* TODO: add a test for this case (non-privileged user etc) */
577+
578+
/* Only SU can read this GUC */
561579
#if PG_VERSION_NUM >= 90600
562580
spl_value = GetConfigOptionByName("shared_preload_libraries", NULL, false);
563581
#else
564582
spl_value = GetConfigOptionByName("shared_preload_libraries", NULL);
565583
#endif
566584

585+
/* Restore user's privileges */
586+
if (need_priv_escalation)
587+
SetUserIdAndSecContext(save_userid, save_sec_context);
588+
567589
ereport(ERROR,
568590
(errmsg("extension conflict has been detected"),
569591
errdetail("shared_preload_libraries = \"%s\"", spl_value),

‎src/pg_pathman.c

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -340,6 +340,11 @@ append_child_relation(PlannerInfo *root, Relation parent_relation,
340340
/* Now append 'appinfo' to 'root->append_rel_list' */
341341
root->append_rel_list = lappend(root->append_rel_list, appinfo);
342342

343+
/* Adjust join quals for this child */
344+
child_rel->joininfo = (List *) adjust_appendrel_attrs(root,
345+
(Node *) parent_rel->joininfo,
346+
appinfo);
347+
343348
/* Adjust target list for this child */
344349
adjust_rel_targetlist_compat(root, child_rel, parent_rel, appinfo);
345350

‎src/utility_stmt_hooking.c

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -587,6 +587,7 @@ PathmanCopyFrom(CopyState cstate, Relation parent_rel,
587587

588588
/* Place tuple in tuple slot --- but slot shouldn't free it */
589589
slot = myslot;
590+
ExecSetSlotDescriptor(slot, RelationGetDescr(child_result_rel->ri_RelationDesc));
590591
ExecStoreTuple(tuple, slot, InvalidBuffer, false);
591592

592593
skip_tuple = false;

0 commit comments

Comments
 (0)