Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

Commit a5cf808

Browse files
danielgustafssondanielgustafsson
committed
Read include/exclude commands for dump/restore from file
When there is a need to filter multiple tables with include and/or exclude options it's quite possible to run into the limitations of the commandline. This adds a --filter=FILENAME feature to pg_dump, pg_dumpall and pg_restore which is used to supply a file containing object exclude/include commands which work just like their commandline counterparts. The format of the file is one command per row like: <command> <object> <objectpattern> <command> can be "include" or "exclude", <object> can be table_data, index table_data_and_children, database, extension, foreign_data, function, table schema, table_and_children or trigger. This patch has gone through many revisions and design changes over a long period of time, the list of reviewers reflect reviewers of some version of the patch, not necessarily the final version. Patch by Pavel Stehule with some additional hacking by me. Author: Pavel Stehule <pavel.stehule@gmail.com> Reviewed-by: Justin Pryzby <pryzby@telsasoft.com> Reviewed-by: vignesh C <vignesh21@gmail.com> Reviewed-by: Dean Rasheed <dean.a.rasheed@gmail.com> Reviewed-by: Tomas Vondra <tomas.vondra@enterprisedb.com> Reviewed-by: Julien Rouhaud <rjuju123@gmail.com> Reviewed-by: Erik Rijkers <er@xs4all.nl> Discussion: https://postgr.es/m/CAFj8pRB10wvW0CC9Xq=1XDs=zCQxer3cbLcNZa+qiX4cUH-G_A@mail.gmail.com
1 parent 15c9ac3 commit a5cf808

File tree

13 files changed

+1872
-2
lines changed

13 files changed

+1872
-2
lines changed

‎doc/src/sgml/ref/pg_dump.sgml

+117
Original file line numberDiff line numberDiff line change
@@ -836,6 +836,109 @@ PostgreSQL documentation
836836
</listitem>
837837
</varlistentry>
838838

839+
<varlistentry>
840+
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
841+
<listitem>
842+
<para>
843+
Specify a filename from which to read patterns for objects to include
844+
or exclude from the dump. The patterns are interpreted according to the
845+
same rules as the corresponding options:
846+
<option>-t</option>/<option>--table</option>,
847+
<option>--table-and-children</option>,
848+
<option>--exclude-table-and-children</option> or
849+
<option>-T</option> for tables,
850+
<option>-n</option>/<option>--schema</option> for schemas,
851+
<option>--include-foreign-data</option> for data on foreign servers and
852+
<option>--exclude-table-data</option>,
853+
<option>--exclude-table-data-and-children</option> for table data,
854+
<option>-e</option>/<option>--extension</option> for extensions.
855+
To read from <literal>STDIN</literal>, use <filename>-</filename> as the
856+
filename. The <option>--filter</option> option can be specified in
857+
conjunction with the above listed options for including or excluding
858+
objects, and can also be specified more than once for multiple filter
859+
files.
860+
</para>
861+
862+
<para>
863+
The file lists one object pattern per row, with the following format:
864+
<synopsis>
865+
{ include | exclude } { extension | foreign_data | table | table_and_children | table_data | table_data_and_children | schema } <replaceable class="parameter">PATTERN</replaceable>
866+
</synopsis>
867+
</para>
868+
869+
<para>
870+
The first keyword specifies whether the objects matched by the pattern
871+
are to be included or excluded. The second keyword specifies the type
872+
of object to be filtered using the pattern:
873+
<itemizedlist>
874+
<listitem>
875+
<para>
876+
<literal>extension</literal>: extensions, works like the
877+
<option>--extension</option> option. This keyword can only be
878+
used with the <literal>include</literal> keyword.
879+
</para>
880+
</listitem>
881+
<listitem>
882+
<para>
883+
<literal>foreign_data</literal>: data on foreign servers, works like
884+
the <option>--include-foreign-data</option> option. This keyword can
885+
only be used with the <literal>include</literal> keyword.
886+
</para>
887+
</listitem>
888+
<listitem>
889+
<para>
890+
<literal>table</literal>: tables, works like the
891+
<option>-t</option>/<option>--table</option> option.
892+
</para>
893+
</listitem>
894+
<listitem>
895+
<para>
896+
<literal>table_and_children</literal>: tables including any partitions
897+
or inheritance child tables, works like the
898+
<option>--table-and-children</option> option.
899+
</para>
900+
</listitem>
901+
<listitem>
902+
<para>
903+
<literal>table_data</literal>: table data of any tables matching
904+
<replaceable>pattern</replaceable>, works like the
905+
<option>--exclude-table-data</option> option. This keyword can only
906+
be used with the <literal>exclude</literal> keyword.
907+
</para>
908+
</listitem>
909+
<listitem>
910+
<para>
911+
<literal>table_data_and_children</literal>: table data of any tables
912+
matching <replaceable>pattern</replaceable> as well as any partitions
913+
or inheritance children of the table(s), works like the
914+
<option>--exclude-table-data-and-children</option> option. This
915+
keyword can only be used with the <literal>exclude</literal> keyword.
916+
</para>
917+
</listitem>
918+
<listitem>
919+
<para>
920+
<literal>schema</literal>: schemas, works like the
921+
<option>-n</option>/<option>--schema</option> option.
922+
</para>
923+
</listitem>
924+
</itemizedlist>
925+
</para>
926+
927+
<para>
928+
Lines starting with <literal>#</literal> are considered comments and
929+
ignored. Comments can be placed after an object pattern row as well.
930+
Blank lines are also ignored. See <xref linkend="app-psql-patterns"/>
931+
for how to perform quoting in patterns.
932+
</para>
933+
934+
<para>
935+
Example files are listed below in the <xref linkend="pg-dump-examples"/>
936+
section.
937+
</para>
938+
939+
</listitem>
940+
</varlistentry>
941+
839942
<varlistentry>
840943
<term><option>--if-exists</option></term>
841944
<listitem>
@@ -1168,6 +1271,7 @@ PostgreSQL documentation
11681271
schema (<option>-n</option>/<option>--schema</option>) and
11691272
table (<option>-t</option>/<option>--table</option>) pattern
11701273
match at least one extension/schema/table in the database to be dumped.
1274+
This also applies to filters used with <option>--filter</option>.
11711275
Note that if none of the extension/schema/table patterns find
11721276
matches, <application>pg_dump</application> will generate an error
11731277
even without <option>--strict-names</option>.
@@ -1611,6 +1715,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
16111715

16121716
<screen>
16131717
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb &gt; mytab.sql</userinput>
1718+
</screen></para>
1719+
1720+
<para>
1721+
To dump all tables whose names start with <literal>mytable</literal>, except
1722+
for table <literal>mytable2</literal>, specify a filter file
1723+
<filename>filter.txt</filename> like:
1724+
<programlisting>
1725+
include table mytable*
1726+
exclude table mytable2
1727+
</programlisting>
1728+
1729+
<screen>
1730+
<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb &gt; db.sql</userinput>
16141731
</screen></para>
16151732

16161733
</refsect1>

‎doc/src/sgml/ref/pg_dumpall.sgml

+31
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,37 @@ PostgreSQL documentation
125125
</listitem>
126126
</varlistentry>
127127

128+
<varlistentry>
129+
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
130+
<listitem>
131+
<para>
132+
Specify a filename from which to read patterns for databases excluded
133+
from the dump. The patterns are interpreted according to the same rules
134+
as <option>--exclude-database</option>.
135+
To read from <literal>STDIN</literal>, use <filename>-</filename> as the
136+
filename. The <option>--filter</option> option can be specified in
137+
conjunction with <option>--exclude-database</option> for excluding
138+
databases, and can also be specified more than once for multiple filter
139+
files.
140+
</para>
141+
142+
<para>
143+
The file lists one database pattern per row, with the following format:
144+
<synopsis>
145+
exclude database <replaceable class="parameter">PATTERN</replaceable>
146+
</synopsis>
147+
</para>
148+
149+
<para>
150+
Lines starting with <literal>#</literal> are considered comments and
151+
ignored. Comments can be placed after an object pattern row as well.
152+
Blank lines are also ignored. See <xref linkend="app-psql-patterns"/>
153+
for how to perform quoting in patterns.
154+
</para>
155+
156+
</listitem>
157+
</varlistentry>
158+
128159
<varlistentry>
129160
<term><option>-g</option></term>
130161
<term><option>--globals-only</option></term>

‎doc/src/sgml/ref/pg_restore.sgml

+80
Original file line numberDiff line numberDiff line change
@@ -190,6 +190,86 @@ PostgreSQL documentation
190190
</listitem>
191191
</varlistentry>
192192

193+
<varlistentry>
194+
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
195+
<listitem>
196+
<para>
197+
Specify a filename from which to read patterns for objects excluded
198+
or included from restore. The patterns are interpreted according to the
199+
same rules as
200+
<option>-n</option>/<option>--schema</option> for including objects in schemas,
201+
<option>-N</option>/<option>--exclude-schema</option>for excluding objects in schemas,
202+
<option>-P</option>/<option>--function</option> for restoring named functions,
203+
<option>-I</option>/<option>--index</option> for restoring named indexes,
204+
<option>-t</option>/<option>--table</option> for restoring named tables
205+
or <option>-T</option>/<option>--trigger</option> for restoring triggers.
206+
To read from <literal>STDIN</literal>, use <filename>-</filename> as the
207+
filename. The <option>--filter</option> option can be specified in
208+
conjunction with the above listed options for including or excluding
209+
objects, and can also be specified more than once for multiple filter
210+
files.
211+
</para>
212+
213+
<para>
214+
The file lists one database pattern per row, with the following format:
215+
<synopsis>
216+
{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
217+
</synopsis>
218+
</para>
219+
220+
<para>
221+
The first keyword specifies whether the objects matched by the pattern
222+
are to be included or excluded. The second keyword specifies the type
223+
of object to be filtered using the pattern:
224+
<itemizedlist>
225+
<listitem>
226+
<para>
227+
<literal>function</literal>: functions, works like the
228+
<option>-P</option>/<option>--function</option> option. This keyword
229+
can only be used with the <literal>include</literal> keyword.
230+
</para>
231+
</listitem>
232+
<listitem>
233+
<para>
234+
<literal>index</literal>: indexes, works like the
235+
<option>-I</option>/<option>--indexes</option> option. This keyword
236+
can only be used with the <literal>include</literal> keyword.
237+
</para>
238+
</listitem>
239+
<listitem>
240+
<para>
241+
<literal>schema</literal>: schemas, works like the
242+
<option>-n</option>/<option>--schema</option> and
243+
<option>-N</option>/<option>--exclude-schema</option> options.
244+
</para>
245+
</listitem>
246+
<listitem>
247+
<para>
248+
<literal>table</literal>: tables, works like the
249+
<option>-t</option>/<option>--table</option> option. This keyword
250+
can only be used with the <literal>include</literal> keyword.
251+
</para>
252+
</listitem>
253+
<listitem>
254+
<para>
255+
<literal>trigger</literal>: triggers, works like the
256+
<option>-T</option>/<option>--trigger</option> option. This keyword
257+
can only be used with the <literal>include</literal> keyword.
258+
</para>
259+
</listitem>
260+
</itemizedlist>
261+
</para>
262+
263+
<para>
264+
Lines starting with <literal>#</literal> are considered comments and
265+
ignored. Comments can be placed after an object pattern row as well.
266+
Blank lines are also ignored. See <xref linkend="app-psql-patterns"/>
267+
for how to perform quoting in patterns.
268+
</para>
269+
270+
</listitem>
271+
</varlistentry>
272+
193273
<varlistentry>
194274
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
195275
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>

‎src/bin/pg_dump/Makefile

+3-2
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ OBJS = \
3232
compress_none.o \
3333
compress_zstd.o \
3434
dumputils.o \
35+
filter.o \
3536
parallel.o \
3637
pg_backup_archiver.o \
3738
pg_backup_custom.o \
@@ -49,8 +50,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
4950
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
5051
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
5152

52-
pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
53-
$(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
53+
pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
54+
$(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
5455

5556
install: all installdirs
5657
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)

0 commit comments

Comments
 (0)