Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

Commit cc333f3

Browse files
robertmhaasrobertmhaas
committed
Modify pg_basebackup to use a new COPY subprotocol for base backups.
In the new approach, all files across all tablespaces are sent in a single COPY OUT operation. The CopyData messages are no longer raw archive content; rather, each message is prefixed with a type byte that describes its purpose, e.g. 'n' signifies the start of a new archive and 'd' signifies archive or manifest data. This protocol is significantly more extensible than the old approach, since we can later create more message types, though not without concern for backward compatibility. The new protocol sends a few things to the client that the old one did not. First, it sends the name of each archive explicitly, instead of letting the client compute it. This is intended to make it easier to write future patches that might send archives in a format other that tar (e.g. cpio, pax, tar.gz). Second, it sends explicit progress messages rather than allowing the client to assume that progress is defined by the number of bytes received. This will help with future features where the server compresses the data, or sends it someplace directly rather than transmitting it to the client. The old protocol is still supported for compatibility with previous releases. The new protocol is selected by means of a new TARGET option to the BASE_BACKUP command. Currently, the only supported target is 'client'. Support for additional targets will be added in a later commit. Patch by me. The patch set of which this is a part has had review and/or testing from Jeevan Ladhe, Tushar Ahuja, Suraj Kharage, Dipesh Pandit, and Mark Dilger. Discussion: http://postgr.es/m/CA+TgmoaYZbz0=Yk797aOJwkGJC-LK3iXn+wzzMx7KdwNpZhS5g@mail.gmail.com
1 parent 3414099 commit cc333f3

File tree

5 files changed

+806
-48
lines changed

5 files changed

+806
-48
lines changed

‎doc/src/sgml/protocol.sgml

Lines changed: 120 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2630,6 +2630,22 @@ The commands accepted in replication mode are:
26302630
</listitem>
26312631
</varlistentry>
26322632

2633+
<varlistentry>
2634+
<term><literal>TARGET</literal> <replaceable>'target'</replaceable></term>
2635+
<listitem>
2636+
<para>
2637+
Tells the server where to send the backup. If not specified,
2638+
the legacy base backup protocol will be used. Otherwise, the new
2639+
protocol will be used, as described below.
2640+
</para>
2641+
2642+
<para>
2643+
At present, the only supported value for this parameter is
2644+
<literal>client</literal>.
2645+
</para>
2646+
</listitem>
2647+
</varlistentry>
2648+
26332649
<varlistentry>
26342650
<term><literal>PROGRESS [ <replaceable class="parameter">boolean</replaceable> ]</literal></term>
26352651
<listitem>
@@ -2805,19 +2821,113 @@ The commands accepted in replication mode are:
28052821

28062822
<para>
28072823
After the second regular result set, one or more CopyOutResponse results
2808-
will be sent, one for the main data directory and one for each additional tablespace other
2809-
than <literal>pg_default</literal> and <literal>pg_global</literal>. The data in
2810-
the CopyOutResponse results will be a tar format (following the
2811-
<quote>ustar interchange format</quote> specified in the POSIX 1003.1-2008
2812-
standard) dump of the tablespace contents. Prior to
2824+
will be sent. If the <literal>TARGET</literal> option is not specified,
2825+
the legacy base backup protocol will be used. In this mode,
2826+
there will be one CopyOutResponse for the main directory, one for each
2827+
additional tablespace other than <literal>pg_default</literal> and
2828+
<literal>pg_global</literal>, and one for the backup manifested if
2829+
requested. The main data directory and any additional tablespaces will
2830+
be sent in tar format (following the <quote>ustar interchange
2831+
format</quote> specified in the POSIX 1003.1-2008 standard), and
2832+
the manifest will sent as a plain file. Prior to
28132833
<literal>PostgreSQL</literal> 15, the server omitted the two trailing
28142834
blocks of zeroes specified in the standard, but this is no longer the
28152835
case.
2816-
After the tar data is complete, and if a backup manifest was requested,
2817-
another CopyOutResponse result is sent, containing the manifest data for the
2818-
current base backup. In any case, a final ordinary result set will be
2819-
sent, containing the WAL end position of the backup, in the same format as
2820-
the start position.
2836+
</para>
2837+
2838+
<para>
2839+
New applications should specify the <literal>TARGET</literal> option.
2840+
When that option is used, a single CopyOutResponse will be sent, and
2841+
the payload of each CopyData message will contain a message in one of
2842+
the following formats:
2843+
</para>
2844+
2845+
<para>
2846+
<variablelist>
2847+
2848+
<varlistentry>
2849+
<term>new archive (B)</term>
2850+
<listitem><para><variablelist>
2851+
<varlistentry>
2852+
<term>Byte1('n')</term>
2853+
<listitem><para>
2854+
Identifes the messaage as indicating the start of a new archive.
2855+
</para></listitem>
2856+
</varlistentry>
2857+
<varlistentry>
2858+
<term>String</term>
2859+
<listitem><para>
2860+
The file name for this archive.
2861+
</para></listitem>
2862+
</varlistentry>
2863+
<varlistentry>
2864+
<term>String</term>
2865+
<listitem><para>
2866+
For the main data directory, an empty string. For other
2867+
tablespaces, the full path to the directory from which this
2868+
archive was created.
2869+
</para></listitem>
2870+
</varlistentry>
2871+
</variablelist></para></listitem>
2872+
</varlistentry>
2873+
2874+
<varlistentry>
2875+
<term>manifest (B)</term>
2876+
<listitem><para><variablelist>
2877+
<varlistentry>
2878+
<term>Byte1('m')</term>
2879+
<listitem><para>
2880+
Identifes the message as indicating the start of the backup
2881+
manifest.
2882+
</para></listitem>
2883+
</varlistentry>
2884+
</variablelist></para></listitem>
2885+
</varlistentry>
2886+
2887+
<varlistentry>
2888+
<term>archive or manifest data (B)</term>
2889+
<listitem><para><variablelist>
2890+
<varlistentry>
2891+
<term>Byte1('d')</term>
2892+
<listitem><para>
2893+
Identifes the message as containing archive or manifest data.
2894+
</para></listitem>
2895+
</varlistentry>
2896+
<varlistentry>
2897+
<term>Byte<replaceable>n</replaceable></term>
2898+
<listitem><para>
2899+
Data bytes.
2900+
</para></listitem>
2901+
</varlistentry>
2902+
</variablelist></para></listitem>
2903+
</varlistentry>
2904+
2905+
<varlistentry>
2906+
<term>progress report (B)</term>
2907+
<listitem><para><variablelist>
2908+
<varlistentry>
2909+
<term>Byte1('p')</term>
2910+
<listitem><para>
2911+
Identifes the message as a progress report.
2912+
</para></listitem>
2913+
</varlistentry>
2914+
<varlistentry>
2915+
<term>Int64</term>
2916+
<listitem><para>
2917+
The number of bytes from the current tablespace for which
2918+
processing has been completed.
2919+
</para></listitem>
2920+
</varlistentry>
2921+
</variablelist></para></listitem>
2922+
</varlistentry>
2923+
2924+
</variablelist>
2925+
</para>
2926+
2927+
<para>
2928+
After the CopyOutResponse, or all such responses, have been sent, a
2929+
final ordinary result set will be sent, containing the WAL end position
2930+
of the backup, in the same format as the start position.
28212931
</para>
28222932

28232933
<para>

‎src/backend/replication/basebackup.c

Lines changed: 34 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,12 @@
5353
*/
5454
#define SINK_BUFFER_LENGTH Max(32768, BLCKSZ)
5555

56+
typedef enum
57+
{
58+
BACKUP_TARGET_COMPAT,
59+
BACKUP_TARGET_CLIENT
60+
} backup_target_type;
61+
5662
typedef struct
5763
{
5864
const char *label;
@@ -62,6 +68,7 @@ typedef struct
6268
bool includewal;
6369
uint32 maxrate;
6470
bool sendtblspcmapfile;
71+
backup_target_type target;
6572
backup_manifest_option manifest;
6673
pg_checksum_type manifest_checksum_type;
6774
} basebackup_options;
@@ -694,8 +701,10 @@ parse_basebackup_options(List *options, basebackup_options *opt)
694701
bool o_noverify_checksums = false;
695702
bool o_manifest = false;
696703
bool o_manifest_checksums = false;
704+
bool o_target = false;
697705

698706
MemSet(opt, 0, sizeof(*opt));
707+
opt->target = BACKUP_TARGET_COMPAT;
699708
opt->manifest = MANIFEST_OPTION_NO;
700709
opt->manifest_checksum_type = CHECKSUM_TYPE_CRC32C;
701710

@@ -836,6 +845,22 @@ parse_basebackup_options(List *options, basebackup_options *opt)
836845
optval)));
837846
o_manifest_checksums = true;
838847
}
848+
else if (strcmp(defel->defname, "target") == 0)
849+
{
850+
char *optval = defGetString(defel);
851+
852+
if (o_target)
853+
ereport(ERROR,
854+
(errcode(ERRCODE_SYNTAX_ERROR),
855+
errmsg("duplicate option \"%s\"", defel->defname)));
856+
if (strcmp(optval, "client") == 0)
857+
opt->target = BACKUP_TARGET_CLIENT;
858+
else
859+
ereport(ERROR,
860+
(errcode(ERRCODE_SYNTAX_ERROR),
861+
errmsg("unrecognized target: \"%s\"", optval)));
862+
o_target = true;
863+
}
839864
else
840865
ereport(ERROR,
841866
errcode(ERRCODE_SYNTAX_ERROR),
@@ -881,8 +906,15 @@ SendBaseBackup(BaseBackupCmd *cmd)
881906
set_ps_display(activitymsg);
882907
}
883908

884-
/* Create a basic basebackup sink. */
885-
sink = bbsink_copytblspc_new();
909+
/*
910+
* If the TARGET option was specified, then we can use the new copy-stream
911+
* protocol. If not, we must fall back to the old and less capable
912+
* copy-tablespace protocol.
913+
*/
914+
if (opt.target != BACKUP_TARGET_COMPAT)
915+
sink = bbsink_copystream_new();
916+
else
917+
sink = bbsink_copytblspc_new();
886918

887919
/* Set up network throttling, if client requested it */
888920
if (opt.maxrate > 0)

0 commit comments

Comments
 (0)