Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

Commit 5c279a6

Browse files
jeff-davisjeff-davis
committed
Custom WAL Resource Managers.
Allow extensions to specify a new custom resource manager (rmgr), which allows specialized WAL. This is meant to be used by a Table Access Method or Index Access Method. Prior to this commit, only Generic WAL was available, which offers support for recovery and physical replication but not logical replication. Reviewed-by: Julien Rouhaud, Bharath Rupireddy, Andres Freund Discussion: https://postgr.es/m/ed1fb2e22d15d3563ae0eb610f7b61bb15999c0a.camel%40j-davis.com
1 parent a8cfb0c commit 5c279a6

File tree

24 files changed

+526
-70
lines changed

24 files changed

+526
-70
lines changed

‎doc/src/sgml/config.sgml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11189,8 +11189,8 @@ LOG: CleanUpLock: deleting: lock(0xb7acd844) id(24688,24696,0,0,0,1)
1118911189
<literal>heap2</literal>, <literal>btree</literal>, <literal>hash</literal>,
1119011190
<literal>gin</literal>, <literal>gist</literal>, <literal>sequence</literal>,
1119111191
<literal>spgist</literal>, <literal>brin</literal>, and <literal>generic</literal>.
11192-
Only superusers and users with the appropriate <literal>SET</literal>
11193-
privilege can change this setting.
11192+
Extensions may define additional resource managers. Only superusers and users with
11193+
the appropriate <literal>SET</literal> privilege can change this setting.
1119411194
</para>
1119511195
</listitem>
1119611196
</varlistentry>

‎doc/src/sgml/custom-rmgr.sgml

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
<!-- doc/src/sgml/custom-rmgr.sgml -->
2+
3+
<chapter id="custom-rmgr">
4+
<title>Custom WAL Resource Managers</title>
5+
6+
<para>
7+
This chapter explains the interface between the core
8+
<productname>PostgreSQL</productname> system and custom WAL resource
9+
managers, which enable extensions to integrate directly with the <link
10+
linkend="wal"><acronym>WAL</acronym></link>.
11+
</para>
12+
<para>
13+
An extension, especially a <link linkend="tableam">Table Access
14+
Method</link> or <link linkend="indexam">Index Access Method</link>, may
15+
need to use WAL for recovery, replication, and/or <link
16+
linkend="logicaldecoding">Logical Decoding</link>. Custom resource managers
17+
are a more flexible alternative to <link linkend="generic-wal">Generic
18+
WAL</link> (which does not support logical decoding), but more complex for
19+
an extension to implement.
20+
</para>
21+
<para>
22+
To create a new custom WAL resouce manager, first define an
23+
<structname>RmgrData</structname> structure with implementations for the
24+
resource manager methods. Refer to
25+
<filename>src/backend/access/transam/README</filename> and
26+
<filename>src/include/access/xlog_internal.h</filename> in the
27+
<productname>PostgreSQL</productname> source.
28+
<programlisting>
29+
/*
30+
* Method table for resource managers.
31+
*
32+
* This struct must be kept in sync with the PG_RMGR definition in
33+
* rmgr.c.
34+
*
35+
* rm_identify must return a name for the record based on xl_info (without
36+
* reference to the rmid). For example, XLOG_BTREE_VACUUM would be named
37+
* "VACUUM". rm_desc can then be called to obtain additional detail for the
38+
* record, if available (e.g. the last block).
39+
*
40+
* rm_mask takes as input a page modified by the resource manager and masks
41+
* out bits that shouldn't be flagged by wal_consistency_checking.
42+
*
43+
* RmgrTable[] is indexed by RmgrId values (see rmgrlist.h). If rm_name is
44+
* NULL, the corresponding RmgrTable entry is considered invalid.
45+
*/
46+
typedef struct RmgrData
47+
{
48+
const char *rm_name;
49+
void (*rm_redo) (XLogReaderState *record);
50+
void (*rm_desc) (StringInfo buf, XLogReaderState *record);
51+
const char *(*rm_identify) (uint8 info);
52+
void (*rm_startup) (void);
53+
void (*rm_cleanup) (void);
54+
void (*rm_mask) (char *pagedata, BlockNumber blkno);
55+
void (*rm_decode) (struct LogicalDecodingContext *ctx,
56+
struct XLogRecordBuffer *buf);
57+
} RmgrData;
58+
</programlisting>
59+
</para>
60+
<para>
61+
Then, register your new resource
62+
manager.
63+
64+
<programlisting>
65+
/*
66+
* Register a new custom WAL resource manager.
67+
*
68+
* Resource manager IDs must be globally unique across all extensions. Refer
69+
* to https://wiki.postgresql.org/wiki/CustomWALResourceManager to reserve a
70+
* unique RmgrId for your extension, to avoid conflicts with other extension
71+
* developers. During development, use RM_EXPERIMENTAL_ID to avoid needlessly
72+
* reserving a new ID.
73+
*/
74+
extern void RegisterCustomRmgr(RmgrId rmid, RmgrData *rmgr);
75+
</programlisting>
76+
<function>RegisterCustomRmgr</function> must be called from the
77+
extension module's <link linkend="xfunc-c-dynload">_PG_init</link> function.
78+
While developing a new extension, use <literal>RM_EXPERIMENTAL_ID</literal>
79+
for <parameter>rmid</parameter>. When you ready to release the extension to
80+
users, reserve a new resource manager ID at the <ulink
81+
url="https://wiki.postgresql.org/wiki/CustomWALResourceManagers">Custom WAL
82+
Resource Manager</ulink> page.
83+
</para>
84+
85+
<para>
86+
Place the extension module implementing the custom resource manager in <xref
87+
linkend="guc-shared-preload-libraries"/> so that it will be loaded early
88+
during <productname>PostgreSQL</productname> startup.
89+
</para>
90+
<note>
91+
<para>
92+
The extension must remain in shared_preload_libraries as long as any
93+
custom WAL records may exist in the system. Otherwise
94+
<productname>PostgreSQL</productname> will not be able to apply or decode
95+
the custom WAL records, which may prevent the server from starting.
96+
</para>
97+
</note>
98+
</chapter>

‎doc/src/sgml/filelist.sgml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@
105105
<!ENTITY storage SYSTEM "storage.sgml">
106106
<!ENTITY tablesample-method SYSTEM "tablesample-method.sgml">
107107
<!ENTITY generic-wal SYSTEM "generic-wal.sgml">
108+
<!ENTITY custom-rmgr SYSTEM "custom-rmgr.sgml">
108109
<!ENTITY backup-manifest SYSTEM "backup-manifest.sgml">
109110

110111
<!-- contrib information -->

‎doc/src/sgml/func.sgml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26001,6 +26001,25 @@ postgres=# SELECT * FROM pg_walfile_name_offset((pg_backup_stop()).lsn);
2600126001
without recovery, the function returns <literal>NULL</literal>.
2600226002
</para></entry>
2600326003
</row>
26004+
26005+
<row>
26006+
<entry role="func_table_entry"><para role="func_signature">
26007+
<indexterm>
26008+
<primary>pg_get_wal_resource_managers</primary>
26009+
</indexterm>
26010+
<function>pg_get_wal_resource_managers</function> ()
26011+
<returnvalue>setof record</returnvalue>
26012+
( <parameter>rm_id</parameter> <type>integer</type>,
26013+
<parameter>rm_name</parameter> <type>text</type>,
26014+
<parameter>rm_builtin</parameter> <type>boolean</type> )
26015+
</para>
26016+
<para>
26017+
Returns the currently-loaded WAL resource managers in the system. The
26018+
column <parameter>rm_builtin</parameter> indicates whether it's a
26019+
built-in resource manager, or a custom resource manager loaded by an
26020+
extension.
26021+
</para></entry>
26022+
</row>
2600426023
</tbody>
2600526024
</tgroup>
2600626025
</table>

‎doc/src/sgml/generic-wal.sgml

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,25 @@
66
<para>
77
Although all built-in WAL-logged modules have their own types of WAL
88
records, there is also a generic WAL record type, which describes changes
9-
to pages in a generic way. This is useful for extensions that provide
10-
custom access methods, because they cannot register their own WAL redo
11-
routines.
9+
to pages in a generic way. This is useful for extensions that provide
10+
custom access methods.
1211
</para>
1312

13+
<para>
14+
In comparison with <link linkend="custom-rmgr">Custom WAL Resource
15+
Managers</link>, Generic WAL is simpler for an extension to implement and
16+
does not require the extension library to be loaded in order to apply the
17+
records.
18+
</para>
19+
20+
<note>
21+
<para>
22+
Generic WAL records are ignored during <link
23+
linkend="logicaldecoding">Logical Decoding</link>. If logical decoding is
24+
required for your extension, consider a Custom WAL Resource Manager.
25+
</para>
26+
</note>
27+
1428
<para>
1529
The API for constructing generic WAL records is defined in
1630
<filename>access/generic_xlog.h</filename> and implemented

‎doc/src/sgml/postgres.sgml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -262,6 +262,7 @@ break is not needed in a wider output rendering.
262262
&tableam;
263263
&indexam;
264264
&generic-wal;
265+
&custom-rmgr;
265266
&btree;
266267
&gist;
267268
&spgist;

‎doc/src/sgml/ref/pg_waldump.sgml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,14 @@ PostgreSQL documentation
173173
If <literal>list</literal> is passed as name, print a list of valid resource manager
174174
names, and exit.
175175
</para>
176+
<para>
177+
Extensions may define custom resource managers, but pg_waldump does
178+
not load the extension module and therefore does not recognize custom
179+
resource managers by name. Instead, you can specify the custom
180+
resource managers as <literal>custom###</literal> where
181+
"<literal>###</literal>" is the three-digit resource manager ID. Names
182+
of this form will always be considered valid.
183+
</para>
176184
</listitem>
177185
</varlistentry>
178186

‎src/backend/access/transam/rmgr.c

Lines changed: 123 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,16 +24,138 @@
2424
#include "commands/dbcommands_xlog.h"
2525
#include "commands/sequence.h"
2626
#include "commands/tablespace.h"
27+
#include "fmgr.h"
28+
#include "funcapi.h"
29+
#include "miscadmin.h"
2730
#include "replication/decode.h"
2831
#include "replication/message.h"
2932
#include "replication/origin.h"
3033
#include "storage/standby.h"
34+
#include "utils/builtins.h"
3135
#include "utils/relmapper.h"
3236

3337
/* must be kept in sync with RmgrData definition in xlog_internal.h */
3438
#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,decode) \
3539
{ name, redo, desc, identify, startup, cleanup, mask, decode },
3640

37-
const RmgrData RmgrTable[RM_MAX_ID + 1] = {
41+
RmgrData RmgrTable[RM_MAX_ID + 1] = {
3842
#include "access/rmgrlist.h"
3943
};
44+
45+
/*
46+
* Start up all resource managers.
47+
*/
48+
void
49+
RmgrStartup(void)
50+
{
51+
for (int rmid = 0; rmid <= RM_MAX_ID; rmid++)
52+
{
53+
if (!RmgrIdExists(rmid))
54+
continue;
55+
56+
if (RmgrTable[rmid].rm_startup != NULL)
57+
RmgrTable[rmid].rm_startup();
58+
}
59+
}
60+
61+
/*
62+
* Clean up all resource managers.
63+
*/
64+
void
65+
RmgrCleanup(void)
66+
{
67+
for (int rmid = 0; rmid <= RM_MAX_ID; rmid++)
68+
{
69+
if (!RmgrIdExists(rmid))
70+
continue;
71+
72+
if (RmgrTable[rmid].rm_cleanup != NULL)
73+
RmgrTable[rmid].rm_cleanup();
74+
}
75+
}
76+
77+
/*
78+
* Emit ERROR when we encounter a record with an RmgrId we don't
79+
* recognize.
80+
*/
81+
void
82+
RmgrNotFound(RmgrId rmid)
83+
{
84+
ereport(ERROR, (errmsg("resource manager with ID %d not registered", rmid),
85+
errhint("Include the extension module that implements this resource manager in shared_preload_libraries.")));
86+
}
87+
88+
/*
89+
* Register a new custom WAL resource manager.
90+
*
91+
* Resource manager IDs must be globally unique across all extensions. Refer
92+
* to https://wiki.postgresql.org/wiki/CustomWALResourceManager to reserve a
93+
* unique RmgrId for your extension, to avoid conflicts with other extension
94+
* developers. During development, use RM_EXPERIMENTAL_ID to avoid needlessly
95+
* reserving a new ID.
96+
*/
97+
void
98+
RegisterCustomRmgr(RmgrId rmid, RmgrData *rmgr)
99+
{
100+
if (rmgr->rm_name == NULL || strlen(rmgr->rm_name) == 0)
101+
ereport(ERROR, (errmsg("custom resource manager name is invalid"),
102+
errhint("Provide a non-empty name for the custom resource manager.")));
103+
104+
if (!RMID_IS_CUSTOM(rmid))
105+
ereport(ERROR, (errmsg("custom resource manager ID %d is out of range", rmid),
106+
errhint("Provide a custom resource manager ID between %d and %d.",
107+
RM_MIN_CUSTOM_ID, RM_MAX_CUSTOM_ID)));
108+
109+
if (!process_shared_preload_libraries_in_progress)
110+
ereport(ERROR,
111+
(errmsg("failed to register custom resource manager \"%s\" with ID %d", rmgr->rm_name, rmid),
112+
errdetail("Custom resource manager must be registered while initializing modules in shared_preload_libraries.")));
113+
114+
if (RmgrTable[rmid].rm_name != NULL)
115+
ereport(ERROR,
116+
(errmsg("failed to register custom resource manager \"%s\" with ID %d", rmgr->rm_name, rmid),
117+
errdetail("Custom resource manager \"%s\" already registered with the same ID.",
118+
RmgrTable[rmid].rm_name)));
119+
120+
/* check for existing rmgr with the same name */
121+
for (int existing_rmid = 0; existing_rmid <= RM_MAX_ID; existing_rmid++)
122+
{
123+
if (!RmgrIdExists(existing_rmid))
124+
continue;
125+
126+
if (!pg_strcasecmp(RmgrTable[existing_rmid].rm_name, rmgr->rm_name))
127+
ereport(ERROR,
128+
(errmsg("failed to register custom resource manager \"%s\" with ID %d", rmgr->rm_name, rmid),
129+
errdetail("Existing resource manager with ID %d has the same name.", existing_rmid)));
130+
}
131+
132+
/* register it */
133+
RmgrTable[rmid] = *rmgr;
134+
ereport(LOG,
135+
(errmsg("registered custom resource manager \"%s\" with ID %d",
136+
rmgr->rm_name, rmid)));
137+
}
138+
139+
/* SQL SRF showing loaded resource managers */
140+
Datum
141+
pg_get_wal_resource_managers(PG_FUNCTION_ARGS)
142+
{
143+
#define PG_GET_RESOURCE_MANAGERS_COLS 3
144+
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
145+
Datum values[PG_GET_RESOURCE_MANAGERS_COLS];
146+
bool nulls[PG_GET_RESOURCE_MANAGERS_COLS] = {0};
147+
148+
SetSingleFuncCall(fcinfo, 0);
149+
150+
for (int rmid = 0; rmid <= RM_MAX_ID; rmid++)
151+
{
152+
if (!RmgrIdExists(rmid))
153+
continue;
154+
values[0] = Int32GetDatum(rmid);
155+
values[1] = CStringGetTextDatum(GetRmgr(rmid).rm_name);
156+
values[2] = BoolGetDatum(RMID_IS_BUILTIN(rmid));
157+
tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
158+
}
159+
160+
return (Datum) 0;
161+
}

‎src/backend/access/transam/xlogreader.c

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1102,7 +1102,7 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr,
11021102
(uint32) SizeOfXLogRecord, record->xl_tot_len);
11031103
return false;
11041104
}
1105-
if (record->xl_rmid > RM_MAX_ID)
1105+
if (!RMID_IS_VALID(record->xl_rmid))
11061106
{
11071107
report_invalid_record(state,
11081108
"invalid resource manager ID %u at %X/%X",

0 commit comments

Comments
 (0)