Shared memory overflow #4
Comments
kelvich
added a commit
that referenced
this issue
Oct 17, 2016
kelvich
added a commit
that referenced
this issue
Oct 21, 2016
kelvich
added a commit
that referenced
this issue
Nov 13, 2017
kelvich
added a commit
that referenced
this issue
Nov 13, 2017
kelvich
pushed a commit
that referenced
this issue
May 2, 2018
refresh_by_match_merge() has some issues in the way it builds a SQL query to construct the "diff" table: 1. It doesn't require the selected unique index(es) to be indimmediate. 2. It doesn't pay attention to the particular equality semantics enforced by a given index, but just assumes that they must be those of the column datatype's default btree opclass. 3. It doesn't check that the indexes are btrees. 4. It's insufficiently careful to ensure that the parser will pick the intended operator when parsing the query. (This would have been a security bug before CVE-2018-1058.) 5. It's not careful about indexes on system columns. The way to fix #4 is to make use of the existing code in ri_triggers.c for generating an arbitrary binary operator clause. I chose to move that to ruleutils.c, since that seems a more reasonable place to be exporting such functionality from than ri_triggers.c. While #1, #3, and #5 are just latent given existing feature restrictions, and #2 doesn't arise in the core system for lack of alternate opclasses with different equality behaviors, #4 seems like an issue worth back-patching. That's the bulk of the change anyway, so just back-patch the whole thing to 9.4 where this code was introduced. Discussion: https://postgr.es/m/13836.1521413227@sss.pgh.pa.us
arssher
added a commit
that referenced
this issue
Jul 18, 2018
This is the commit message #2: Tracking on which foreign servers we already started COPY. If foreign server holds several partitions, COPY FROM to local root partition will try to perform several copies at the same time through one connection, obviously without much success. Now we track that and start/end COPY only once. We also allow to pass destination relation name which may be different from foreing table -- so we can copy into foreign root partition in shardman. This is pretty narrow solution. However, keeping several connections to the same foreign server requires significant changes, especially in 2pc handling, so staying here for now. This is the commit message #3: Allow COPY FROM to par8d table even if some FDW parts can't do that. This behaviour was broken in patches allowing COPY FROM to FDW tables. This is the commit message #4: COPY FROM deparse more complete, PG_SHARDMAN macro. Now column names, FORCE NULL and FORCE NOT NULL are deparsed too. PG_SHARDMAN macro ensures this PG contains patches for Postgres. This is the commit message #5: Disable COPY FROM to foreign parts, because no generic impl exists. This is the commit message #6: Fix COPY FROM deparse, forgotten comma for FORCE_NULL etc.
arssher
added a commit
that referenced
this issue
Jul 19, 2018
Cherry-picked from af234df. This is the commit message #2: Tracking on which foreign servers we already started COPY. If foreign server holds several partitions, COPY FROM to local root partition will try to perform several copies at the same time through one connection, obviously without much success. Now we track that and start/end COPY only once. We also allow to pass destination relation name which may be different from foreing table -- so we can copy into foreign root partition in shardman. This is pretty narrow solution. However, keeping several connections to the same foreign server requires significant changes, especially in 2pc handling, so staying here for now. This is the commit message #3: Allow COPY FROM to par8d table even if some FDW parts can't do that. This behaviour was broken in patches allowing COPY FROM to FDW tables. This is the commit message #4: COPY FROM deparse more complete, PG_SHARDMAN macro. Now column names, FORCE NULL and FORCE NOT NULL are deparsed too. PG_SHARDMAN macro ensures this PG contains patches for Postgres. This is the commit message #5: Disable COPY FROM to foreign parts, because no generic impl exists. This is the commit message #6: Fix COPY FROM deparse, forgotten comma for FORCE_NULL etc.
arssher
added a commit
that referenced
this issue
Sep 17, 2018
Cherry-picked from af234df. This is the commit message #2: Tracking on which foreign servers we already started COPY. If foreign server holds several partitions, COPY FROM to local root partition will try to perform several copies at the same time through one connection, obviously without much success. Now we track that and start/end COPY only once. We also allow to pass destination relation name which may be different from foreing table -- so we can copy into foreign root partition in shardman. This is pretty narrow solution. However, keeping several connections to the same foreign server requires significant changes, especially in 2pc handling, so staying here for now. This is the commit message #3: Allow COPY FROM to par8d table even if some FDW parts can't do that. This behaviour was broken in patches allowing COPY FROM to FDW tables. This is the commit message #4: COPY FROM deparse more complete, PG_SHARDMAN macro. Now column names, FORCE NULL and FORCE NOT NULL are deparsed too. PG_SHARDMAN macro ensures this PG contains patches for Postgres. This is the commit message #5: Disable COPY FROM to foreign parts, because no generic impl exists. This is the commit message #6: Fix COPY FROM deparse, forgotten comma for FORCE_NULL etc.
arssher
added a commit
that referenced
this issue
Oct 5, 2018
Cherry-picked from af234df. This is the commit message #2: Tracking on which foreign servers we already started COPY. If foreign server holds several partitions, COPY FROM to local root partition will try to perform several copies at the same time through one connection, obviously without much success. Now we track that and start/end COPY only once. We also allow to pass destination relation name which may be different from foreing table -- so we can copy into foreign root partition in shardman. This is pretty narrow solution. However, keeping several connections to the same foreign server requires significant changes, especially in 2pc handling, so staying here for now. This is the commit message #3: Allow COPY FROM to par8d table even if some FDW parts can't do that. This behaviour was broken in patches allowing COPY FROM to FDW tables. This is the commit message #4: COPY FROM deparse more complete, PG_SHARDMAN macro. Now column names, FORCE NULL and FORCE NOT NULL are deparsed too. PG_SHARDMAN macro ensures this PG contains patches for Postgres. This is the commit message #5: Disable COPY FROM to foreign parts, because no generic impl exists. This is the commit message #6: Fix COPY FROM deparse, forgotten comma for FORCE_NULL etc.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There is a problem with "frozen" procArray->replication_slot_xmin.
This variable is adjusted by ProcArraySetReplicationSlotXmin which is invoked by ReplicationSlotsComputeRequiredXmin, which
is in turn is called by LogicalConfirmReceivedLocation. If transactions are executed at all nodes of multimaster, then everything works fine: replication_slot_xmin is advanced. But if we send transactions only to one multimaster node and broadcast this changes to other nodes, then no data is send through replications slot at this nodes. No data sends - no confirmations, LogicalConfirmReceivedLocation is not called and procArray->replication_slot_xmin preserves original value 599.
As a result GetOldestXmin function always returns 599, so autovacuum is actually blocked and our multimaster is not able to perform cleanup of XID->CSN map, which cause shared memory overflow. This situation happens only when write transactions are sent only to one node or if there are no write transactions at all.
The text was updated successfully, but these errors were encountered: