From cb92703384e2bb3fa0a690e5dbb95ad333c2b44c Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Tue, 8 Jun 2021 20:22:18 +0200 Subject: Adjust batch size in postgres_fdw to not use too many parameters The FE/BE protocol identifies parameters with an Int16 index, which limits the maximum number of parameters per query to 65535. With batching added to postges_fdw this limit is much easier to hit, as the whole batch is essentially a single query, making this error much easier to hit. The failures are a bit unpredictable, because it also depends on the number of columns in the query. So instead of just failing, this patch tweaks the batch_size to not exceed the maximum number of parameters. Reported-by: Hou Zhijie Reviewed-by: Bharath Rupireddy Discussion: https://postgr.es/m/OS0PR01MB571603973C0AC2874AD6BF2594299%40OS0PR01MB5716.jpnprd01.prod.outlook.com --- doc/src/sgml/postgres-fdw.sgml | 11 +++++++++++ 1 file changed, 11 insertions(+) (limited to 'doc/src/sgml/postgres-fdw.sgml') diff --git a/doc/src/sgml/postgres-fdw.sgml b/doc/src/sgml/postgres-fdw.sgml index 5aced083e9e..d96c3d0f0cd 100644 --- a/doc/src/sgml/postgres-fdw.sgml +++ b/doc/src/sgml/postgres-fdw.sgml @@ -372,6 +372,17 @@ OPTIONS (ADD password_required 'false'); overrides an option specified for the server. The default is 1. + + + Note the actual number of rows postgres_fdw inserts at + once depends on the number of columns and the provided + batch_size value. The batch is executed as a single + query, and the libpq protocol (which postgres_fdw + uses to connect to a remote server) limits the number of parameters in a + single query to 65535. When the number of columns * batch_size + exceeds the limit, the batch_size will be adjusted to + avoid an error. + -- cgit v1.2.3