Support quorum-based synchronous replication.

This feature is also known as "quorum commit" especially in discussion
on pgsql-hackers.

This commit adds the following new syntaxes into synchronous_standby_names
GUC. By using FIRST and ANY keywords, users can specify the method to
choose synchronous standbys from the listed servers.

  FIRST num_sync (standby_name [, ...])
  ANY num_sync (standby_name [, ...])

The keyword FIRST specifies a priority-based synchronous replication
which was available also in 9.6 or before. This method makes transaction
commits wait until their WAL records are replicated to num_sync
synchronous standbys chosen based on their priorities.

The keyword ANY specifies a quorum-based synchronous replication
and makes transaction commits wait until their WAL records are
replicated to *at least* num_sync listed standbys. In this method,
the values of sync_state.pg_stat_replication for the listed standbys
are reported as "quorum". The priority is still assigned to each standby,
but not used in this method.

The existing syntaxes having neither FIRST nor ANY keyword are still
supported. They are the same as new syntax with FIRST keyword, i.e.,
a priorirty-based synchronous replication.

Author: Masahiko Sawada
Reviewed-By: Michael Paquier, Amit Kapila and me
Discussion: <CAD21AoAACi9NeC_ecm+Vahm+MMA6nYh=Kqs3KB3np+MBOS_gZg@mail.gmail.com>

Many thanks to the various individuals who were involved in
discussing and developing this feature.

Author: MasaoFujii

doc/src/sgml/config.sgml

@@ -3054,41 +3054,71 @@ include_dir 'conf.d'
         transactions waiting for commit will be allowed to proceed after
         these standby servers confirm receipt of their data.
         The synchronous standbys will be those whose names appear
-        earlier in this list, and
+        in this list, and
         that are both currently connected and streaming data in real-time
         (as shown by a state of <literal>streaming</literal> in the
         <link linkend="monitoring-stats-views-table">
         <literal>pg_stat_replication</></link> view).
-        Other standby servers appearing later in this list represent potential
-        synchronous standbys. If any of the current synchronous
-        standbys disconnects for whatever reason,
-        it will be replaced immediately with the next-highest-priority standby.
-        Specifying more than one standby name can allow very high availability.
+        Specifying more than one standby names can allow very high availability.
        </para>
        <para>
         This parameter specifies a list of standby servers using
         either of the following syntaxes:
 <synopsis>
-<replaceable class="parameter">num_sync</replaceable> ( <replaceable class="parameter">standby_name</replaceable> [, ...] )
+[FIRST] <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="parameter">standby_name</replaceable> [, ...] )
+ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="parameter">standby_name</replaceable> [, ...] )
 <replaceable class="parameter">standby_name</replaceable> [, ...]
 </synopsis>
         where <replaceable class="parameter">num_sync</replaceable> is
         the number of synchronous standbys that transactions need to
         wait for replies from,
         and <replaceable class="parameter">standby_name</replaceable>
-        is the name of a standby server. For example, a setting of
-        <literal>3 (s1, s2, s3, s4)</> makes transaction commits wait
-        until their WAL records are received by three higher-priority standbys
-        chosen from standby servers <literal>s1</>, <literal>s2</>,
-        <literal>s3</> and <literal>s4</>.
-        </para>
-        <para>
-        The second syntax was used before <productname>PostgreSQL</>
+        is the name of a standby server.
+        <literal>FIRST</> and <literal>ANY</> specify the method to choose
+        synchronous standbys from the listed servers.
+       </para>
+       <para>
+        The keyword <literal>FIRST</>, coupled with
+        <replaceable class="parameter">num_sync</replaceable>, specifies a
+        priority-based synchronous replication and makes transaction commits
+        wait until their WAL records are replicated to
+        <replaceable class="parameter">num_sync</replaceable> synchronous
+        standbys chosen based on their priorities. For example, a setting of
+        <literal>FIRST 3 (s1, s2, s3, s4)</> will cause each commit to wait for
+        replies from three higher-priority standbys chosen from standby servers
+        <literal>s1</>, <literal>s2</>, <literal>s3</> and <literal>s4</>.
+        The standbys whose names appear earlier in the list are given higher
+        priority and will be considered as synchronous. Other standby servers
+        appearing later in this list represent potential synchronous standbys.
+        If any of the current synchronous standbys disconnects for whatever
+        reason, it will be replaced immediately with the next-highest-priority
+        standby. The keyword <literal>FIRST</> is optional.
+       </para>
+       <para>
+        The keyword <literal>ANY</>, coupled with
+        <replaceable class="parameter">num_sync</replaceable>, specifies a
+        quorum-based synchronous replication and makes transaction commits
+        wait until their WAL records are replicated to <emphasis>at least</>
+        <replaceable class="parameter">num_sync</replaceable> listed standbys.
+        For example, a setting of <literal>ANY 3 (s1, s2, s3, s4)</> will cause
+        each commit to proceed as soon as at least any three standbys of
+        <literal>s1</>, <literal>s2</>, <literal>s3</> and <literal>s4</>
+        reply.
+       </para>
+       <para>
+        <literal>FIRST</> and <literal>ANY</> are case-insensitive. If these
+        keywords are used as the name of a standby server,
+        its <replaceable class="parameter">standby_name</replaceable> must
+        be double-quoted.
+       </para>
+       <para>
+        The third syntax was used before <productname>PostgreSQL</>
         version 9.6 and is still supported. It's the same as the first syntax
-        with <replaceable class="parameter">num_sync</replaceable> equal to 1.
-        For example, <literal>1 (s1, s2)</> and
-        <literal>s1, s2</> have the same meaning: either <literal>s1</>
-        or <literal>s2</> is chosen as a synchronous standby.
+        with <literal>FIRST</> and
+        <replaceable class="parameter">num_sync</replaceable> equal to 1.
+        For example, <literal>FIRST 1 (s1, s2)</> and <literal>s1, s2</> have
+        the same meaning: either <literal>s1</> or <literal>s2</> is chosen
+        as a synchronous standby.
        </para>
        <para>
         The name of a standby server for this purpose is the

doc/src/sgml/high-availability.sgml

@@ -1138,19 +1138,25 @@ primary_slot_name = 'node_a_slot'
     as synchronous confirm receipt of their data. The number of synchronous
     standbys that transactions must wait for replies from is specified in
     <varname>synchronous_standby_names</>. This parameter also specifies
-    a list of standby names, which determines the priority of each standby
-    for being chosen as a synchronous standby. The standbys whose names
-    appear earlier in the list are given higher priority and will be considered
-    as synchronous. Other standby servers appearing later in this list
-    represent potential synchronous standbys. If any of the current
-    synchronous standbys disconnects for whatever reason, it will be replaced
-    immediately with the next-highest-priority standby.
+    a list of standby names and the method (<literal>FIRST</> and
+    <literal>ANY</>) to choose synchronous standbys from the listed ones.
    </para>
    <para>
-    An example of <varname>synchronous_standby_names</> for multiple
-    synchronous standbys is:
+    The method <literal>FIRST</> specifies a priority-based synchronous
+    replication and makes transaction commits wait until their WAL records are
+    replicated to the requested number of synchronous standbys chosen based on
+    their priorities. The standbys whose names appear earlier in the list are
+    given higher priority and will be considered as synchronous. Other standby
+    servers appearing later in this list represent potential synchronous
+    standbys. If any of the current synchronous standbys disconnects for
+    whatever reason, it will be replaced immediately with the
+    next-highest-priority standby.
+   </para>
+   <para>
+    An example of <varname>synchronous_standby_names</> for
+    a priority-based multiple synchronous standbys is:
 <programlisting>
-synchronous_standby_names = '2 (s1, s2, s3)'
+synchronous_standby_names = 'FIRST 2 (s1, s2, s3)'
 </programlisting>
     In this example, if four standby servers <literal>s1</>, <literal>s2</>,
     <literal>s3</> and <literal>s4</> are running, the two standbys
@@ -1161,6 +1167,24 @@ synchronous_standby_names = '2 (s1, s2, s3)'
     <literal>s2</> fails. <literal>s4</> is an asynchronous standby since
     its name is not in the list.
    </para>
+   <para>
+    The method <literal>ANY</> specifies a quorum-based synchronous
+    replication and makes transaction commits wait until their WAL records
+    are replicated to <emphasis>at least</> the requested number of
+    synchronous standbys in the list.
+   </para>
+   <para>
+    An example of <varname>synchronous_standby_names</> for
+    a quorum-based multiple synchronous standbys is:
+<programlisting>
+ synchronous_standby_names = 'ANY 2 (s1, s2, s3)'
+</programlisting>
+    In this example, if four standby servers <literal>s1</>, <literal>s2</>,
+    <literal>s3</> and <literal>s4</> are running, transaction commits will
+    wait for replies from at least any two standbys of <literal>s1</>,
+    <literal>s2</> and <literal>s3</>. <literal>s4</> is an asynchronous
+    standby since its name is not in the list.
+   </para>
    <para>
     The synchronous states of standby servers can be viewed using
     the <structname>pg_stat_replication</structname> view.

doc/src/sgml/monitoring.sgml

@@ -1404,7 +1404,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
      <entry><structfield>sync_priority</></entry>
      <entry><type>integer</></entry>
      <entry>Priority of this standby server for being chosen as the
-      synchronous standby</entry>
+      synchronous standby in a priority-based synchronous replication.
+      This has no effect in a quorum-based synchronous replication.</entry>
     </row>
     <row>
      <entry><structfield>sync_state</></entry>
@@ -1429,6 +1430,12 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
            <literal>sync</>: This standby server is synchronous.
           </para>
          </listitem>
+         <listitem>
+          <para>
+           <literal>quorum</>: This standby server is considered as a candidate
+           for quorum standbys.
+          </para>
+         </listitem>
        </itemizedlist>
      </entry>
     </row>

src/backend/replication/Makefile

@@ -26,7 +26,7 @@ repl_gram.o: repl_scanner.c
 
 # syncrep_scanner is complied as part of syncrep_gram
 syncrep_gram.o: syncrep_scanner.c
-syncrep_scanner.c: FLEXFLAGS = -CF -p
+syncrep_scanner.c: FLEXFLAGS = -CF -p -i
 syncrep_scanner.c: FLEX_NO_BACKUP=yes
 
 # repl_gram.c, repl_scanner.c, syncrep_gram.c and syncrep_scanner.c