Just-in-time background writing strategy. This code avoids re-scanning

buffers that cannot possibly need to be cleaned, and estimates how many
buffers it should try to clean based on moving averages of recent allocation
requests and density of reusable buffers.  The patch also adds a couple
more columns to pg_stat_bgwriter to help measure the effectiveness of the
bgwriter.

Greg Smith, building on his own work and ideas from several other people,
in particular a much older patch from Itagaki Takahiro.

Author: tglsfdc

doc/src/sgml/config.sgml

@@ -1,4 +1,4 @@
-<!-- $PostgreSQL: pgsql/doc/src/sgml/config.sgml,v 1.146 2007/09/24 03:12:23 tgl Exp $ -->
+<!-- $PostgreSQL: pgsql/doc/src/sgml/config.sgml,v 1.147 2007/09/25 20:03:37 tgl Exp $ -->
 
 <chapter Id="runtime-config">
   <title>Server Configuration</title>
@@ -1205,35 +1205,42 @@ SET ENABLE_SEQSCAN TO OFF;
        </listitem>
       </varlistentry>
 
-      <varlistentry id="guc-bgwriter-lru-percent" xreflabel="bgwriter_lru_percent">
-       <term><varname>bgwriter_lru_percent</varname> (<type>floating point</type>)</term>
+      <varlistentry id="guc-bgwriter-lru-maxpages" xreflabel="bgwriter_lru_maxpages">
+       <term><varname>bgwriter_lru_maxpages</varname> (<type>integer</type>)</term>
        <indexterm>
-        <primary><varname>bgwriter_lru_percent</> configuration parameter</primary>
+        <primary><varname>bgwriter_lru_maxpages</> configuration parameter</primary>
        </indexterm>
        <listitem>
         <para>
-         To reduce the probability that server processes will need to issue
-         their own writes, the background writer tries to write buffers that
-         are likely to be recycled soon.  In each round, it examines up to
-         <varname>bgwriter_lru_percent</> of the buffers that are nearest to
-         being recycled, and writes any that are dirty.
-         The default value is 1.0 (1% of the total number of shared buffers).
+         In each round, no more than this many buffers will be written
+         by the background writer.  Setting this to zero disables
+         background writing (except for checkpoint activity).
+         The default value is 100 buffers.
          This parameter can only be set in the <filename>postgresql.conf</>
          file or on the server command line.
         </para>
        </listitem>
       </varlistentry>
 
-      <varlistentry id="guc-bgwriter-lru-maxpages" xreflabel="bgwriter_lru_maxpages">
-       <term><varname>bgwriter_lru_maxpages</varname> (<type>integer</type>)</term>
+      <varlistentry id="guc-bgwriter-lru-multiplier" xreflabel="bgwriter_lru_multiplier">
+       <term><varname>bgwriter_lru_multiplier</varname> (<type>floating point</type>)</term>
        <indexterm>
-        <primary><varname>bgwriter_lru_maxpages</> configuration parameter</primary>
+        <primary><varname>bgwriter_lru_multiplier</> configuration parameter</primary>
        </indexterm>
        <listitem>
         <para>
-         In each round, no more than this many buffers will be written
-         as a result of scanning soon-to-be-recycled buffers.
-         The default value is five buffers.
+         Unless limited by <varname>bgwriter_lru_maxpages</>, the number
+         of dirty buffers written in each round is determined by reference
+         to the number of new buffers that have been needed by server
+         processes during recent rounds.  This number is multiplied by
+         <varname>bgwriter_lru_multiplier</> to arrive at the estimate
+         of the number of buffers that will be needed during the next round.
+         Thus, a setting of 1.0 represents a <quote>just in time</> policy
+         of writing exactly the number of buffers predicted to be needed.
+         Larger values provide some cushion against spikes in demand,
+         while smaller values intentionally leave writes to be done by
+         server processes.
+         The default is 2.0.
          This parameter can only be set in the <filename>postgresql.conf</>
          file or on the server command line.
         </para>
@@ -1242,8 +1249,8 @@ SET ENABLE_SEQSCAN TO OFF;
      </variablelist>
 
      <para>
-      Smaller values of <varname>bgwriter_lru_percent</varname> and
-      <varname>bgwriter_lru_maxpages</varname> reduce the extra I/O load
+      Smaller values of <varname>bgwriter_lru_maxpages</varname> and
+      <varname>bgwriter_lru_multiplier</varname> reduce the extra I/O load
       caused by the background writer, but make it more likely that server
       processes will have to issue writes for themselves, delaying interactive
       queries.

doc/src/sgml/monitoring.sgml

@@ -1,4 +1,4 @@
-<!-- $PostgreSQL: pgsql/doc/src/sgml/monitoring.sgml,v 1.53 2007/09/24 03:12:23 tgl Exp $ -->
+<!-- $PostgreSQL: pgsql/doc/src/sgml/monitoring.sgml,v 1.54 2007/09/25 20:03:37 tgl Exp $ -->
 
 <chapter id="monitoring">
  <title>Monitoring Database Activity</title>
@@ -237,7 +237,10 @@ postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
       background writer: number of scheduled checkpoints, requested
       checkpoints, buffers written by checkpoints and cleaning scans,
       and the number of times the bgwriter stopped a cleaning scan
-      because it had written too many buffers.
+      because it had written too many buffers.  Also includes
+      statistics about the shared buffer pool, including buffers written
+      by backends (that is, not by the background writer) and total buffers
+      allocated.
      </entry>
      </row>
 
@@ -817,6 +820,23 @@ postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re
       </entry>
      </row>
 
+     <row>
+      <entry><literal><function>pg_stat_get_buf_written_backend</function>()</literal></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       The number of buffers written by backends because they needed
+       to allocate a new buffer
+      </entry>
+     </row>
+
+     <row>
+      <entry><literal><function>pg_stat_get_buf_alloc</function>()</literal></entry>
+      <entry><type>bigint</type></entry>
+      <entry>
+       The total number of buffer allocations
+      </entry>
+     </row>
+
      <row>
       <entry><literal><function>pg_stat_clear_snapshot</function>()</literal></entry>
       <entry><type>void</type></entry>

src/backend/catalog/system_views.sql

@@ -3,7 +3,7 @@
  *
  * Copyright (c) 1996-2007, PostgreSQL Global Development Group
  *
- * $PostgreSQL: pgsql/src/backend/catalog/system_views.sql,v 1.45 2007/09/20 17:56:30 tgl Exp $
+ * $PostgreSQL: pgsql/src/backend/catalog/system_views.sql,v 1.46 2007/09/25 20:03:37 tgl Exp $
  */
 
 CREATE VIEW pg_roles AS 
@@ -382,7 +382,9 @@ CREATE VIEW pg_stat_bgwriter AS
         pg_stat_get_bgwriter_requested_checkpoints() AS checkpoints_req,
         pg_stat_get_bgwriter_buf_written_checkpoints() AS buffers_checkpoint,
         pg_stat_get_bgwriter_buf_written_clean() AS buffers_clean,
-        pg_stat_get_bgwriter_maxwritten_clean() AS maxwritten_clean;
+        pg_stat_get_bgwriter_maxwritten_clean() AS maxwritten_clean,
+        pg_stat_get_buf_written_backend() AS buffers_backend,
+        pg_stat_get_buf_alloc() AS buffers_alloc;
 
 -- Tsearch debug function. Defined here because it'd be pretty unwieldy
 -- to put it into pg_proc.h

src/backend/postmaster/bgwriter.c

@@ -37,7 +37,7 @@
  *
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/postmaster/bgwriter.c,v 1.43 2007/09/16 16:33:04 tgl Exp $
+ *	  $PostgreSQL: pgsql/src/backend/postmaster/bgwriter.c,v 1.44 2007/09/25 20:03:37 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -98,8 +98,14 @@
  * requesting backends since the last checkpoint start.  The flags are
  * chosen so that OR'ing is the correct way to combine multiple requests.
  *
+ * num_backend_writes is used to count the number of buffer writes performed
+ * by non-bgwriter processes.  This counter should be wide enough that it
+ * can't overflow during a single bgwriter cycle.
+ *
  * The requests array holds fsync requests sent by backends and not yet
- * absorbed by the bgwriter.  Unlike the checkpoint fields, the requests
+ * absorbed by the bgwriter.
+ *
+ * Unlike the checkpoint fields, num_backend_writes and the requests
  * fields are protected by BgWriterCommLock.
  *----------
  */
@@ -122,6 +128,8 @@ typedef struct
 
 	int			ckpt_flags;		/* checkpoint flags, as defined in xlog.h */
 
+	uint32		num_backend_writes;	/* counts non-bgwriter buffer writes */
+
 	int			num_requests;	/* current # of requests */
 	int			max_requests;	/* allocated array size */
 	BgWriterRequest requests[1];	/* VARIABLE LENGTH ARRAY */
@@ -566,8 +574,7 @@ BgWriterNap(void)
 	 *
 	 * We absorb pending requests after each short sleep.
 	 */
-	if ((bgwriter_lru_percent > 0.0 && bgwriter_lru_maxpages > 0) ||
-		ckpt_active)
+	if (bgwriter_lru_maxpages > 0 || ckpt_active)
 		udelay = BgWriterDelay * 1000L;
 	else if (XLogArchiveTimeout > 0)
 		udelay = 1000000L;	/* One second */
@@ -648,12 +655,13 @@ CheckpointWriteDelay(int flags, double progress)
 			got_SIGHUP = false;
 			ProcessConfigFile(PGC_SIGHUP);
 		}
-		BgBufferSync();
-		CheckArchiveTimeout();
-		BgWriterNap();
 
 		AbsorbFsyncRequests();
 		absorb_counter = WRITES_PER_ABSORB;
+
+		BgBufferSync();
+		CheckArchiveTimeout();
+		BgWriterNap();
 	}
 	else if (--absorb_counter <= 0)
 	{
@@ -963,7 +971,8 @@ RequestCheckpoint(int flags)
  * Whenever a backend is compelled to write directly to a relation
  * (which should be seldom, if the bgwriter is getting its job done),
  * the backend calls this routine to pass over knowledge that the relation
- * is dirty and must be fsync'd before next checkpoint.
+ * is dirty and must be fsync'd before next checkpoint.  We also use this
+ * opportunity to count such writes for statistical purposes.
  *
  * segno specifies which segment (not block!) of the relation needs to be
  * fsync'd.  (Since the valid range is much less than BlockNumber, we can
@@ -987,7 +996,13 @@ ForwardFsyncRequest(RelFileNode rnode, BlockNumber segno)
 	if (!IsUnderPostmaster)
 		return false;			/* probably shouldn't even get here */
 
+	Assert(!am_bg_writer);
+
 	LWLockAcquire(BgWriterCommLock, LW_EXCLUSIVE);
+
+	/* we count non-bgwriter writes even when the request queue overflows */
+	BgWriterShmem->num_backend_writes++;
+
 	if (BgWriterShmem->bgwriter_pid == 0 ||
 		BgWriterShmem->num_requests >= BgWriterShmem->max_requests)
 	{
@@ -1035,6 +1050,10 @@ AbsorbFsyncRequests(void)
 	 */
 	LWLockAcquire(BgWriterCommLock, LW_EXCLUSIVE);
 
+	/* Transfer write count into pending pgstats message */
+	BgWriterStats.m_buf_written_backend += BgWriterShmem->num_backend_writes;
+	BgWriterShmem->num_backend_writes = 0;
+
 	n = BgWriterShmem->num_requests;
 	if (n > 0)
 	{

src/backend/postmaster/pgstat.c

@@ -13,7 +13,7 @@
  *
  *	Copyright (c) 2001-2007, PostgreSQL Global Development Group
  *
- *	$PostgreSQL: pgsql/src/backend/postmaster/pgstat.c,v 1.165 2007/09/24 03:12:23 tgl Exp $
+ *	$PostgreSQL: pgsql/src/backend/postmaster/pgstat.c,v 1.166 2007/09/25 20:03:37 tgl Exp $
  * ----------
  */
 #include "postgres.h"
@@ -3165,4 +3165,6 @@ pgstat_recv_bgwriter(PgStat_MsgBgWriter *msg, int len)
 	globalStats.buf_written_checkpoints += msg->m_buf_written_checkpoints;
 	globalStats.buf_written_clean += msg->m_buf_written_clean;
 	globalStats.maxwritten_clean += msg->m_maxwritten_clean;
+	globalStats.buf_written_backend += msg->m_buf_written_backend;
+	globalStats.buf_alloc += msg->m_buf_alloc;
 }