Change planner to use the current true disk file size as its estimate of

a relation's number of blocks, rather than the possibly-obsolete value
in pg_class.relpages.  Scale the value in pg_class.reltuples correspondingly
to arrive at a hopefully more accurate number of rows.  When pg_class
contains 0/0, estimate a tuple width from the column datatypes and divide
that into current file size to estimate number of rows.  This improved
methodology allows us to jettison the ancient hacks that put bogus default
values into pg_class when a table is first created.  Also, per a suggestion
from Simon, make VACUUM (but not VACUUM FULL or ANALYZE) adjust the value
it puts into pg_class.reltuples to try to represent the mean tuple density
instead of the minimal density that actually prevails just after VACUUM.
These changes alter the plans selected for certain regression tests, so
update the expected files accordingly.  (I removed join_1.out because
it's not clear if it still applies; we can add back any variant versions
as they are shown to be needed.)

Author: tglsfdc

doc/src/sgml/catalogs.sgml

@@ -1,6 +1,6 @@
 <!--
  Documentation of the system catalogs, directed toward PostgreSQL developers
- $PostgreSQL: pgsql/doc/src/sgml/catalogs.sgml,v 2.92 2004/11/15 06:32:13 neilc Exp $
+ $PostgreSQL: pgsql/doc/src/sgml/catalogs.sgml,v 2.93 2004/12/01 19:00:27 tgl Exp $
  -->
 
 <chapter id="catalogs">
@@ -1079,11 +1079,12 @@
       <entry><type>int4</type></entry>
       <entry></entry>
       <entry>
-       Size of the on-disk representation of this table in pages (size
+       Size of the on-disk representation of this table in pages (of size
        <symbol>BLCKSZ</symbol>).
        This is only an estimate used by the planner.
        It is updated by <command>VACUUM</command>,
-       <command>ANALYZE</command>, and <command>CREATE INDEX</command>.
+       <command>ANALYZE</command>, and a few DDL commands
+       such as <command>CREATE INDEX</command>.
       </entry>
      </row>
 
@@ -1095,7 +1096,8 @@
        Number of rows in the table.
        This is only an estimate used by the planner.
        It is updated by <command>VACUUM</command>,
-       <command>ANALYZE</command>, and <command>CREATE INDEX</command>.
+       <command>ANALYZE</command>, and a few DDL commands
+       such as <command>CREATE INDEX</command>.
       </entry>
      </row>
 

doc/src/sgml/diskusage.sgml

@@ -1,17 +1,13 @@
 <!--
-$PostgreSQL: pgsql/doc/src/sgml/diskusage.sgml,v 1.11 2004/06/21 04:06:03 tgl Exp $
+$PostgreSQL: pgsql/doc/src/sgml/diskusage.sgml,v 1.12 2004/12/01 19:00:27 tgl Exp $
 -->
 
 <chapter id="diskusage">
  <title>Monitoring Disk Usage</title>
 
  <para>
   This chapter discusses how to monitor the disk usage of a
-  <productname>PostgreSQL</> database system.  In the current
-  release, the database administrator does not have much control over
-  the on-disk storage layout, so this chapter is mostly informative
-  and can give you some ideas how to manage the disk usage with
-  operating system tools.
+  <productname>PostgreSQL</> database system.
  </para>
 
  <sect1 id="disk-usage">
@@ -23,11 +19,12 @@ $PostgreSQL: pgsql/doc/src/sgml/diskusage.sgml,v 1.11 2004/06/21 04:06:03 tgl Ex
 
   <para>
    Each table has a primary heap disk file where most of the data is
-   stored. To store long column values, there is also a
-   <acronym>TOAST</> file associated with the table, named based on the
-   table's OID (actually <literal>pg_class.relfilenode</>), and an index on the
-   <acronym>TOAST</> table. There also may be indexes associated with
-   the base table.
+   stored. If the table has any columns with potentially-wide values,
+   there is also a <acronym>TOAST</> file associated with the table,
+   which is used to store values too wide to fit comfortably in the main
+   table.  There will be one index on the
+   <acronym>TOAST</> table, if present. There may also be indexes associated
+   with the base table.
   </para>
 
   <para>
@@ -45,18 +42,24 @@ SELECT relfilenode, relpages FROM pg_class WHERE relname = 'customer';
        16806 |       60
 (1 row)
 </programlisting>
-   Each page is typically 8 kilobytes. (Remember, <literal>relpages</>
-   is only updated by <command>VACUUM</> and <command>ANALYZE</>.)
+   Each page is typically 8 kilobytes. (Remember, <structfield>relpages</>
+   is only updated by <command>VACUUM</>, <command>ANALYZE</>, and
+   a few DDL commands such as <command>CREATE INDEX</>.)  The
+   <structfield>relfilenode</> value is of interest if you want to examine
+   the table's disk file directly.
   </para>
 
   <para>
    To show the space used by <acronym>TOAST</> tables, use a query
-   like the following, substituting the <literal>relfilenode</literal>
-   number of the heap (determined by the query above):
+   like the following:
 <programlisting>
 SELECT relname, relpages
-    FROM pg_class
-    WHERE relname = 'pg_toast_16806' OR relname = 'pg_toast_16806_index'
+    FROM pg_class,
+         (SELECT reltoastrelid FROM pg_class
+          WHERE relname = 'customer') ss
+    WHERE oid = ss.reltoastrelid
+       OR oid = (SELECT reltoastidxid FROM pg_class
+                 WHERE oid = ss.reltoastrelid)
     ORDER BY relname;
 
        relname        | relpages 
@@ -74,7 +77,7 @@ SELECT c2.relname, c2.relpages
     WHERE c.relname = 'customer'
         AND c.oid = i.indrelid
         AND c2.oid = i.indexrelid
-        ORDER BY c2.relname;
+    ORDER BY c2.relname;
 
        relname        | relpages 
 ----------------------+----------
@@ -113,13 +116,10 @@ SELECT relname, relpages FROM pg_class ORDER BY relpages DESC;
 
   <para>
    The most important disk monitoring task of a database administrator
-   is to make sure the disk doesn't grow full.  A filled data disk may
-   result in subsequent corruption of database indexes, but not of the
-   tables themselves. If the WAL files are on the same disk (as
-   is the case for a default configuration) then a filled disk during
-   database initialization may result in corrupted or incomplete WAL
-   files. This failure condition is detected and the database server
-   will refuse to start up.
+   is to make sure the disk doesn't grow full.  A filled data disk will
+   not result in data corruption, but it may well prevent useful activity
+   from occurring. If the disk holding the WAL files grows full, database
+   server panic and consequent shutdown may occur.
   </para>
 
   <para>

doc/src/sgml/perform.sgml

@@ -1,5 +1,5 @@
 <!--
-$PostgreSQL: pgsql/doc/src/sgml/perform.sgml,v 1.47 2004/11/15 06:32:14 neilc Exp $
+$PostgreSQL: pgsql/doc/src/sgml/perform.sgml,v 1.48 2004/12/01 19:00:27 tgl Exp $
 -->
 
  <chapter id="performance-tips">
@@ -389,14 +389,15 @@ SELECT relname, relkind, reltuples, relpages FROM pg_class WHERE relname LIKE 't
   <para>
    For efficiency reasons, <structfield>reltuples</structfield> 
    and <structfield>relpages</structfield> are not updated on-the-fly,
-   and so they usually contain only approximate values (which is good
-   enough for the planner's purposes).  They are initialized with dummy
-   values (presently 1000 and 10 respectively) when a table is created.
-   They are updated by certain commands, presently <command>VACUUM</>,
-   <command>ANALYZE</>, and <command>CREATE INDEX</>.  A stand-alone
+   and so they usually contain somewhat out-of-date values.
+   They are updated by <command>VACUUM</>, <command>ANALYZE</>, and a
+   few DDL commands such as <command>CREATE INDEX</>.  A stand-alone
    <command>ANALYZE</>, that is one not part of <command>VACUUM</>,
    generates an approximate <structfield>reltuples</structfield> value
-   since it does not read every row of the table.
+   since it does not read every row of the table.  The planner
+   will scale the values it finds in <structname>pg_class</structname>
+   to match the current physical table size, thus obtaining a closer
+   approximation.
   </para>
 
   <indexterm>

doc/src/sgml/release.sgml

@@ -1,5 +1,5 @@
 <!--
-$PostgreSQL: pgsql/doc/src/sgml/release.sgml,v 1.314 2004/11/27 21:56:04 petere Exp $
+$PostgreSQL: pgsql/doc/src/sgml/release.sgml,v 1.315 2004/12/01 19:00:27 tgl Exp $
 -->
 
 <appendix id="release">
@@ -527,6 +527,24 @@ $PostgreSQL: pgsql/doc/src/sgml/release.sgml,v 1.314 2004/11/27 21:56:04 petere
      </para>
     </listitem>
 
+    <listitem>
+     <para>
+      Use dynamically-generated table size estimates while planning (Tom)
+     </para>
+     <para>
+      The optimizer now uses a table's current actual size on disk as its
+      estimate of the number of blocks in the table, and it makes an estimate
+      of the number of rows in the table based on the current size on disk.
+      Formerly, the
+      <structname>pg_class</structname>.<structfield>relpages</structfield>
+      and
+      <structname>pg_class</structname>.<structfield>reltuples</structfield>
+      fields were used as-is, but these values might be quite out-of-date,
+      leading to poor choices of plans.  They are now treated only as an
+      indication of the table's density (rows per page).
+     </para>
+    </listitem>
+
     <listitem>
      <para>
       Improved index usage with <literal>OR</> clauses (Tom)

src/backend/access/nbtree/nbtree.c

@@ -12,7 +12,7 @@
  * Portions Copyright (c) 1994, Regents of the University of California
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/access/nbtree/nbtree.c,v 1.122 2004/11/17 03:13:38 neilc Exp $
+ *	  $PostgreSQL: pgsql/src/backend/access/nbtree/nbtree.c,v 1.123 2004/12/01 19:00:37 tgl Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -785,6 +785,10 @@ btvacuumcleanup(PG_FUNCTION_ARGS)
 			 * Do the physical truncation.
 			 */
 			RelationTruncate(rel, new_pages);
+
+			/* update statistics */
+			stats->pages_removed = num_pages - new_pages;
+
 			num_pages = new_pages;
 		}
 	}

src/backend/catalog/heap.c

@@ -8,7 +8,7 @@
  *
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/catalog/heap.c,v 1.276 2004/08/31 17:10:36 tgl Exp $
+ *	  $PostgreSQL: pgsql/src/backend/catalog/heap.c,v 1.277 2004/12/01 19:00:39 tgl Exp $
  *
  *
  * INTERFACE ROUTINES
@@ -607,37 +607,22 @@ AddNewRelationTuple(Relation pg_class_desc,
 	 */
 	new_rel_reltup = new_rel_desc->rd_rel;
 
-	/*
-	 * Here we insert bogus estimates of the size of the new relation. In
-	 * reality, of course, the new relation has 0 tuples and pages, and if
-	 * we were tracking these statistics accurately then we'd set the
-	 * fields that way.  But at present the stats will be updated only by
-	 * VACUUM or CREATE INDEX, and the user might insert a lot of tuples
-	 * before he gets around to doing either of those.	So, instead of
-	 * saying the relation is empty, we insert guesstimates.  The point is
-	 * to keep the optimizer from making really stupid choices on
-	 * never-yet-vacuumed tables; so the estimates need only be large
-	 * enough to discourage the optimizer from using nested-loop plans.
-	 * With this hack, nested-loop plans will be preferred only after the
-	 * table has been proven to be small by VACUUM or CREATE INDEX.
-	 * Maintaining the stats on-the-fly would solve the problem more
-	 * cleanly, but the overhead of that would likely cost more than it'd
-	 * save. (NOTE: CREATE INDEX inserts the same bogus estimates if it
-	 * finds the relation has 0 rows and pages. See index.c.)
-	 */
 	switch (relkind)
 	{
 		case RELKIND_RELATION:
 		case RELKIND_INDEX:
 		case RELKIND_TOASTVALUE:
-			new_rel_reltup->relpages = 10;		/* bogus estimates */
-			new_rel_reltup->reltuples = 1000;
+			/* The relation is real, but as yet empty */
+			new_rel_reltup->relpages = 0;
+			new_rel_reltup->reltuples = 0;
 			break;
 		case RELKIND_SEQUENCE:
+			/* Sequences always have a known size */
 			new_rel_reltup->relpages = 1;
 			new_rel_reltup->reltuples = 1;
 			break;
-		default:				/* views, etc */
+		default:
+			/* Views, etc, have no disk storage */
 			new_rel_reltup->relpages = 0;
 			new_rel_reltup->reltuples = 0;
 			break;

src/backend/catalog/index.c

@@ -8,7 +8,7 @@
  *
  *
  * IDENTIFICATION
- *	  $PostgreSQL: pgsql/src/backend/catalog/index.c,v 1.241 2004/10/15 22:39:53 tgl Exp $
+ *	  $PostgreSQL: pgsql/src/backend/catalog/index.c,v 1.242 2004/12/01 19:00:39 tgl Exp $
  *
  *
  * INTERFACE ROUTINES
@@ -33,35 +33,22 @@
 #include "catalog/index.h"
 #include "catalog/indexing.h"
 #include "catalog/pg_constraint.h"
-#include "catalog/pg_index.h"
 #include "catalog/pg_opclass.h"
-#include "catalog/pg_proc.h"
 #include "catalog/pg_type.h"
 #include "executor/executor.h"
 #include "miscadmin.h"
 #include "optimizer/clauses.h"
-#include "optimizer/prep.h"
 #include "parser/parse_expr.h"
-#include "parser/parse_func.h"
 #include "storage/sinval.h"
 #include "storage/smgr.h"
 #include "utils/builtins.h"
-#include "utils/catcache.h"
 #include "utils/fmgroids.h"
 #include "utils/inval.h"
 #include "utils/lsyscache.h"
 #include "utils/relcache.h"
 #include "utils/syscache.h"
 
 
-/*
- * macros used in guessing how many tuples are on a page.
- */
-#define AVG_ATTR_SIZE 8
-#define NTUPLES_PER_PAGE(natts) \
-	((BLCKSZ - MAXALIGN(sizeof(PageHeaderData))) / \
-	((natts) * AVG_ATTR_SIZE + MAXALIGN(sizeof(HeapTupleHeaderData))))
-
 /* non-export function prototypes */
 static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
 						 IndexInfo *indexInfo,
@@ -1153,6 +1140,8 @@ setNewRelfilenode(Relation relation)
 
 	/* update the pg_class row */
 	rd_rel->relfilenode = newrelfilenode;
+	rd_rel->relpages = 0;		/* it's empty until further notice */
+	rd_rel->reltuples = 0;
 	simple_heap_update(pg_class, &tuple->t_self, tuple);
 	CatalogUpdateIndexes(pg_class, tuple);
 
@@ -1170,7 +1159,7 @@ setNewRelfilenode(Relation relation)
  *
  * Update pg_class' relpages and reltuples statistics for the given relation
  * (which can be either a table or an index).  Note that this is not used
- * in the context of VACUUM.
+ * in the context of VACUUM, only CREATE INDEX.
  * ----------------
  */
 void
@@ -1209,7 +1198,8 @@ UpdateStats(Oid relid, double reltuples)
 	 * Find the tuple to update in pg_class.  Normally we make a copy of
 	 * the tuple using the syscache, modify it, and apply heap_update. But
 	 * in bootstrap mode we can't use heap_update, so we cheat and
-	 * overwrite the tuple in-place.
+	 * overwrite the tuple in-place.  (Note: as of PG 8.0 this isn't called
+	 * during bootstrap, but leave the code here for possible future use.)
 	 *
 	 * We also must cheat if reindexing pg_class itself, because the target
 	 * index may presently not be part of the set of indexes that
@@ -1245,45 +1235,14 @@ UpdateStats(Oid relid, double reltuples)
 		elog(ERROR, "could not find tuple for relation %u", relid);
 	rd_rel = (Form_pg_class) GETSTRUCT(tuple);
 
-	/*
-	 * Figure values to insert.
-	 *
-	 * If we found zero tuples in the scan, do NOT believe it; instead put a
-	 * bogus estimate into the statistics fields.  Otherwise, the common
-	 * pattern "CREATE TABLE; CREATE INDEX; insert data" leaves the table
-	 * with zero size statistics until a VACUUM is done.  The optimizer
-	 * will generate very bad plans if the stats claim the table is empty
-	 * when it is actually sizable.  See also CREATE TABLE in heap.c.
-	 *
-	 * Note: this path is also taken during bootstrap, because bootstrap.c
-	 * passes reltuples = 0 after loading a table.	We have to estimate
-	 * some number for reltuples based on the actual number of pages.
-	 */
-	relpages = RelationGetNumberOfBlocks(whichRel);
-
-	if (reltuples == 0)
-	{
-		if (relpages == 0)
-		{
-			/* Bogus defaults for a virgin table, same as heap.c */
-			reltuples = 1000;
-			relpages = 10;
-		}
-		else if (whichRel->rd_rel->relkind == RELKIND_INDEX && relpages <= 2)
-		{
-			/* Empty index, leave bogus defaults in place */
-			reltuples = 1000;
-		}
-		else
-			reltuples = ((double) relpages) * NTUPLES_PER_PAGE(whichRel->rd_rel->relnatts);
-	}
-
 	/*
 	 * Update statistics in pg_class, if they changed.	(Avoiding an
 	 * unnecessary update is not just a tiny performance improvement; it
 	 * also reduces the window wherein concurrent CREATE INDEX commands
 	 * may conflict.)
 	 */
+	relpages = RelationGetNumberOfBlocks(whichRel);
+
 	if (rd_rel->relpages != (int32) relpages ||
 		rd_rel->reltuples != (float4) reltuples)
 	{