Add description of new parallel regression testing method; update

procedure to cover running either test script; other minor improvements.

Author: tglsfdc

doc/src/sgml/regress.sgml

@@ -1,5 +1,5 @@
 <Chapter Id="regress">
-<Title>Regression Test</Title>
+<Title id="regress-title">Regression Test</Title>
 
 <Abstract>
 <Para>
@@ -13,21 +13,40 @@ Regression test instructions and analysis.
   operations as well as the extended capabilities of PostgreSQL.
 </Para>
 
+<Para>
+  There are two different ways in which the regression tests can be run:
+  the "sequential" method and the "parallel" method.  The sequential method
+  runs each test script in turn, whereas the parallel method starts up
+  multiple server processes to run groups of tests in parallel.  Parallel
+  testing gives confidence that interprocess communication and locking
+  are working correctly.  Another key difference is that the sequential
+  test procedure uses an already-installed postmaster, whereas the
+  parallel test procedure tests a system that has been built but not yet
+  installed.  (The parallel test script actually does an installation into
+  a temporary directory and fires up a private postmaster therein.)
+</Para>
+
+<Para>
+  Some properly installed and fully functional PostgreSQL installations
+  can "fail" some of these regression tests due to artifacts of floating point
+  representation and time zone support. The tests are currently evaluated
+  using a simple <application>diff</application> comparison against the
+  outputs generated on a reference system, so the results are sensitive to
+  small system differences.
+  When a test is reported as "failed", always examine the differences
+  between expected and actual results; you may well find that the differences
+  are not significant.
+</Para>
+
 <Para>
   The regression tests were originally developed by Jolly Chen and Andrew Yu,
   and were extensively revised/repackaged by Marc Fournier and Thomas Lockhart.
   From <ProductName>PostgreSQL</ProductName> v6.1 onward
   the regression tests are current for every official release. 
 </Para>
 
-<Para>
-  Some properly installed and fully functional PostgreSQL installations
-  can fail some of these regression tests due to artifacts of floating point
-  representation and time zone support. The current tests are evaluated
-  using a simple "diff" algorithm, and are sensitive to small system
-  differences. For apparently failed tests, examining the differences
-  may reveal that the differences are not significant.
-</Para>
+<Sect1>
+<Title>Regression Environment</Title>
 
 <Para>
 The regression testing notes below assume the following (except where noted):
@@ -60,22 +79,11 @@ The runtime path is /usr/local/pgsql (other paths are possible).
 </ItemizedList>
 </Para>
 
-<Sect1>
-<Title>Regression Environment</Title>
-
-<Para>
-  To prepare for regression testing, do <Command>make all</Command> in the regression test
-  directory.  This compiles a <Acronym>C</Acronym> program with PostgreSQL extension functions
-  into a shared library.  Localized SQL scripts and output-comparison
-  files are also created for the tests that need them.  The localization
-  replaces macros in the source files with absolute pathnames and user names.  
-</Para>
-
 <Para>
   Normally, the regression tests should be run as the postgres user since
   the 'src/test/regress' directory and sub-directories are owned by the
   postgres user. If you run the regression test as another user the
-  'src/test/regress' directory tree must be writeable to that user.
+  'src/test/regress' directory tree must be writeable by that user.
 </Para>
 
 <Para>
@@ -131,6 +139,8 @@ The runtime path is /usr/local/pgsql (other paths are possible).
 
   results/ .. .out files that contain what the results *actually* look
               like. Also used as temporary storage for table copy testing.
+
+  tmp_check/  temporary installation created by parallel testing script.
       </ProgramListing>
     </Para>
   </Sect1>
@@ -144,39 +154,48 @@ The runtime path is /usr/local/pgsql (other paths are possible).
       like <FileName>ps</FileName> and <FileName>tar</FileName> vary wildly on what options you should use on each
       platform. <Emphasis>Use common sense</Emphasis> before typing in these commands.
     </Para>
-      
-      <Para>
-	For a fresh install or upgrading from previous releases of
-	<ProductName>Postgres</ProductName>:
-      </Para>
 
     <Procedure>
-      <Title><ProductName>Postgres</ProductName> Regression Configuration</Title>
-
+      <Title><ProductName>Postgres</ProductName> Regression Test</Title>
+      
       <Step Performance="required">
 	<Para>
-	  The file /usr/src/pgsql/src/test/regress/README has detailed
-	  instructions for running and interpreting the regression tests.
-	  A short version follows here:
+	  Prepare the files needed for the regression test with:
+	  <ProgramListing>
+	    cd /usr/src/pgsql/src/test/regress
+	    gmake clean
+	    gmake all
+	  </ProgramListing>
+	  You can skip "gmake clean" if this is the first time you
+	  are running the tests.
+	</para>
+	<Para>
+	  This step compiles a <Acronym>C</Acronym>
+	  program with PostgreSQL extension functions into a shared library.
+	  Localized SQL scripts and output-comparison files are also created
+	  for the tests that need them.  The localization replaces macros in
+	  the source files with absolute pathnames and user names.
 	</Para>
-	
+      </step>
+
+      <Step Performance="optional">
 	<Para>
-	  If the postmaster is not already running, start the postmaster on an
-	  available window by typing
+	  If you intend to use the "sequential" test procedure, which tests
+	  an already-installed postmaster, be sure that the postmaster
+	  is running.  If it isn't already running,
+	  start the postmaster in an available window by typing
 	  <ProgramListing>
 	    postmaster
 	  </ProgramListing>
-	  
 	  or start the postmaster daemon running in the background by typing
 	  <ProgramListing>
 	    cd
 	    nohup postmaster > regress.log 2>&1 &
 	  </ProgramListing>
-	</Para>
-	
-	<Para>
-	  Run postmaster from your <ProductName>Postgres</ProductName> super user account (typically
-	  account postgres).
+	  The latter is probably preferable, since the regression test log
+	  will be quite lengthy (60K or so, in
+	  <ProductName>Postgres</ProductName> 7.0) and you might want to
+	  review it for clues if things go wrong.
 
 	  <Note>
 	    <Para>
@@ -186,68 +205,54 @@ The runtime path is /usr/local/pgsql (other paths are possible).
 	</Para>
       </Step>
 
-      <Step Performance="optional">
-	<Para>
-	  If you have previously invoked the regression test, clean up the
-	  working directory with:
-	  
-	  <ProgramListing>
-	    cd /usr/src/pgsql/src/test/regress
-	    gmake clean
-	  </ProgramListing>
-	</para>
-	<Para>
-	  You do not need to type "gmake clean" if this is the first time you
-	  are running the tests.
-	</Para>
-      </step>
-
-      
       <Step Performance="required">
 	<Para>
-	  Build the regression test. Type
+	  Run the regression tests.  For a sequential test, type
 	  <ProgramListing>
 	    cd /usr/src/pgsql/src/test/regress
-	    gmake all
+	    gmake runtest
 	  </ProgramListing>
-	</Para>
-      </Step>
-      
-      <Step Performance="required">
-	<Para>
-	  Run the regression tests.  Type
+	  For a parallel test, type
 	  <ProgramListing>
 	    cd /usr/src/pgsql/src/test/regress
-	    gmake runtest
+	    gmake runcheck
 	  </ProgramListing>
+	  The sequential test just runs the test scripts using your
+	  already-running postmaster.
+	  The parallel test will perform a complete installation of
+	  <ProductName>Postgres</ProductName> into a temporary directory,
+	  start a private postmaster therein, and then run the test scripts.
+	  Finally it will kill the private postmaster (but the temporary
+	  directory isn't removed automatically).
 	</Para>
       </Step>
 
       <Step Performance="required">
 	<Para>
-	  
 	  You should get on the screen (and also written to file ./regress.out)
 	  a series of statements stating which tests passed and which tests
 	  failed.  Please note that it can be normal for some of the tests to
-	  "fail".  For the failed tests, use diff to compare the files in
-	  directories ./results and ./expected.  If float8 failed, type
-	  something like:
-	  <ProgramListing>
-	    cd /usr/src/pgsql/src/test/regress
-	    diff -w expected/float8.out results
-	  </ProgramListing>
+	  "fail" due to platform-specific variations.  See the next section
+	  for details on determining whether a "failure" is significant.
+	</Para>
+	<Para>
+	  Some of the tests, notably "numeric", can take a while, especially
+	  on slower platforms.  Have patience.
 	</Para>
       </Step>
 
       <Step Performance="required">
 	<Para>
 	  After running the tests and examining the results, type
 	  <ProgramListing>
-	    dropdb regression
 	    cd /usr/src/pgsql/src/test/regress
 	    gmake clean
 	  </ProgramListing>
 	  to recover the temporary disk space used by the tests.
+	  If you ran a sequential test, also type
+	  <ProgramListing>
+	    dropdb regression
+	  </ProgramListing>
 	</Para>
       </Step>
     </procedure>
@@ -257,10 +262,14 @@ The runtime path is /usr/local/pgsql (other paths are possible).
     <Title>Regression Analysis</Title>
 
      <Para>
-       The results are in files in the ./results directory. These results
-       can be compared with results in the ./expected directory using 'diff'.
-       (The test script does this for you, and leaves the differences
-       in ./regression.diffs.)
+       The actual outputs of the regression tests are in files in the
+       <filename>./results</filename> directory. The test script
+       uses <application>diff</application> to compare each output file
+       against the reference outputs stored in the
+       <filename>./expected</filename> directory.  Any differences are
+       saved for your inspection in
+       <filename>./regression.diffs</filename>.  (Or you can run
+       <application>diff</application> yourself, if you prefer.)
      </Para>
 
      <Para>
@@ -354,7 +363,7 @@ The runtime path is /usr/local/pgsql (other paths are possible).
 
 	When comparing the results from different platforms, differences occur
 	in the 2nd or 3rd place to the right of the decimal point. The SQL
-	statements where these problems occur are the folowing:
+	statements where these problems occur are the following:
 
 	<ProgramListing>
 	  QUERY: SELECT * from street;
@@ -368,18 +377,19 @@ The runtime path is /usr/local/pgsql (other paths are possible).
       <Title>Random differences</Title>
 
       <Para>
-	There is at least one test case in random.out which is intended to produce
-	random results. This causes random to fail the regression testing
-	once in a while.
+	There is at least one case in the "random" test script that is
+	intended to produce
+	random results. This causes random to fail the regression test
+	once in a while (perhaps once in every five to ten trials).
 	Typing
 	<ProgramListing>
 	  diff results/random.out expected/random.out
 	</ProgramListing>
-	
-	should produce only
-	one or a few lines of differences for this reason, but other floating
-	point differences on dissimilar architectures might cause many more
-	differences. See the release notes below.
+	should produce only one or a few lines of differences.  You need
+	not worry unless the random test always fails in repeated attempts.
+	(On the other hand, if the random test is <emphasis>never</emphasis>
+	reported to fail even in many trials of the regress tests, you
+	probably <emphasis>should</emphasis> worry.)
       </Para>
 
     </Sect2>
@@ -398,10 +408,10 @@ The runtime path is /usr/local/pgsql (other paths are possible).
 	system using the <FileName>postgres5-1.02a5.tar.gz</FileName> source tree. It was compared
 	with a file created on an I386 Solaris 2.4 system and the differences
 	were only in the floating point polygons in the 3rd digit to the right
-	of the decimal point. (see below)
+	of the decimal point.
 
 	The original <FileName>sample.regress.out</FileName> file was from the postgres-1.01 release
-	constructed by Jolly Chen and is included here for reference. It may
+	constructed by Jolly Chen. It may
 	have been created on a DEC ALPHA machine as the <FileName>Makefile.global</FileName>
 	in the postgres-1.01 release has PORTNAME=alpha.
       </Para>