Allow most keywords to be used as column labels without requiring AS.

Up to now, if you tried to omit "AS" before a column label in a SELECT
list, it would only work if the column label was an IDENT, that is not
any known keyword.  This is rather unfriendly considering that we have
so many keywords and are constantly growing more.  In the wake of commit
1ed6b89 it's possible to improve matters quite a bit.

We'd originally tried to make this work by having some of the existing
keyword categories be allowed without AS, but that didn't work too well,
because each category contains a few special cases that don't work
without AS.  Instead, invent an entirely orthogonal keyword property
"can be bare column label", and mark all keywords that way for which
we don't get shift/reduce errors by doing so.

It turns out that of our 450 current keywords, all but 39 can be made
bare column labels, improving the situation by over 90%.  This number
might move around a little depending on future grammar work, but it's
a pretty nice improvement.

Mark Dilger, based on work by myself and Robert Haas;
review by John Naylor

Discussion: https://postgr.es/m/38ca86db-42ab-9b48-2902-337a0d6b8311@2ndquadrant.com

Author: tglsfdc

doc/src/sgml/func.sgml

@@ -22173,7 +22173,9 @@ SELECT pg_type_is_visible('myschema.widget'::regtype);
         <returnvalue>setof record</returnvalue>
         ( <parameter>word</parameter> <type>text</type>,
         <parameter>catcode</parameter> <type>"char"</type>,
-        <parameter>catdesc</parameter> <type>text</type> )
+        <parameter>barelabel</parameter> <type>boolean</type>,
+        <parameter>catdesc</parameter> <type>text</type>,
+        <parameter>baredesc</parameter> <type>text</type> )
        </para>
        <para>
         Returns a set of records describing the SQL keywords recognized by the
@@ -22183,8 +22185,15 @@ SELECT pg_type_is_visible('myschema.widget'::regtype);
         keyword, <literal>C</literal> for a keyword that can be a column
         name, <literal>T</literal> for a keyword that can be a type or
         function name, or <literal>R</literal> for a fully reserved keyword.
+        The <parameter>barelabel</parameter> column
+        contains <literal>true</literal> if the keyword can be used as
+        a <quote>bare</quote> column label in <command>SELECT</command> lists,
+        or <literal>false</literal> if it can only be used
+        after <literal>AS</literal>.
         The <parameter>catdesc</parameter> column contains a
-        possibly-localized string describing the category.
+        possibly-localized string describing the keyword's category.
+        The <parameter>baredesc</parameter> column contains a
+        possibly-localized string describing the keyword's column label status.
        </para></entry>
       </row>
 

doc/src/sgml/generate-keywords-table.pl

@@ -1,6 +1,7 @@
 #!/usr/bin/perl
 #
-# Generate the keywords table file
+# Generate the keywords table for the documentation's SQL Key Words appendix
+#
 # Copyright (c) 2019-2020, PostgreSQL Global Development Group
 
 use strict;
@@ -11,8 +12,9 @@
 my $srcdir = $ARGV[0];
 
 my %keywords;
+my %as_keywords;
 
-# read SQL keywords
+# read SQL-spec keywords
 
 foreach my $ver (@sql_versions)
 {
@@ -39,9 +41,10 @@
 
 while (<$fh>)
 {
-	if (/^PG_KEYWORD\("(\w+)", \w+, (\w+)_KEYWORD\)/)
+	if (/^PG_KEYWORD\("(\w+)", \w+, (\w+)_KEYWORD\, (\w+)\)/)
 	{
 		$keywords{ uc $1 }{'pg'}{ lc $2 } = 1;
+		$as_keywords{ uc $1 } = 1 if $3 eq 'AS_LABEL';
 	}
 }
 
@@ -107,6 +110,10 @@ END
 	{
 		print "reserved";
 	}
+	if ($as_keywords{$word})
+	{
+		print ", requires <literal>AS</literal>";
+	}
 	print "</entry>\n";
 
 	foreach my $ver (@sql_versions)

doc/src/sgml/keywords.sgml

@@ -32,11 +32,11 @@
  </para>
 
  <para>
-  In the <productname>PostgreSQL</productname> parser life is a bit
+  In the <productname>PostgreSQL</productname> parser, life is a bit
   more complicated. There are several different classes of tokens
   ranging from those that can never be used as an identifier to those
-  that have absolutely no special status in the parser as compared to
-  an ordinary identifier.  (The latter is usually the case for
+  that have absolutely no special status in the parser, but are considered
+  ordinary identifiers.  (The latter is usually the case for
   functions specified by SQL.)  Even reserved key words are not
   completely reserved in <productname>PostgreSQL</productname>, but
   can be used as column labels (for example, <literal>SELECT 55 AS
@@ -57,14 +57,24 @@
   <quote>reserved</quote> are those tokens that are not allowed as
   column or table names.  Some reserved key words are
   allowable as names for functions or data types; this is also shown in the
-  table.  If not so marked, a reserved key word is only allowed as an
-  <quote>AS</quote> column label name.
+  table.  If not so marked, a reserved key word is only allowed as a
+  column label.
+  A blank entry in this column means that the word is treated as an
+  ordinary identifier by <productname>PostgreSQL</productname>.
+ </para>
+
+ <para>
+  Furthermore, while most key words can be used as <quote>bare</quote>
+  column labels without writing <literal>AS</literal> before them (as
+  described in <xref linkend="queries-column-labels"/>), there are a few
+  that require a leading <literal>AS</literal> to avoid ambiguity.  These
+  are marked in the table as <quote>requires <literal>AS</literal></quote>.
  </para>
 
  <para>
   As a general rule, if you get spurious parser errors for commands
-  that contain any of the listed key words as an identifier you should
-  try to quote the identifier to see if the problem goes away.
+  that use any of the listed key words as an identifier, you should
+  try quoting the identifier to see if the problem goes away.
  </para>
 
  <para>

doc/src/sgml/queries.sgml

@@ -1496,21 +1496,25 @@ SELECT a AS value, b + c AS sum FROM ...
    </para>
 
    <para>
-    The <literal>AS</literal> keyword is optional, but only if the new column
-    name does not match any
-    <productname>PostgreSQL</productname> keyword (see <xref
-    linkend="sql-keywords-appendix"/>).  To avoid an accidental match to
-    a keyword, you can double-quote the column name.  For example,
-    <literal>VALUE</literal> is a keyword, so this does not work:
+    The <literal>AS</literal> key word is usually optional, but in some
+    cases where the desired column name matches a
+    <productname>PostgreSQL</productname> key word, you must write
+    <literal>AS</literal> or double-quote the column name in order to
+    avoid ambiguity.
+    (<xref linkend="sql-keywords-appendix"/> shows which key words
+    require <literal>AS</literal> to be used as a column label.)
+    For example, <literal>FROM</literal> is one such key word, so this
+    does not work:
 <programlisting>
-SELECT a value, b + c AS sum FROM ...
+SELECT a from, b + c AS sum FROM ...
 </programlisting>
-    but this does:
+    but either of these do:
 <programlisting>
-SELECT a "value", b + c AS sum FROM ...
+SELECT a AS from, b + c AS sum FROM ...
+SELECT a "from", b + c AS sum FROM ...
 </programlisting>
-    For protection against possible
-    future keyword additions, it is recommended that you always either
+    For greatest safety against possible
+    future key word additions, it is recommended that you always either
     write <literal>AS</literal> or double-quote the output column name.
    </para>
 

src/backend/parser/check_keywords.pl

@@ -21,6 +21,28 @@ sub error
 	return;
 }
 
+# Check alphabetical order of a set of keyword symbols
+# (note these are NOT the actual keyword strings)
+sub check_alphabetical_order
+{
+	my ($listname, $list) = @_;
+	my $prevkword = '';
+
+	foreach my $kword (@$list)
+	{
+		# Some symbols have a _P suffix. Remove it for the comparison.
+		my $bare_kword = $kword;
+		$bare_kword =~ s/_P$//;
+		if ($bare_kword le $prevkword)
+		{
+			error
+			  "'$bare_kword' after '$prevkword' in $listname list is misplaced";
+		}
+		$prevkword = $bare_kword;
+	}
+	return;
+}
+
 $, = ' ';     # set output field separator
 $\ = "\n";    # set output record separator
 
@@ -33,9 +55,11 @@ sub error
 open(my $gram, '<', $gram_filename) || die("Could not open : $gram_filename");
 
 my $kcat;
+my $in_bare_labels;
 my $comment;
 my @arr;
 my %keywords;
+my @bare_label_keywords;
 
 line: while (my $S = <$gram>)
 {
@@ -51,7 +75,7 @@ sub error
 	$s = '[/][*]', $S =~ s#$s# /* #g;
 	$s = '[*][/]', $S =~ s#$s# */ #g;
 
-	if (!($kcat))
+	if (!($kcat) && !($in_bare_labels))
 	{
 
 		# Is this the beginning of a keyword list?
@@ -63,6 +87,10 @@ sub error
 				next line;
 			}
 		}
+
+		# Is this the beginning of the bare_label_keyword list?
+		$in_bare_labels = 1 if ($S =~ m/^bare_label_keyword:/);
+
 		next line;
 	}
 
@@ -97,7 +125,8 @@ sub error
 		{
 
 			# end of keyword list
-			$kcat = '';
+			undef $kcat;
+			undef $in_bare_labels;
 			next;
 		}
 
@@ -107,31 +136,21 @@ sub error
 		}
 
 		# Put this keyword into the right list
-		push @{ $keywords{$kcat} }, $arr[$fieldIndexer];
+		if ($in_bare_labels)
+		{
+			push @bare_label_keywords, $arr[$fieldIndexer];
+		}
+		else
+		{
+			push @{ $keywords{$kcat} }, $arr[$fieldIndexer];
+		}
 	}
 }
 close $gram;
 
 # Check that each keyword list is in alphabetical order (just for neatnik-ism)
-my ($prevkword, $bare_kword);
-foreach my $kcat (keys %keyword_categories)
-{
-	$prevkword = '';
-
-	foreach my $kword (@{ $keywords{$kcat} })
-	{
-
-		# Some keyword have a _P suffix. Remove it for the comparison.
-		$bare_kword = $kword;
-		$bare_kword =~ s/_P$//;
-		if ($bare_kword le $prevkword)
-		{
-			error
-			  "'$bare_kword' after '$prevkword' in $kcat list is misplaced";
-		}
-		$prevkword = $bare_kword;
-	}
-}
+check_alphabetical_order($_, $keywords{$_}) for (keys %keyword_categories);
+check_alphabetical_order('bare_label_keyword', \@bare_label_keywords);
 
 # Transform the keyword lists into hashes.
 # kwhashes is a hash of hashes, keyed by keyword category id,
@@ -147,6 +166,7 @@ sub error
 
 	$kwhashes{$kcat_id} = $hash;
 }
+my %bare_label_keywords = map { $_ => 1 } @bare_label_keywords;
 
 # Now read in kwlist.h
 
@@ -160,11 +180,12 @@ sub error
 {
 	my ($line) = $_;
 
-	if ($line =~ /^PG_KEYWORD\(\"(.*)\", (.*), (.*)\)/)
+	if ($line =~ /^PG_KEYWORD\(\"(.*)\", (.*), (.*), (.*)\)/)
 	{
 		my ($kwstring) = $1;
 		my ($kwname)   = $2;
 		my ($kwcat_id) = $3;
+		my ($collabel) = $4;
 
 		# Check that the list is in alphabetical order (critical!)
 		if ($kwstring le $prevkwstring)
@@ -197,7 +218,7 @@ sub error
 			  "keyword name '$kwname' doesn't match keyword string '$kwstring'";
 		}
 
-		# Check that the keyword is present in the grammar
+		# Check that the keyword is present in the right category list
 		%kwhash = %{ $kwhashes{$kwcat_id} };
 
 		if (!(%kwhash))
@@ -219,6 +240,29 @@ sub error
 				delete $kwhashes{$kwcat_id}->{$kwname};
 			}
 		}
+
+		# Check that the keyword's collabel property matches gram.y
+		if ($collabel eq 'BARE_LABEL')
+		{
+			unless ($bare_label_keywords{$kwname})
+			{
+				error
+				  "'$kwname' is marked as BARE_LABEL in kwlist.h, but it is missing from gram.y's bare_label_keyword rule";
+			}
+		}
+		elsif ($collabel eq 'AS_LABEL')
+		{
+			if ($bare_label_keywords{$kwname})
+			{
+				error
+				  "'$kwname' is marked as AS_LABEL in kwlist.h, but it is listed in gram.y's bare_label_keyword rule";
+			}
+		}
+		else
+		{
+			error
+			  "'$collabel' not recognized in kwlist.h.  Expected either 'BARE_LABEL' or 'AS_LABEL'";
+		}
 	}
 }
 close $kwlist;