Change unknown-type literals to type text in SELECT and RETURNING lists.

Previously, we left such literals alone if the query or subquery had
no properties forcing a type decision to be made (such as an ORDER BY or
DISTINCT clause using that output column).  This meant that "unknown" could
be an exposed output column type, which has never been a great idea because
it could result in strange failures later on.  For example, an outer query
that tried to do any operations on an unknown-type subquery output would
generally fail with some weird error like "failed to find conversion
function from unknown to text" or "could not determine which collation to
use for string comparison".  Also, if the case occurred in a CREATE VIEW's
query then the view would have an unknown-type column, causing similar
failures in queries trying to use the view.

To fix, at the tail end of parse analysis of a query, forcibly convert any
remaining "unknown" literals in its SELECT or RETURNING list to type text.
However, provide a switch to suppress that, and use it in the cases of
SELECT inside a set operation or INSERT command.  In those cases we already
had type resolution rules that make use of context information from outside
the subquery proper, and we don't want to change that behavior.

Also, change creation of an unknown-type column in a relation from a
warning to a hard error.  The error should be unreachable now in CREATE
VIEW or CREATE MATVIEW, but it's still possible to explicitly say "unknown"
in CREATE TABLE or CREATE (composite) TYPE.  We want to forbid that because
it's nothing but a foot-gun.

This change creates a pg_upgrade failure case: a matview that contains an
unknown-type column can't be pg_upgraded, because reparsing the matview's
defining query will now decide that the column is of type text, which
doesn't match the cstring-like storage that the old materialized column
would actually have.  Add a checking pass to detect that.  While at it,
we can detect tables or composite types that would fail, essentially
for free.  Those would fail safely anyway later on, but we might as
well fail earlier.

This patch is by me, but it owes something to previous investigations
by Rahila Syed.  Also thanks to Ashutosh Bapat and Michael Paquier for
review.

Discussion: https://postgr.es/m/CAH2L28uwwbL9HUM-WR=hromW1Cvamkn7O-g8fPY2m=_7muJ0oA@mail.gmail.com

Author: tglsfdc

doc/src/sgml/ref/create_view.sgml

@@ -251,9 +251,10 @@ CREATE VIEW [ <replaceable>schema</> . ] <replaceable>view_name</> AS WITH RECUR
 <programlisting>
 CREATE VIEW vista AS SELECT 'Hello World';
 </programlisting>
-    is bad form in two ways: the column name defaults to <literal>?column?</>,
-    and the column data type defaults to <type>unknown</>.  If you want a
-    string literal in a view's result, use something like:
+    is bad form because the column name defaults to <literal>?column?</>;
+    also, the column data type defaults to <type>text</>, which might not
+    be what you wanted.  Better style for a string literal in a view's
+    result is something like:
 <programlisting>
 CREATE VIEW vista AS SELECT text 'Hello World' AS hello;
 </programlisting>

doc/src/sgml/typeconv.sgml

@@ -984,7 +984,8 @@ domain's base type for all subsequent steps.
 <para>
 If all inputs are of type <type>unknown</type>, resolve as type
 <type>text</type> (the preferred type of the string category).
-Otherwise, <type>unknown</type> inputs are ignored.
+Otherwise, <type>unknown</type> inputs are ignored for the purposes
+of the remaining rules.
 </para>
 </step>
 
@@ -1076,6 +1077,53 @@ but <type>integer</> can be implicitly cast to <type>real</>, the union
 result type is resolved as <type>real</>.
 </para>
 </example>
+</sect1>
+
+<sect1 id="typeconv-select">
+<title><literal>SELECT</literal> Output Columns</title>
+
+<indexterm zone="typeconv-select">
+ <primary>SELECT</primary>
+ <secondary>determination of result type</secondary>
+</indexterm>
+
+<para>
+The rules given in the preceding sections will result in assignment
+of non-<type>unknown</> data types to all expressions in a SQL query,
+except for unspecified-type literals that appear as simple output
+columns of a <command>SELECT</> command.  For example, in
+
+<screen>
+SELECT 'Hello World';
+</screen>
+
+there is nothing to identify what type the string literal should be
+taken as.  In this situation <productname>PostgreSQL</> will fall back
+to resolving the literal's type as <type>text</>.
+</para>
+
+<para>
+When the <command>SELECT</> is one arm of a <literal>UNION</>
+(or <literal>INTERSECT</> or <literal>EXCEPT</>) construct, or when it
+appears within <command>INSERT ... SELECT</>, this rule is not applied
+since rules given in preceding sections take precedence.  The type of an
+unspecified-type literal can be taken from the other <literal>UNION</> arm
+in the first case, or from the destination column in the second case.
+</para>
+
+<para>
+<literal>RETURNING</> lists are treated the same as <command>SELECT</>
+output lists for this purpose.
+</para>
+
+<note>
+ <para>
+  Prior to <productname>PostgreSQL</> 10, this rule did not exist, and
+  unspecified-type literals in a <command>SELECT</> output list were
+  left as type <type>unknown</>.  That had assorted bad consequences,
+  so it's been changed.
+ </para>
+</note>
 
 </sect1>
 </chapter>

src/backend/catalog/heap.c

@@ -490,18 +490,8 @@ CheckAttributeType(const char *attname,
 	char		att_typtype = get_typtype(atttypid);
 	Oid			att_typelem;
 
-	if (atttypid == UNKNOWNOID)
-	{
-		/*
-		 * Warn user, but don't fail, if column to be created has UNKNOWN type
-		 * (usually as a result of a 'retrieve into' - jolly)
-		 */
-		ereport(WARNING,
-				(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
-				 errmsg("column \"%s\" has type %s", attname, "unknown"),
-				 errdetail("Proceeding with relation creation anyway.")));
-	}
-	else if (att_typtype == TYPTYPE_PSEUDO)
+	if (atttypid == UNKNOWNOID ||
+		att_typtype == TYPTYPE_PSEUDO)
 	{
 		/*
 		 * Refuse any attempt to create a pseudo-type column, except for a

src/backend/parser/analyze.c

@@ -156,13 +156,15 @@ parse_analyze_varparams(RawStmt *parseTree, const char *sourceText,
 Query *
 parse_sub_analyze(Node *parseTree, ParseState *parentParseState,
 				  CommonTableExpr *parentCTE,
-				  bool locked_from_parent)
+				  bool locked_from_parent,
+				  bool resolve_unknowns)
 {
 	ParseState *pstate = make_parsestate(parentParseState);
 	Query	   *query;
 
 	pstate->p_parent_cte = parentCTE;
 	pstate->p_locked_from_parent = locked_from_parent;
+	pstate->p_resolve_unknowns = resolve_unknowns;
 
 	query = transformStmt(pstate, parseTree);
 
@@ -570,10 +572,17 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
 		 * otherwise the behavior of SELECT within INSERT might be different
 		 * from a stand-alone SELECT. (Indeed, Postgres up through 6.5 had
 		 * bugs of just that nature...)
+		 *
+		 * The sole exception is that we prevent resolving unknown-type
+		 * outputs as TEXT.  This does not change the semantics since if the
+		 * column type matters semantically, it would have been resolved to
+		 * something else anyway.  Doing this lets us resolve such outputs as
+		 * the target column's type, which we handle below.
 		 */
 		sub_pstate->p_rtable = sub_rtable;
 		sub_pstate->p_joinexprs = NIL;	/* sub_rtable has no joins */
 		sub_pstate->p_namespace = sub_namespace;
+		sub_pstate->p_resolve_unknowns = false;
 
 		selectQuery = transformStmt(sub_pstate, stmt->selectStmt);
 
@@ -1269,6 +1278,10 @@ transformSelectStmt(ParseState *pstate, SelectStmt *stmt)
 												   pstate->p_windowdefs,
 												   &qry->targetList);
 
+	/* resolve any still-unresolved output columns as being type text */
+	if (pstate->p_resolve_unknowns)
+		resolveTargetListUnknowns(pstate, qry->targetList);
+
 	qry->rtable = pstate->p_rtable;
 	qry->jointree = makeFromExpr(pstate->p_joinlist, qual);
 
@@ -1843,11 +1856,19 @@ transformSetOperationTree(ParseState *pstate, SelectStmt *stmt,
 		/*
 		 * Transform SelectStmt into a Query.
 		 *
+		 * This works the same as SELECT transformation normally would, except
+		 * that we prevent resolving unknown-type outputs as TEXT.  This does
+		 * not change the subquery's semantics since if the column type
+		 * matters semantically, it would have been resolved to something else
+		 * anyway.  Doing this lets us resolve such outputs using
+		 * select_common_type(), below.
+		 *
 		 * Note: previously transformed sub-queries don't affect the parsing
 		 * of this sub-query, because they are not in the toplevel pstate's
 		 * namespace list.
 		 */
-		selectQuery = parse_sub_analyze((Node *) stmt, pstate, NULL, false);
+		selectQuery = parse_sub_analyze((Node *) stmt, pstate,
+										NULL, false, false);
 
 		/*
 		 * Check for bogus references to Vars on the current query level (but
@@ -2350,6 +2371,10 @@ transformReturningList(ParseState *pstate, List *returningList)
 	/* mark column origins */
 	markTargetListOrigins(pstate, rlist);
 
+	/* resolve any still-unresolved output columns as being type text */
+	if (pstate->p_resolve_unknowns)
+		resolveTargetListUnknowns(pstate, rlist);
+
 	/* restore state */
 	pstate->p_next_resno = save_next_resno;
 

src/backend/parser/parse_clause.c

@@ -471,7 +471,8 @@ transformRangeSubselect(ParseState *pstate, RangeSubselect *r)
 	 * Analyze and transform the subquery.
 	 */
 	query = parse_sub_analyze(r->subquery, pstate, NULL,
-							  isLockedRefname(pstate, r->alias->aliasname));
+							  isLockedRefname(pstate, r->alias->aliasname),
+							  true);
 
 	/* Restore state */
 	pstate->p_lateral_active = false;

src/backend/parser/parse_cte.c

@@ -241,7 +241,7 @@ analyzeCTE(ParseState *pstate, CommonTableExpr *cte)
 	/* Analysis not done already */
 	Assert(!IsA(cte->ctequery, Query));
 
-	query = parse_sub_analyze(cte->ctequery, pstate, cte, false);
+	query = parse_sub_analyze(cte->ctequery, pstate, cte, false, true);
 	cte->ctequery = (Node *) query;
 
 	/*
@@ -393,11 +393,10 @@ analyzeCTETargetList(ParseState *pstate, CommonTableExpr *cte, List *tlist)
 
 		/*
 		 * If the CTE is recursive, force the exposed column type of any
-		 * "unknown" column to "text".  This corresponds to the fact that
-		 * SELECT 'foo' UNION SELECT 'bar' will ultimately produce text. We
-		 * might see "unknown" as a result of an untyped literal in the
-		 * non-recursive term's select list, and if we don't convert to text
-		 * then we'll have a mismatch against the UNION result.
+		 * "unknown" column to "text".  We must deal with this here because
+		 * we're called on the non-recursive term before there's been any
+		 * attempt to force unknown output columns to some other type.  We
+		 * have to resolve unknowns before looking at the recursive term.
 		 *
 		 * The column might contain 'foo' COLLATE "bar", so don't override
 		 * collation if it's already set.

src/backend/parser/parse_expr.c

@@ -1846,7 +1846,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink)
 	/*
 	 * OK, let's transform the sub-SELECT.
 	 */
-	qtree = parse_sub_analyze(sublink->subselect, pstate, NULL, false);
+	qtree = parse_sub_analyze(sublink->subselect, pstate, NULL, false, true);
 
 	/*
 	 * Check that we got a SELECT.  Anything else should be impossible given

src/backend/parser/parse_node.c

@@ -51,6 +51,7 @@ make_parsestate(ParseState *parentParseState)
 
 	/* Fill in fields that don't start at null/false/zero */
 	pstate->p_next_resno = 1;
+	pstate->p_resolve_unknowns = true;
 
 	if (parentParseState)
 	{

src/backend/parser/parse_target.c

@@ -288,13 +288,42 @@ transformExpressionList(ParseState *pstate, List *exprlist,
 }
 
 
+/*
+ * resolveTargetListUnknowns()
+ *		Convert any unknown-type targetlist entries to type TEXT.
+ *
+ * We do this after we've exhausted all other ways of identifying the output
+ * column types of a query.
+ */
+void
+resolveTargetListUnknowns(ParseState *pstate, List *targetlist)
+{
+	ListCell   *l;
+
+	foreach(l, targetlist)
+	{
+		TargetEntry *tle = (TargetEntry *) lfirst(l);
+		Oid			restype = exprType((Node *) tle->expr);
+
+		if (restype == UNKNOWNOID)
+		{
+			tle->expr = (Expr *) coerce_type(pstate, (Node *) tle->expr,
+											 restype, TEXTOID, -1,
+											 COERCION_IMPLICIT,
+											 COERCE_IMPLICIT_CAST,
+											 -1);
+		}
+	}
+}
+
+
 /*
  * markTargetListOrigins()
  *		Mark targetlist columns that are simple Vars with the source
  *		table's OID and column number.
  *
- * Currently, this is done only for SELECT targetlists, since we only
- * need the info if we are going to send it to the frontend.
+ * Currently, this is done only for SELECT targetlists and RETURNING lists,
+ * since we only need the info if we are going to send it to the frontend.
  */
 void
 markTargetListOrigins(ParseState *pstate, List *targetlist)

src/bin/pg_upgrade/check.c

@@ -99,6 +99,10 @@ check_and_dump_old_cluster(bool live_check)
 	check_for_reg_data_type_usage(&old_cluster);
 	check_for_isn_and_int8_passing_mismatch(&old_cluster);
 
+	/* Pre-PG 10 allowed tables with 'unknown' type columns */
+	if (GET_MAJOR_VERSION(old_cluster.major_version) <= 906)
+		old_9_6_check_for_unknown_data_type_usage(&old_cluster);
+
 	/* 9.5 and below should not have roles starting with pg_ */
 	if (GET_MAJOR_VERSION(old_cluster.major_version) <= 905)
 		check_for_pg_role_prefix(&old_cluster);

src/bin/pg_upgrade/pg_upgrade.h

@@ -442,6 +442,7 @@ void		pg_putenv(const char *var, const char *val);
 void new_9_0_populate_pg_largeobject_metadata(ClusterInfo *cluster,
 										 bool check_mode);
 void		old_9_3_check_for_line_data_type_usage(ClusterInfo *cluster);
+void		old_9_6_check_for_unknown_data_type_usage(ClusterInfo *cluster);
 
 /* parallel.c */
 void parallel_exec_prog(const char *log_file, const char *opt_log_file,

src/bin/pg_upgrade/version.c

@@ -185,3 +185,100 @@ old_9_3_check_for_line_data_type_usage(ClusterInfo *cluster)
 	else
 		check_ok();
 }
+
+
+/*
+ * old_9_6_check_for_unknown_data_type_usage()
+ *	9.6 -> 10
+ *	It's no longer allowed to create tables or views with "unknown"-type
+ *	columns.  We do not complain about views with such columns, because
+ *	they should get silently converted to "text" columns during the DDL
+ *	dump and reload; it seems unlikely to be worth making users do that
+ *	by hand.  However, if there's a table with such a column, the DDL
+ *	reload will fail, so we should pre-detect that rather than failing
+ *	mid-upgrade.  Worse, if there's a matview with such a column, the
+ *	DDL reload will silently change it to "text" which won't match the
+ *	on-disk storage (which is like "cstring").  So we *must* reject that.
+ *	Also check composite types, in case they are used for table columns.
+ *	We needn't check indexes, because "unknown" has no opclasses.
+ */
+void
+old_9_6_check_for_unknown_data_type_usage(ClusterInfo *cluster)
+{
+	int			dbnum;
+	FILE	   *script = NULL;
+	bool		found = false;
+	char		output_path[MAXPGPATH];
+
+	prep_status("Checking for invalid \"unknown\" user columns");
+
+	snprintf(output_path, sizeof(output_path), "tables_using_unknown.txt");
+
+	for (dbnum = 0; dbnum < cluster->dbarr.ndbs; dbnum++)
+	{
+		PGresult   *res;
+		bool		db_used = false;
+		int			ntups;
+		int			rowno;
+		int			i_nspname,
+					i_relname,
+					i_attname;
+		DbInfo	   *active_db = &cluster->dbarr.dbs[dbnum];
+		PGconn	   *conn = connectToServer(cluster, active_db->db_name);
+
+		res = executeQueryOrDie(conn,
+								"SELECT n.nspname, c.relname, a.attname "
+								"FROM	pg_catalog.pg_class c, "
+								"		pg_catalog.pg_namespace n, "
+								"		pg_catalog.pg_attribute a "
+								"WHERE	c.oid = a.attrelid AND "
+								"		NOT a.attisdropped AND "
+								"		a.atttypid = 'pg_catalog.unknown'::pg_catalog.regtype AND "
+						   "		c.relkind IN ('r', 'c', 'm') AND "
+								"		c.relnamespace = n.oid AND "
+		/* exclude possible orphaned temp tables */
+								"		n.nspname !~ '^pg_temp_' AND "
+						 "		n.nspname !~ '^pg_toast_temp_' AND "
+								"		n.nspname NOT IN ('pg_catalog', 'information_schema')");
+
+		ntups = PQntuples(res);
+		i_nspname = PQfnumber(res, "nspname");
+		i_relname = PQfnumber(res, "relname");
+		i_attname = PQfnumber(res, "attname");
+		for (rowno = 0; rowno < ntups; rowno++)
+		{
+			found = true;
+			if (script == NULL && (script = fopen_priv(output_path, "w")) == NULL)
+				pg_fatal("could not open file \"%s\": %s\n", output_path,
+						 strerror(errno));
+			if (!db_used)
+			{
+				fprintf(script, "Database: %s\n", active_db->db_name);
+				db_used = true;
+			}
+			fprintf(script, "  %s.%s.%s\n",
+					PQgetvalue(res, rowno, i_nspname),
+					PQgetvalue(res, rowno, i_relname),
+					PQgetvalue(res, rowno, i_attname));
+		}
+
+		PQclear(res);
+
+		PQfinish(conn);
+	}
+
+	if (script)
+		fclose(script);
+
+	if (found)
+	{
+		pg_log(PG_REPORT, "fatal\n");
+		pg_fatal("Your installation contains the \"unknown\" data type in user tables.  This\n"
+				 "data type is no longer allowed in tables, so this cluster cannot currently\n"
+				 "be upgraded.  You can remove the problem tables and restart the upgrade.\n"
+				 "A list of the problem columns is in the file:\n"
+				 "    %s\n\n", output_path);
+	}
+	else
+		check_ok();
+}