Virtual generated columns

This adds a new variant of generated columns that are computed on read
(like a view, unlike the existing stored generated columns, which are
computed on write, like a materialized view).

The syntax for the column definition is

    ... GENERATED ALWAYS AS (...) VIRTUAL

and VIRTUAL is also optional.  VIRTUAL is the default rather than
STORED to match various other SQL products.  (The SQL standard makes
no specification about this, but it also doesn't know about VIRTUAL or
STORED.)  (Also, virtual views are the default, rather than
materialized views.)

Virtual generated columns are stored in tuples as null values.  (A
very early version of this patch had the ambition to not store them at
all.  But so much stuff breaks or gets confused if you have tuples
where a column in the middle is completely missing.  This is a
compromise, and it still saves space over being forced to use stored
generated columns.  If we ever find a way to improve this, a bit of
pg_upgrade cleverness could allow for upgrades to a newer scheme.)

The capabilities and restrictions of virtual generated columns are
mostly the same as for stored generated columns.  In some cases, this
patch keeps virtual generated columns more restricted than they might
technically need to be, to keep the two kinds consistent.  Some of
that could maybe be relaxed later after separate careful
considerations.

Some functionality that is currently not supported, but could possibly
be added as incremental features, some easier than others:

- index on or using a virtual column
- hence also no unique constraints on virtual columns
- extended statistics on virtual columns
- foreign-key constraints on virtual columns
- not-null constraints on virtual columns (check constraints are supported)
- ALTER TABLE / DROP EXPRESSION
- virtual column cannot have domain type
- virtual columns are not supported in logical replication

The tests in generated_virtual.sql have been copied over from
generated_stored.sql with the keyword replaced.  This way we can make
sure the behavior is mostly aligned, and the differences can be
visible.  Some tests for currently not supported features are
currently commented out.

Reviewed-by: Jian He <jian.universality@gmail.com>
Reviewed-by: Dean Rasheed <dean.a.rasheed@gmail.com>
Tested-by: Shlok Kyal <shlok.kyal.oss@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/a368248e-69e4-40be-9c07-6c3b5880b0a6@eisentraut.org

Author: petere

contrib/pageinspect/expected/page.out

@@ -208,6 +208,43 @@ select tuple_data_split('test8'::regclass, t_data, t_infomask, t_infomask2, t_bi
 (1 row)
 
 drop table test8;
+-- check storage of generated columns
+-- stored
+create table test9s (a int not null, b int generated always as (a * 2) stored);
+insert into test9s values (131584);
+select raw_flags, t_bits, t_data
+    from heap_page_items(get_raw_page('test9s', 0)), lateral heap_tuple_infomask_flags(t_infomask, t_infomask2);
+      raw_flags      | t_bits |       t_data       
+---------------------+--------+--------------------
+ {HEAP_XMAX_INVALID} |        | \x0002020000040400
+(1 row)
+
+select tuple_data_split('test9s'::regclass, t_data, t_infomask, t_infomask2, t_bits)
+    from heap_page_items(get_raw_page('test9s', 0));
+       tuple_data_split        
+-------------------------------
+ {"\\x00020200","\\x00040400"}
+(1 row)
+
+drop table test9s;
+-- virtual
+create table test9v (a int not null, b int generated always as (a * 2) virtual);
+insert into test9v values (131584);
+select raw_flags, t_bits, t_data
+    from heap_page_items(get_raw_page('test9v', 0)), lateral heap_tuple_infomask_flags(t_infomask, t_infomask2);
+            raw_flags             |  t_bits  |   t_data   
+----------------------------------+----------+------------
+ {HEAP_HASNULL,HEAP_XMAX_INVALID} | 10000000 | \x00020200
+(1 row)
+
+select tuple_data_split('test9v'::regclass, t_data, t_infomask, t_infomask2, t_bits)
+    from heap_page_items(get_raw_page('test9v', 0));
+   tuple_data_split   
+----------------------
+ {"\\x00020200",NULL}
+(1 row)
+
+drop table test9v;
 -- Failure with incorrect page size
 -- Suppress the DETAIL message, to allow the tests to work across various
 -- page sizes.

contrib/pageinspect/sql/page.sql

@@ -84,6 +84,25 @@ select tuple_data_split('test8'::regclass, t_data, t_infomask, t_infomask2, t_bi
     from heap_page_items(get_raw_page('test8', 0));
 drop table test8;
 
+-- check storage of generated columns
+-- stored
+create table test9s (a int not null, b int generated always as (a * 2) stored);
+insert into test9s values (131584);
+select raw_flags, t_bits, t_data
+    from heap_page_items(get_raw_page('test9s', 0)), lateral heap_tuple_infomask_flags(t_infomask, t_infomask2);
+select tuple_data_split('test9s'::regclass, t_data, t_infomask, t_infomask2, t_bits)
+    from heap_page_items(get_raw_page('test9s', 0));
+drop table test9s;
+
+-- virtual
+create table test9v (a int not null, b int generated always as (a * 2) virtual);
+insert into test9v values (131584);
+select raw_flags, t_bits, t_data
+    from heap_page_items(get_raw_page('test9v', 0)), lateral heap_tuple_infomask_flags(t_infomask, t_infomask2);
+select tuple_data_split('test9v'::regclass, t_data, t_infomask, t_infomask2, t_bits)
+    from heap_page_items(get_raw_page('test9v', 0));
+drop table test9v;
+
 -- Failure with incorrect page size
 -- Suppress the DETAIL message, to allow the tests to work across various
 -- page sizes.

contrib/postgres_fdw/expected/postgres_fdw.out

@@ -7395,94 +7395,97 @@ select * from rem1;
 -- ===================================================================
 create table gloc1 (
   a int,
-  b int generated always as (a * 2) stored);
+  b int generated always as (a * 2) stored,
+  c int
+);
 alter table gloc1 set (autovacuum_enabled = 'false');
 create foreign table grem1 (
   a int,
-  b int generated always as (a * 2) stored)
-  server loopback options(table_name 'gloc1');
+  b int generated always as (a * 2) stored,
+  c int generated always as (a * 3) virtual
+) server loopback options(table_name 'gloc1');
 explain (verbose, costs off)
 insert into grem1 (a) values (1), (2);
-                            QUERY PLAN                             
--------------------------------------------------------------------
+                                  QUERY PLAN                                   
+-------------------------------------------------------------------------------
  Insert on public.grem1
-   Remote SQL: INSERT INTO public.gloc1(a, b) VALUES ($1, DEFAULT)
+   Remote SQL: INSERT INTO public.gloc1(a, b, c) VALUES ($1, DEFAULT, DEFAULT)
    Batch Size: 1
    ->  Values Scan on "*VALUES*"
-         Output: "*VALUES*".column1, NULL::integer
+         Output: "*VALUES*".column1, NULL::integer, NULL::integer
 (5 rows)
 
 insert into grem1 (a) values (1), (2);
 explain (verbose, costs off)
 update grem1 set a = 22 where a = 2;
-                                     QUERY PLAN                                     
-------------------------------------------------------------------------------------
+                                       QUERY PLAN                                       
+----------------------------------------------------------------------------------------
  Update on public.grem1
-   Remote SQL: UPDATE public.gloc1 SET a = $2, b = DEFAULT WHERE ctid = $1
+   Remote SQL: UPDATE public.gloc1 SET a = $2, b = DEFAULT, c = DEFAULT WHERE ctid = $1
    ->  Foreign Scan on public.grem1
          Output: 22, ctid, grem1.*
-         Remote SQL: SELECT a, b, ctid FROM public.gloc1 WHERE ((a = 2)) FOR UPDATE
+         Remote SQL: SELECT a, b, c, ctid FROM public.gloc1 WHERE ((a = 2)) FOR UPDATE
 (5 rows)
 
 update grem1 set a = 22 where a = 2;
 select * from gloc1;
- a  | b  
-----+----
-  1 |  2
- 22 | 44
+ a  | b  | c 
+----+----+---
+  1 |  2 |  
+ 22 | 44 |  
 (2 rows)
 
 select * from grem1;
- a  | b  
-----+----
-  1 |  2
- 22 | 44
+ a  | b  | c  
+----+----+----
+  1 |  2 |  3
+ 22 | 44 | 66
 (2 rows)
 
 delete from grem1;
 -- test copy from
 copy grem1 from stdin;
 select * from gloc1;
- a | b 
----+---
- 1 | 2
- 2 | 4
+ a | b | c 
+---+---+---
+ 1 | 2 |  
+ 2 | 4 |  
 (2 rows)
 
 select * from grem1;
- a | b 
----+---
- 1 | 2
- 2 | 4
+ a | b | c 
+---+---+---
+ 1 | 2 | 3
+ 2 | 4 | 6
 (2 rows)
 
 delete from grem1;
 -- test batch insert
 alter server loopback options (add batch_size '10');
 explain (verbose, costs off)
 insert into grem1 (a) values (1), (2);
-                            QUERY PLAN                             
--------------------------------------------------------------------
+                                  QUERY PLAN                                   
+-------------------------------------------------------------------------------
  Insert on public.grem1
-   Remote SQL: INSERT INTO public.gloc1(a, b) VALUES ($1, DEFAULT)
+   Remote SQL: INSERT INTO public.gloc1(a, b, c) VALUES ($1, DEFAULT, DEFAULT)
    Batch Size: 10
    ->  Values Scan on "*VALUES*"
-         Output: "*VALUES*".column1, NULL::integer
+         Output: "*VALUES*".column1, NULL::integer, NULL::integer
 (5 rows)
 
 insert into grem1 (a) values (1), (2);
 select * from gloc1;
- a | b 
----+---
- 1 | 2
- 2 | 4
+ a | b | c 
+---+---+---
+ 1 | 2 |  
+ 2 | 4 |  
 (2 rows)
 
 select * from grem1;
- a | b 
----+---
- 1 | 2
- 2 | 4
+ a | b | c 
+---+---+---
+ 1 | 2 | 3
+ 2 | 4 | 6
 (2 rows)
 
 delete from grem1;

contrib/postgres_fdw/sql/postgres_fdw.sql

@@ -1859,12 +1859,15 @@ select * from rem1;
 -- ===================================================================
 create table gloc1 (
   a int,
-  b int generated always as (a * 2) stored);
+  b int generated always as (a * 2) stored,
+  c int
+);
 alter table gloc1 set (autovacuum_enabled = 'false');
 create foreign table grem1 (
   a int,
-  b int generated always as (a * 2) stored)
-  server loopback options(table_name 'gloc1');
+  b int generated always as (a * 2) stored,
+  c int generated always as (a * 3) virtual
+) server loopback options(table_name 'gloc1');
 explain (verbose, costs off)
 insert into grem1 (a) values (1), (2);
 insert into grem1 (a) values (1), (2);

doc/src/sgml/catalogs.sgml

@@ -1307,8 +1307,10 @@
       </para>
       <para>
        If a zero byte (<literal>''</literal>), then not a generated column.
-       Otherwise, <literal>s</literal> = stored.  (Other values might be added
-       in the future.)
+       Otherwise, <literal>s</literal> = stored, <literal>v</literal> =
+       virtual.  A stored generated column is physically stored like a normal
+       column.  A virtual generated column is physically stored as a null
+       value, with the actual value being computed at run time.
       </para></entry>
      </row>
 

doc/src/sgml/ddl.sgml

@@ -361,7 +361,6 @@ INSERT INTO people (id, name, address) VALUES (<emphasis>DEFAULT</emphasis>, 'C'
    storage and is computed when it is read.  Thus, a virtual generated column
    is similar to a view and a stored generated column is similar to a
    materialized view (except that it is always updated automatically).
-   <productname>PostgreSQL</productname> currently implements only stored generated columns.
   </para>
 
   <para>
@@ -371,12 +370,12 @@ INSERT INTO people (id, name, address) VALUES (<emphasis>DEFAULT</emphasis>, 'C'
 CREATE TABLE people (
     ...,
     height_cm numeric,
-    height_in numeric <emphasis>GENERATED ALWAYS AS (height_cm / 2.54) STORED</emphasis>
+    height_in numeric <emphasis>GENERATED ALWAYS AS (height_cm / 2.54)</emphasis>
 );
 </programlisting>
-   The keyword <literal>STORED</literal> must be specified to choose the
-   stored kind of generated column.  See <xref linkend="sql-createtable"/> for
-   more details.
+   A generated column is by default of the virtual kind.  Use the keywords
+   <literal>VIRTUAL</literal> or <literal>STORED</literal> to make the choice
+   explicit.  See <xref linkend="sql-createtable"/> for more details.
   </para>
 
   <para>
@@ -442,12 +441,18 @@ CREATE TABLE people (
       <listitem>
        <para>
         If a parent column is a generated column, its child column must also
-        be a generated column; however, the child column can have a
-        different generation expression.  The generation expression that is
+        be a generated column of the same kind (stored or virtual); however,
+        the child column can have a different generation expression.
+       </para>
+
+       <para>
+        For stored generated columns, the generation expression that is
         actually applied during insert or update of a row is the one
-        associated with the table that the row is physically in.
-        (This is unlike the behavior for column defaults: for those, the
-        default value associated with the table named in the query applies.)
+        associated with the table that the row is physically in.  (This is
+        unlike the behavior for column defaults: for those, the default value
+        associated with the table named in the query applies.)  For virtual
+        generated columns, the generation expression of the table named in the
+        query applies when a table is read.
        </para>
       </listitem>
       <listitem>
@@ -502,6 +507,26 @@ CREATE TABLE people (
       particular role can read from a generated column but not from the
       underlying base columns.
      </para>
+
+     <para>
+      For virtual generated columns, this is only fully secure if the
+      generation expression uses only leakproof functions (see <xref
+      linkend="sql-createfunction"/>), but this is not enforced by the system.
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      Privileges of functions used in generation expressions are checked when
+      the expression is actually executed, on write or read respectively, as
+      if the generation expression had been called directly from the query
+      using the generated column.  The user of a generated column must have
+      permissions to call all functions used by the generation expression.
+      Functions in the generation expression are executed with the privileges
+      of the user executing the query or the function owner, depending on
+      whether the functions are defined as <literal>SECURITY INVOKER</literal>
+      or <literal>SECURITY DEFINER</literal>.
+      <!-- matches create_view.sgml -->
+     </para>
     </listitem>
     <listitem>
      <para>
@@ -519,6 +544,7 @@ CREATE TABLE people (
       <link linkend="sql-createpublication-params-with-publish-generated-columns">
       <literal>publish_generated_columns</literal></link> or by including them
       in the column list of the <command>CREATE PUBLICATION</command> command.
+      This is currently only supported for stored generated columns.
       See <xref linkend="logical-replication-gencols"/> for details.
      </para>
     </listitem>

doc/src/sgml/ref/alter_table.sgml

@@ -102,7 +102,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
   NULL |
   CHECK ( <replaceable class="parameter">expression</replaceable> ) [ NO INHERIT ] |
   DEFAULT <replaceable>default_expr</replaceable> |
-  GENERATED ALWAYS AS ( <replaceable>generation_expr</replaceable> ) STORED |
+  GENERATED ALWAYS AS ( <replaceable>generation_expr</replaceable> ) [ STORED | VIRTUAL ] |
   GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] |
   UNIQUE [ NULLS [ NOT ] DISTINCT ] <replaceable class="parameter">index_parameters</replaceable> |
   PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> |
@@ -264,8 +264,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
     <listitem>
      <para>
       This form replaces the expression of a generated column.  Existing data
-      in the column is rewritten and all the future changes will apply the new
-      generation expression.
+      in a stored generated column is rewritten and all the future changes
+      will apply the new generation expression.
      </para>
     </listitem>
    </varlistentry>
@@ -279,10 +279,15 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
       longer apply the generation expression.
      </para>
 
+     <para>
+      This form is currently only supported for stored generated columns (not
+      virtual ones).
+     </para>
+
      <para>
       If <literal>DROP EXPRESSION IF EXISTS</literal> is specified and the
-      column is not a stored generated column, no error is thrown.  In this
-      case a notice is issued instead.
+      column is not a generated column, no error is thrown.  In this case a
+      notice is issued instead.
      </para>
     </listitem>
    </varlistentry>

doc/src/sgml/ref/create_foreign_table.sgml

@@ -47,7 +47,7 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name
   NULL |
   CHECK ( <replaceable class="parameter">expression</replaceable> ) [ NO INHERIT ] |
   DEFAULT <replaceable>default_expr</replaceable> |
-  GENERATED ALWAYS AS ( <replaceable>generation_expr</replaceable> ) STORED }
+  GENERATED ALWAYS AS ( <replaceable>generation_expr</replaceable> ) [ STORED | VIRTUAL ] }
 [ ENFORCED | NOT ENFORCED ]
 
 <phrase>and <replaceable class="parameter">table_constraint</replaceable> is:</phrase>
@@ -283,7 +283,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
    </varlistentry>
 
    <varlistentry>
-    <term><literal>GENERATED ALWAYS AS ( <replaceable>generation_expr</replaceable> ) STORED</literal><indexterm><primary>generated column</primary></indexterm></term>
+    <term><literal>GENERATED ALWAYS AS ( <replaceable>generation_expr</replaceable> ) [ STORED | VIRTUAL ]</literal><indexterm><primary>generated column</primary></indexterm></term>
     <listitem>
      <para>
       This clause creates the column as a <firstterm>generated
@@ -292,10 +292,13 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
      </para>
 
      <para>
-      The keyword <literal>STORED</literal> is required to signify that the
+      When <literal>VIRTUAL</literal> is specified, the column will be
+      computed when it is read.  (The foreign-data wrapper will see it as a
+      null value in new rows and may choose to store it as a null value or
+      ignore it altogether.)  When <literal>STORED</literal> is specified, the
       column will be computed on write.  (The computed value will be presented
       to the foreign-data wrapper for storage and must be returned on
-      reading.)
+      reading.)  <literal>VIRTUAL</literal> is the default.
      </para>
 
      <para>