Up to [PostgreSQL CVS Repository] / pgsql / src / backend / rewrite
Request diff between arbitrary revisions - Display revisions graphically
Keyword substitution: kv
Default branch: MAIN
pgindent run for 9.0
Update copyright for the year 2010.
Don't treat NEW and OLD as reserved words anymore. For the purposes of rules it works just as well to have them be ordinary identifiers, and this gets rid of a number of ugly special cases. Plus we aren't interfering with non-rule usage of these names. catversion bump because the names change internally in stored rules.
Fix AcquireRewriteLocks to be sure that it acquires the right lock strength when FOR UPDATE is propagated down into a sub-select expanded from a view. Similar bug to parser's isLockedRel issue that I fixed yesterday; likewise seems not quite worth the effort to back-patch.
When FOR UPDATE/SHARE is used with LIMIT, put the LockRows plan node underneath the Limit node, not atop it. This fixes the old problem that such a query might unexpectedly return fewer rows than the LIMIT says, due to LockRows discarding updated rows. There is a related problem that LockRows might destroy the sort ordering produced by earlier steps; but fixing that by pushing LockRows below Sort would create serious performance problems that are unjustified in many real-world applications, as well as potential deadlock problems from locking many more rows than expected. Instead, keep the present semantics of applying FOR UPDATE after ORDER BY within a single query level; but allow the user to specify the other way by writing FOR UPDATE in a sub-select. To make that work, track whether FOR UPDATE appeared explicitly in sub-selects or got pushed down from the parent, and don't flatten a sub-select that contained an explicit FOR UPDATE.
Make FOR UPDATE/SHARE in the primary query not propagate into WITH queries; for example in WITH w AS (SELECT * FROM foo) SELECT * FROM w, bar ... FOR UPDATE the FOR UPDATE will now affect bar but not foo. This is more useful and consistent than the original 8.4 behavior, which tried to propagate FOR UPDATE into the WITH query but always failed due to assorted implementation restrictions. Even though we are in process of removing those restrictions, it seems correct on philosophical grounds to not let the outer query's FOR UPDATE affect the WITH query. In passing, fix isLockedRel which frequently got things wrong in nested-subquery cases: "FOR UPDATE OF foo" applies to an alias foo in the current query level, not subqueries. This has been broken for a long time, but it doesn't seem worth back-patching further than 8.4 because the actual consequences are minimal. At worst the parser would sometimes get RowShareLock on a relation when it should be AccessShareLock or vice versa. That would only make a difference if someone were using ExclusiveLock concurrently, which no standard operation does, and anyway FOR UPDATE doesn't result in visible changes so it's not clear that the someone would notice any problem. Between that and the fact that FOR UPDATE barely works with subqueries at all in existing releases, I'm not excited about worrying about it.
Make FOR UPDATE/SHARE in the primary query not propagate into WITH queries; for example in WITH w AS (SELECT * FROM foo) SELECT * FROM w, bar ... FOR UPDATE the FOR UPDATE will now affect bar but not foo. This is more useful and consistent than the original 8.4 behavior, which tried to propagate FOR UPDATE into the WITH query but always failed due to assorted implementation restrictions. Even though we are in process of removing those restrictions, it seems correct on philosophical grounds to not let the outer query's FOR UPDATE affect the WITH query. In passing, fix isLockedRel which frequently got things wrong in nested-subquery cases: "FOR UPDATE OF foo" applies to an alias foo in the current query level, not subqueries. This has been broken for a long time, but it doesn't seem worth back-patching further than 8.4 because the actual consequences are minimal. At worst the parser would sometimes get RowShareLock on a relation when it should be AccessShareLock or vice versa. That would only make a difference if someone were using ExclusiveLock concurrently, which no standard operation does, and anyway FOR UPDATE doesn't result in visible changes so it's not clear that the someone would notice any problem. Between that and the fact that FOR UPDATE barely works with subqueries at all in existing releases, I'm not excited about worrying about it.
Re-implement EvalPlanQual processing to improve its performance and eliminate a lot of strange behaviors that occurred in join cases. We now identify the "current" row for every joined relation in UPDATE, DELETE, and SELECT FOR UPDATE/SHARE queries. If an EvalPlanQual recheck is necessary, we jam the appropriate row into each scan node in the rechecking plan, forcing it to emit only that one row. The former behavior could rescan the whole of each joined relation for each recheck, which was terrible for performance, and what's much worse could result in duplicated output tuples. Also, the original implementation of EvalPlanQual could not re-use the recheck execution tree --- it had to go through a full executor init and shutdown for every row to be tested. To avoid this overhead, I've associated a special runtime Param with each LockRows or ModifyTable plan node, and arranged to make every scan node below such a node depend on that Param. Thus, by signaling a change in that Param, the EPQ machinery can just rescan the already-built test plan. This patch also adds a prohibition on set-returning functions in the targetlist of SELECT FOR UPDATE/SHARE. This is needed to avoid the duplicate-output-tuple problem. It seems fairly reasonable since the other restrictions on SELECT FOR UPDATE are meant to ensure that there is a unique correspondence between source tuples and result tuples, which an output SRF destroys as much as anything else does.
Fix subquery pullup to wrap a PlaceHolderVar around the entire RowExpr that's generated for a whole-row Var referencing the subquery, when the subquery is in the nullable side of an outer join. The previous coding instead put PlaceHolderVars around the elements of the RowExpr. The effect was that when the outer join made the subquery outputs go to null, the whole-row Var produced ROW(NULL,NULL,...) rather than just NULL. There are arguments afoot about whether those things ought to be semantically indistinguishable, but for the moment they are not entirely so, and the planner needs to take care that its machinations preserve the difference. Per bug #5025. Making this feasible required refactoring ResolveNew() to allow more caller control over what is substituted for a Var. I chose to make ResolveNew() a wrapper around a new general-purpose function replace_rte_variables(). I also fixed the ancient bogosity that ResolveNew might fail to set a query's hasSubLinks field after inserting a SubLink in it. Although all current callers make sure that happens anyway, we've had bugs of that sort before, and it seemed like a good time to install a proper solution. Back-patch to 8.4. The problem can be demonstrated clear back to 8.0, but the fix would be too invasive in earlier branches; not to mention that people may be depending on the subtly-incorrect behavior. The 8.4 series is new enough that fixing this probably won't cause complaints, but it might in older branches. Also, 8.4 shows the incorrect behavior in more cases than older branches do, because it is able to flatten subqueries in more cases.
Fix subquery pullup to wrap a PlaceHolderVar around the entire RowExpr that's generated for a whole-row Var referencing the subquery, when the subquery is in the nullable side of an outer join. The previous coding instead put PlaceHolderVars around the elements of the RowExpr. The effect was that when the outer join made the subquery outputs go to null, the whole-row Var produced ROW(NULL,NULL,...) rather than just NULL. There are arguments afoot about whether those things ought to be semantically indistinguishable, but for the moment they are not entirely so, and the planner needs to take care that its machinations preserve the difference. Per bug #5025. Making this feasible required refactoring ResolveNew() to allow more caller control over what is substituted for a Var. I chose to make ResolveNew() a wrapper around a new general-purpose function replace_rte_variables(). I also fixed the ancient bogosity that ResolveNew might fail to set a query's hasSubLinks field after inserting a SubLink in it. Although all current callers make sure that happens anyway, we've had bugs of that sort before, and it seemed like a good time to install a proper solution. Back-patch to 8.4. The problem can be demonstrated clear back to 8.0, but the fix would be too invasive in earlier branches; not to mention that people may be depending on the subtly-incorrect behavior. The 8.4 series is new enough that fixing this probably won't cause complaints, but it might in older branches. Also, 8.4 shows the incorrect behavior in more cases than older branches do, because it is able to flatten subqueries in more cases.
8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list provided by Andrew.
Revert updatable views
Support column-level privileges, as required by SQL standard. Stephen Frost, with help from KaiGai Kohei and others
Automatic view update rules Bernd Helmle
Update copyright for 2009.
Implement SQL-standard WITH clauses, including WITH RECURSIVE. There are some unimplemented aspects: recursive queries must use UNION ALL (should allow UNION too), and we don't have SEARCH or CYCLE clauses. These might or might not get done for 8.4, but even without them it's a pretty useful feature. There are also a couple of small loose ends and definitional quibbles, which I'll send a memo about to pgsql-hackers shortly. But let's land the patch now so we can get on with other development. Yoshiyuki Asaba, with lots of help from Tatsuo Ishii and Tom Lane
Fix more problems with rewriter failing to set Query.hasSubLinks when inserting a SubLink expression into a rule query. We missed cases where the original query contained a sub-SELECT in a function in FROM, a multi-row VALUES list, or a RETURNING list. Per bug #4434 from Dean Rasheed and subsequent investigation. Back-patch to 8.1; older releases don't have the issue because they didn't try to be smart about setting hasSubLinks only when needed.
Fix more problems with rewriter failing to set Query.hasSubLinks when inserting a SubLink expression into a rule query. We missed cases where the original query contained a sub-SELECT in a function in FROM, a multi-row VALUES list, or a RETURNING list. Per bug #4434 from Dean Rasheed and subsequent investigation. Back-patch to 8.1; older releases don't have the issue because they didn't try to be smart about setting hasSubLinks only when needed.
Fix more problems with rewriter failing to set Query.hasSubLinks when inserting a SubLink expression into a rule query. We missed cases where the original query contained a sub-SELECT in a function in FROM, a multi-row VALUES list, or a RETURNING list. Per bug #4434 from Dean Rasheed and subsequent investigation. Back-patch to 8.1; older releases don't have the issue because they didn't try to be smart about setting hasSubLinks only when needed.
Fix more problems with rewriter failing to set Query.hasSubLinks when inserting a SubLink expression into a rule query. We missed cases where the original query contained a sub-SELECT in a function in FROM, a multi-row VALUES list, or a RETURNING list. Per bug #4434 from Dean Rasheed and subsequent investigation. Back-patch to 8.1; older releases don't have the issue because they didn't try to be smart about setting hasSubLinks only when needed.
Extend the parser location infrastructure to include a location field in most node types used in expression trees (both before and after parse analysis). This allows us to place an error cursor in many situations where we formerly could not, because the information wasn't available beyond the very first level of parse analysis. There's a fair amount of work still to be done to persuade individual ereport() calls to actually include an error location, but this gets the initdb-forcing part of the work out of the way; and the situation is already markedly better than before for complaints about unimplementable implicit casts, such as CASE and UNION constructs with incompatible alternative data types. Per my proposal of a few days ago.
Move exprType(), exprTypmod(), expression_tree_walker(), and related routines into nodes/nodeFuncs, so as to reduce wanton cross-subsystem #includes inside the backend. There's probably more that should be done along this line, but this is a start anyway.
Update copyrights in source tree to 2008.
Small comment spacing improvement.
Fix pgindent to properly handle 'else' and single-line comments on the same line; previous fix was only partial. Re-run pgindent on files that need it.
Make eval_const_expressions() preserve typmod when simplifying something like null::char(3) to a simple Const node. (It already worked for non-null values, but not when we skipped evaluation of a strict coercion function.) This prevents loss of typmod knowledge in situations such as exhibited in bug #3598. Unfortunately there seems no good way to fix that bug in 8.1 and 8.2, because they simply don't carry a typmod for a plain Const node. In passing I made all the other callers of makeNullConst supply "real" typmod values too, though I think it probably doesn't matter anywhere else.
Changes pg_trigger and extend pg_rewrite in order to allow triggers and rules to be defined with different, per session controllable, behaviors for replication purposes. This will allow replication systems like Slony-I and, as has been stated on pgsql-hackers, other products to control the firing mechanism of triggers and rewrite rules without modifying the system catalog directly. The firing mechanisms are controlled by a new superuser-only GUC variable, session_replication_role, together with a change to pg_trigger.tgenabled and a new column pg_rewrite.ev_enabled. Both columns are a single char data type now (tgenabled was a bool before). The possible values in these attributes are: 'O' - Trigger/Rule fires when session_replication_role is "origin" (default) or "local". This is the default behavior. 'D' - Trigger/Rule is disabled and fires never 'A' - Trigger/Rule fires always regardless of the setting of session_replication_role 'R' - Trigger/Rule fires when session_replication_role is "replica" The GUC variable can only be changed as long as the system does not have any cached query plans. This will prevent changing the session role and accidentally executing stored procedures or functions that have plans cached that expand to the wrong query set due to differences in the rule firing semantics. The SQL syntax for changing a triggers/rules firing semantics is ALTER TABLE <tabname> <when> TRIGGER|RULE <name>; <when> ::= ENABLE | ENABLE ALWAYS | ENABLE REPLICA | DISABLE psql's \d command as well as pg_dump are extended in a backward compatible fashion. Jan
Fix up the remaining places where the expression node structure would lose available information about the typmod of an expression; namely, Const, ArrayRef, ArrayExpr, and EXPR and ARRAY SubLinks. In the ArrayExpr and SubLink cases it wasn't really the data structure's fault, but exprTypmod() being lazy. This seems like a good idea in view of the expected increase in typmod usage from Teodor's work to allow user-defined types to have typmods. In particular this responds to the concerns we had about eliminating the special-purpose hack that exprTypmod() used to have for BPCHAR Consts. We can now tell whether or not such a Const has been cast to a specific length, and report or display properly if so. initdb forced due to changes in stored rules.
Fix markQueryForLocking() to work correctly in the presence of nested views. It has been wrong for this case since it was first written for 7.1 :-( Per report from Pavel Han�k.
Fix markQueryForLocking() to work correctly in the presence of nested views. It has been wrong for this case since it was first written for 7.1 :-( Per report from Pavel Han�k.
Fix markQueryForLocking() to work correctly in the presence of nested views. It has been wrong for this case since it was first written for 7.1 :-( Per report from Pavel Han�k.
Fix markQueryForLocking() to work correctly in the presence of nested views. It has been wrong for this case since it was first written for 7.1 :-( Per report from Pavel Han�k.
Fix markQueryForLocking() to work correctly in the presence of nested views. It has been wrong for this case since it was first written for 7.1 :-( Per report from Pavel Han�k.
Fix markQueryForLocking() to work correctly in the presence of nested views. It has been wrong for this case since it was first written for 7.1 :-( Per report from Pavel Han�k.
Wording cleanup for error messages. Also change can't -> cannot. Standard English uses "may", "can", and "might" in different ways: may - permission, "You may borrow my rake." can - ability, "I can lift that log." might - possibility, "It might rain today." Unfortunately, in conversational English, their use is often mixed, as in, "You may use this variable to do X", when in fact, "can" is a better choice. Similarly, "It may crash" is better stated, "It might crash".
Update CVS HEAD for 2007 copyright. Back branches are typically not back-stamped for this.
Message style improvements
pgindent run for 8.2.
Apply a simple solution to the problem of making INSERT/UPDATE/DELETE RETURNING play nice with views/rules. To wit, have the rule rewriter rewrite any RETURNING clause found in a rule to produce what the rule's triggering query asked for in its RETURNING clause, in particular drop the RETURNING clause if no RETURNING in the triggering query. This leaves the responsibility for knowing how to produce the view's output columns on the rule author, without requiring any fundamental changes in rule semantics such as adding new rule event types would do. The initial implementation constrains things to ensure that there is exactly one, unconditionally invoked RETURNING clause among the rules for an event --- later we might be able to relax that, but for a post feature freeze fix it seems better to minimize how much invention we do. Per gripe from Jaime Casanova.
Add support for multi-row VALUES clauses as part of INSERT statements (e.g. "INSERT ... VALUES (...), (...), ...") and elsewhere as allowed by the spec. (e.g. similar to a FROM clause subselect). initdb required. Joe Conway and Tom Lane.
Remove 576 references of include files that were not needed.
Klugy fix for bug #2447: we can't expand a whole-row reference to NEW in a rule WHERE expression while inserting it into the original query, because the 8.0 ResolveNew API is wrongly designed. This is fixed in 8.1 but I'm disinclined to risk back-porting the changes. Instead, just stop the coredump and instead issue the same 'cannot handle whole-row reference' message that 7.4 and before generated in this situation.
Improve the representation of FOR UPDATE/FOR SHARE so that we can support both FOR UPDATE and FOR SHARE in one command, as well as both NOWAIT and normal WAIT behavior. The more general code is actually simpler and cleaner.
Fix a bunch of problems with domains by making them use special input functions that apply the necessary domain constraint checks immediately. This fixes cases where domain constraints went unchecked for statement parameters, PL function local variables and results, etc. We can also eliminate existing special cases for domains in places that had gotten it right, eg COPY. Also, allow domains over domains (base of a domain is another domain type). This almost worked before, but was disallowed because the original patch hadn't gotten it quite right.
Update copyright for 2006. Update scripts.
Fix problems with rewriter failing to set Query.hasSubLinks when inserting a SubLink expression into a rule query. Pre-8.1 we essentially did this unconditionally; 8.1 tries to do it only when needed, but was missing a couple of cases. Per report from Kyle Bateman. Add some regression test cases covering this area.
Fix problems with rewriter failing to set Query.hasSubLinks when inserting a SubLink expression into a rule query. Pre-8.1 we essentially did this unconditionally; 8.1 tries to do it only when needed, but was missing a couple of cases. Per report from Kyle Bateman. Add some regression test cases covering this area.
Re-run pgindent, fixing a problem where comment lines after a blank comment line where output as too long, and update typedefs for /lib directory. Also fix case where identifiers were used as variable names in the backend, but as typedefs in ecpg (favor the backend for indenting). Backpatch to 8.1.X.
Re-run pgindent, fixing a problem where comment lines after a blank comment line where output as too long, and update typedefs for /lib directory. Also fix case where identifiers were used as variable names in the backend, but as typedefs in ecpg (favor the backend for indenting). Backpatch to 8.1.X.
Standard pgindent run for 8.1.
Add NOWAIT option to SELECT FOR UPDATE/SHARE. Original patch by Hans-Juergen Schoenig, revisions by Karel Zak and Tom Lane.
Make use of new list primitives list_append_unique and list_concat_unique where applicable.
Replace pg_shadow and pg_group by new role-capable catalogs pg_authid and pg_auth_members. There are still many loose ends to finish in this patch (no documentation, no regression tests, no pg_dump support for instance). But I'm going to commit it now anyway so that Alvaro can make some progress on shared dependencies. The catalog changes should be pretty much done.
Change expandRTE() and ResolveNew() back to taking just the single RTE of interest, rather than the whole rangetable list. This makes the API more understandable and avoids duplicate RTE lookups. This patch reverts no-longer-needed portions of my patch of 2004-08-19.
Revise handling of dropped columns in JOIN alias lists to avoid a performance problem pointed out by phil@vodafone: to wit, we were spending O(N^2) time to check dropped-ness in an N-deep join tree, even in the case where the tree was freshly constructed and couldn't possibly mention any dropped columns. Instead of recursing in get_rte_attribute_is_dropped(), change the data structure definition: the joinaliasvars list of a JOIN RTE must have a NULL Const instead of a Var at any position that references a now-dropped column. This costs nothing during normal parse-rewrite-plan path, and instead we have a linear-time update to make when loading a stored rule that might contain now-dropped columns. While at it, move the responsibility for acquring locks on relations referenced by rules into this separate function (which I therefore chose to call AcquireRewriteLocks). This saves effort --- namely, duplicated lock grabs in parser and rewriter --- in the normal path at a cost of one extra non-locked heap_open() in the stored-rule path; seems a good tradeoff. A fringe benefit is that it is now *much* clearer that we acquire lock on relations referenced in rules before we make any rewriter decisions based on their properties. (I don't know of any bug of that ilk, but it wasn't exactly clear before.)
Avoid unnecessary call of rangeTableEntry_used() for the result relation of a query.
Implement sharable row-level locks, and use them for foreign key references to eliminate unnecessary deadlocks. This commit adds SELECT ... FOR SHARE paralleling SELECT ... FOR UPDATE. The implementation uses a new SLRU data structure (managed much like pg_subtrans) to represent multiple- transaction-ID sets. When more than one transaction is holding a shared lock on a particular row, we create a MultiXactId representing that set of transactions and store its ID in the row's XMAX. This scheme allows an effectively unlimited number of row locks, just as we did before, while not costing any extra overhead except when a shared lock actually has to be shared. Still TODO: use the regular lock manager to control the grant order when multiple backends are waiting for a row lock. Alvaro Herrera and Tom Lane.
Merge Resdom nodes into TargetEntry nodes to simplify code and save a few palloc's. I also chose to eliminate the restype and restypmod fields entirely, since they are redundant with information stored in the node's contained expression; re-examining the expression at need seems simpler and more reliable than trying to keep restype/restypmod up to date. initdb forced due to change in contents of stored rules.
Rewrite rewriteTargetList() to avoid O(N^2) behavior on wide target lists.
Make the behavior of HAVING without GROUP BY conform to the SQL spec. Formerly, if such a clause contained no aggregate functions we mistakenly treated it as equivalent to WHERE. Per spec it must cause the query to be treated as a grouped query of a single group, the same as appearance of aggregate functions would do. Also, the HAVING filter must execute after aggregate function computation even if it itself contains no aggregate functions.
Tag appropriate files for rc3 Also performed an initial run through of upgrading our Copyright date to extend to 2005 ... first run here was very simple ... change everything where: grep 1996-2004 && the word 'Copyright' ... scanned through the generated list with 'less' first, and after, to make sure that I only picked up the right entries ...
Fix one more place where we were expecting lcons() to be nondestructive to the original List; per report from Sebastian B�ck. I think this is the last such bug --- I examined every lcons() call in the backend and the rest seem OK --- but it's nervous-making that we're still finding 'em so many months after the List rewrite went in.
When implementing a coercion to a domain type with a combined type-and-length coercion function, make sure that the coercion function is told the correct typmod. Fixes Kris Jurka's example of a domain over bit(N).
Pgindent run for 8.0.
Update copyright to 2004.
Repair some issues with column aliases and RowExpr construction in the presence of dropped columns. Document the already-presumed fact that eref aliases in relation RTEs are supposed to have entries for dropped columns; cause the user alias structs to have such entries too, so that there's always a one-to-one mapping to the underlying physical attnums. Adjust expandRTE() and related code to handle the case where a column that is part of a JOIN has been dropped. Generalize expandRTE()'s API so that it can be used in a couple of places that formerly rolled their own implementation of the same logic. Fix ruleutils.c to suppress display of aliases for columns that were dropped since the rule was made.
Don't try to rewrite NEW references in a utility statement in a rule. There won't be any, and in fact there won't even be an RTE for NEW, which was leading to a core dump in CVS tip. 7.4 and earlier manage not to crash when applying ResolveNew in this scenario, but I think it was just good fortune that they didn't. Per report from Bernd Helmle.
Represent type-specific length coercion functions as pg_cast entries, eliminating the former hard-wired convention about their names. Allow pg_cast entries to represent both type coercion and length coercion in a single step --- this is represented by a function that takes an extra typmod argument, just like a length coercion function. This nicely merges the type and length coercion mechanisms into something at least a little cleaner than we had before. Make use of the single- coercion-step behavior to fix integer-to-bit coercion so that coercing to bit(n) yields the rightmost n bits of the integer instead of the leftmost n bits. This should fix recurrent complaints about the odd behavior of this coercion. Clean up the documentation of the bit string functions, and try to put it where people might actually find it. Also, get rid of the unreliable heuristics in ruleutils.c about whether to display nested coercion steps; instead require parse_coerce.c to label them properly in the first place.
Support assignment to subfields of composite columns in UPDATE and INSERT. As a side effect, cause subscripts in INSERT targetlists to do something more or less sensible; previously we evaluated such subscripts and then effectively ignored them. Another side effect is that UPDATE-ing an element or slice of an array value that is NULL now produces a non-null result, namely an array containing just the assigned-to positions.
Use the new List API function names throughout the backend, and disable the list compatibility API by default. While doing this, I decided to keep the llast() macro around and introduce llast_int() and llast_oid() variants.
Fix another place that assumed 'x = lcons(y, z)' would not have any side-effect on the original list z. I fear we have a few more of these to track down yet :-(.
Reimplement the linked list data structure used throughout the backend. In the past, we used a 'Lispy' linked list implementation: a "list" was merely a pointer to the head node of the list. The problem with that design is that it makes lappend() and length() linear time. This patch fixes that problem (and others) by maintaining a count of the list length and a pointer to the tail node along with each head node pointer. A "list" is now a pointer to a structure containing some meta-data about the list; the head and tail pointers in that structure refer to ListCell structures that maintain the actual linked list of nodes. The function names of the list API have also been changed to, I hope, be more logically consistent. By default, the old function names are still available; they will be disabled-by-default once the rest of the tree has been updated to use the new API names.
Promote row expressions to full-fledged citizens of the expression syntax, rather than allowing them only in a few special cases as before. In particular you can now pass a ROW() construct to a function that accepts a rowtype parameter. Internal generation of RowExprs fixes a number of corner cases that used to not work very well, such as referencing the whole-row result of a JOIN or subquery. This represents a further step in the work I started a month or so back to make rowtype values into first-class citizens.
Replace TupleTableSlot convention for whole-row variables and function results with tuples as ordinary varlena Datums. This commit does not in itself do much for us, except eliminate the horrid memory leak associated with evaluation of whole-row variables. However, it lays the groundwork for allowing composite types as table columns, and perhaps some other useful features as well. Per my proposal of a few days ago.
Fix permission-checking bug reported by Tim Burgess 10-Feb-03 (this time for sure...). Rather than relying on the query context of a rangetable entry to identify what permissions it wants checked, store a full AclMode mask in each RTE, and check exactly those bits. This allows an RTE specifying, say, INSERT privilege on a view to be copied into a derived UPDATE query without changing meaning. Per recent discussion thread. initdb forced due to change of stored rule representation.
Revert ill-starred change of 13-Feb-02: it appeared to fix a problem of incorrect permissions checking, but in fact disabled most all permissions checks for view updates. This corrects problems reported by Sergey Yatskevich among others, at the cost of re-introducing the problem previously reported by Tim Burgess. However, since we'd lived with that problem for quite awhile without knowing it, we can live with it awhile longer until a proper fix can be made in 7.5.
Revert ill-starred change of 13-Feb-02: it appeared to fix a problem of incorrect permissions checking, but in fact disabled most all permissions checks for view updates. This corrects problems reported by Sergey Yatskevich among others, at the cost of re-introducing the problem previously reported by Tim Burgess. However, since we'd lived with that problem for quite awhile without knowing it, we can live with it awhile longer until a proper fix can be made in 7.5.
Revert ill-starred change of 13-Feb-02: it appeared to fix a problem of incorrect permissions checking, but in fact disabled most all permissions checks for view updates. This corrects problems reported by Sergey Yatskevich among others, at the cost of re-introducing the problem previously reported by Tim Burgess. However, since we'd lived with that problem for quite awhile without knowing it, we can live with it awhile longer until a proper fix can be made in 7.5.
$Header: -> $PostgreSQL Changes ...
Message editing: remove gratuitous variations in message wording, standardize terms, add some clarifications, fix some untranslatable attempts at dynamic message building.
Rewriter and planner should use only resno, not resname, to identify target columns in INSERT and UPDATE targetlists. Don't rely on resname to be accurate in ruleutils, either. This fixes bug reported by Donald Fraser, in which renaming a column referenced in a rule did not work very well.
Another pgindent run with updated typedefs.
Update copyrights to 2003.
pgindent run.
Coerce unknown-literal-constant default values to the column type during CREATE TABLE (or ALTER TABLE SET DEFAULT), rather than postponing it to the time that the default is inserted into an INSERT command by the rewriter. This reverses an old decision that was intended to make the world safe for writing f1 timestamp default 'now' but in fact merely made the failure modes subtle rather than obvious. Per recent trouble report and followup discussion. initdb forced since there is a chance that stored default expressions will change.
Error message editing in backend/optimizer, backend/rewrite.
Add defenses against trying to attach qual conditions to a setOperation query node, since that won't work unless the planner is upgraded. Someday we should try to support at least some cases of this, but for now just plug the hole in the dike. Per discussion with Dmitry Tkach.
Code review for UPDATE tab SET col = DEFAULT patch ... whack it around so it has some chance of working in rules ...
UPDATE ... SET <col> = DEFAULT Rod Taylor
Portal and memory management infrastructure for extended query protocol. Both plannable queries and utility commands are now always executed within Portals, which have been revamped so that they can handle the load (they used to be good only for single SELECT queries). Restructure code to push command-completion-tag selection logic out of postgres.c, so that it won't have to be duplicated between simple and extended queries. initdb forced due to addition of a field to Query nodes.
Infrastructure for deducing Param types from context, in the same way that the types of untyped string-literal constants are deduced (ie, when coerce_type is applied to 'em, that's what the type must be). Remove the ancient hack of storing the input Param-types array as a global variable, and put the info into ParseState instead. This touches a lot of files because of adjustment of routine parameter lists, but it's really not a large patch. Note: PREPARE statement still insists on exact specification of parameter types, but that could easily be relaxed now, if we wanted to do so.
Remove REWRITE_INVOKE_MAX in favor of making an accurate check for recursion in RewriteQuery(); also, detect recursion in fireRIRrules(), so as to catch self-referential views per example from Ryan VanderBijl. Minor code restructuring to make it easier to catch recursive case.
Repair rule permissions-checking bug reported by Tim Burgess 10-Feb-02: the table(s) modified by the original query would get checked for the type of write permission needed by a rule query.
Repair rule permissions-checking bug reported by Tim Burgess 10-Feb-02: the table(s) modified by the original query would get checked for the type of write permission needed by a rule query.
Adjust API of expression_tree_mutator and query_tree_mutator to simplify callers. It turns out the common case is that the caller does want to recurse into sub-queries, so push support for that into these subroutines.
To suppress memory leakage in long-lived Lists, lremove() should pfree the cons cell it's deleting from the list. Do this, and fix a few callers that were bogusly assuming it wouldn't free the cons cell.
Phase 2 of read-only-plans project: restructure expression-tree nodes so that all executable expression nodes inherit from a common supertype Expr. This is somewhat of an exercise in code purity rather than any real functional advance, but getting rid of the extra Oper or Func node formerly used in each operator or function call should provide at least a little space and speed improvement. initdb forced by changes in stored-rules representation.
Rule rewriter was doing the wrong thing with conditional INSTEAD rules whose conditions might yield NULL. The negated qual to attach to the original query is properly 'x IS NOT TRUE', not 'NOT x'. This fix produces correct behavior, but we may be taking a performance hit because the planner is much stupider about IS NOT TRUE than it is about NOT clauses. Future TODO: teach prepqual, other parts of planner how to cope with BooleanTest clauses more effectively.
Fix rewrite code so that rules are in fact executed in order by name, rather than being reordered according to INSTEAD attribute for implementation convenience. Also, increase compiled-in recursion depth limit from 10 to 100 rewrite cycles. 10 seems pretty marginal for situations where multiple rules exist for the same query. There was a complaint about this recently, so I'm going to bump it up. (Perhaps we should make the limit a GUC parameter, but that's too close to being a new feature to do in beta.)
Adjust handling of command status strings in the presence of rules, as per recent pghackers discussions. initdb forced due to change in fields of stored Query nodes.
Extend pg_cast castimplicit column to a three-way value; this allows us to be flexible about assignment casts without introducing ambiguity in operator/function resolution. Introduce a well-defined promotion hierarchy for numeric datatypes (int2->int4->int8->numeric->float4->float8). Change make_const to initially label numeric literals as int4, int8, or numeric (never float8 anymore). Explicitly mark Func and RelabelType nodes to indicate whether they came from a function call, explicit cast, or implicit cast; use this to do reverse-listing more accurately and without so many heuristics. Explicit casts to char, varchar, bit, varbit will truncate or pad without raising an error (the pre-7.2 behavior), while assigning to a column without any explicit cast will still raise an error for wrong-length data like 7.3. This more nearly follows the SQL spec than 7.2 behavior (we should be reporting a 'completion condition' in the explicit-cast cases, but we have no mechanism for that, so just do silent truncation). Fix some problems with enforcement of typmod for array elements; it didn't work at all in 'UPDATE ... SET array[n] = foo', for example. Provide a generalized array_length_coerce() function to replace the specialized per-array-type functions that used to be needed (and were missing for NUMERIC as well as all the datetime types). Add missing conversions int8<->float4, text<->numeric, oid<->int8. initdb forced.
Tweak querytree-dependency-extraction code so that columns of tables that are explicitly JOINed are not considered dependencies unless they are actually used in the query: mere presence in the joinaliasvars list of a JOIN RTE doesn't count as being used. The patch touches a number of files because I needed to generalize the API of query_tree_walker to support an additional flag bit, but the changes are otherwise quite small.
pgindent run.
Rephrase 'Cannot insert into a view' and related messages, per pghackers discussion around 31-Jul-02.
Modify array operations to include array's element type OID in the array header, and to compute sizing and alignment of array elements the same way normal tuple access operations do --- viz, using the tupmacs.h macros att_addlength and att_align. This makes the world safe for arrays of cstrings or intervals, and should make it much easier to write array-type-polymorphic functions; as examples see the cleanups of array_out and contrib/array_iterator. By Joe Conway and Tom Lane.
ALTER TABLE DROP COLUMN works. Patch by Christopher Kings-Lynne, code review by Tom Lane. Remaining issues: functions that take or return tuple types are likely to break if one drops (or adds!) a column in the table defining the type. Need to think about what to do here. Along the way: some code review for recent COPY changes; mark system columns attnotnull = true where appropriate, per discussion a month ago.
The attached patch (against HEAD) implements COPY x (a,d,c,b) from stdin; COPY x (a,c) to stdout; as well as the corresponding changes to pg_dump to use the new functionality. This functionality is not available when using the BINARY option. If a column is not specified in the COPY FROM statement, its default values will be used. In addition to this functionality, I tweaked a couple of the error messages emitted by the new COPY <options> checks. Brent Verner
Update copyright to 2002.
Restructure representation of aggregate functions so that they have pg_proc entries, per pghackers discussion. This fixes aggregates to live in namespaces, and also simplifies/speeds up lookup in parse_func.c. Also, add a 'proimplicit' flag to pg_proc that controls whether a type coercion function may be invoked implicitly, or only explicitly. The current settings of these flags are more permissive than I would like, but we will need to debate and refine the behavior; for now, I avoided breaking regression tests as much as I could.
Undo not-so-hot decision to postpone insertion of default values into INSERT statements to the planner. Taking it out of the parser was right (so that defaults don't get into stored rules), but it has to happen before rewrite rule expansion, else references to NEW.field behave incorrectly. Accordingly, add a step to the rewriter to insert defaults just before rewrite-rule expansion.
A little further progress on schemas: push down RangeVars into addRangeTableEntry calls. Remove relname field from RTEs, since it will no longer be a useful unique identifier of relations; we want to encourage people to rely on the relation OID instead. Further work on dumping qual expressions in EXPLAIN, too.
Restructure representation of join alias variables. An explicit JOIN now has an RTE of its own, and references to its outputs now are Vars referencing the JOIN RTE, rather than CASE-expressions. This allows reverse-listing in ruleutils.c to use the correct alias easily, rather than painfully reverse-engineering the alias namespace as it used to do. Also, nested FULL JOINs work correctly, because the result of the inner joins are simple Vars that the planner can cope with. This fixes a bug reported a couple times now, notably by Tatsuo on 18-Nov-01. The alias Vars are expanded into COALESCE expressions where needed at the very end of planning, rather than during parsing. Also, beginnings of support for showing plan qualifier expressions in EXPLAIN. There are probably still cases that need work. initdb forced due to change of stored-rule representation.
pgindent run on all C files. Java run to follow. initdb/regression tests pass.
Fix rule rewriter so that new ordering of ON INSERT actions applies in cases of qualified rules as well as unqualified ones. Tweak rules test to avoid cluttering output with dummy SELECT results. Update documentation to match code.
Fire rule actions ON INSERT after original statement (if not INSTEAD). Jan
Remove some dead code, simplify calling convention.
Repair problem with multi-action rules in combination with any nontrivial manipulation of rtable/jointree by planner. Rewriter was generating actions that shared rtable/jointree substructure, which caused havoc when planner got to the later actions that it'd already mucked up.
Repair problem with multi-action rules in combination with any nontrivial manipulation of rtable/jointree by planner. Rewriter was generating actions that shared rtable/jointree substructure, which caused havoc when planner got to the later actions that it'd already mucked up.
Permissions were not checked correctly when one view invokes another. Per bug report from Lieven Van Acker, 5/2/01.
Add some defenses to guard against case where a rule refers to a table or view that's been dropped and then recreated with the same name (but, perhaps, different columns). Eventually we'd like to support this but for now all we can do is fail cleanly, rather than possibly coredumping if we proceed using the obsolete rule.
Remove dashes in comments that don't need them, rewrap with pgindent.
pgindent run. Make it all clean.
Repair bug reported by Huxton, 1/24/01. We need to include a rule's original table ('OLD' table) in its join tree if OLD is referenced by either the rule action, the rule qual, or the original query qual that will be added to the rule action. However, we only want one instance of the original table to be included; so beware of the possibility that the rule action already has a jointree entry for OLD.
Change Copyright from PostgreSQL, Inc to PostgreSQL Global Development Group.
Fix breakage of rules using NOTIFY actions, per bug report and patch from sergiop@sinectis.com.ar.
Make application of FOR UPDATE to a view work exactly like the parser's transformForUpdate does: it should recurse into subqueries.
Clean up handling of FOR UPDATE inside views and subselects ... make it work where we can (given that the executor only handles it at top level) and generate an error where we can't. Note that while the parser has been allowing views to say SELECT FOR UPDATE for a few weeks now, that hasn't actually worked until just now.
Repair breakage of rules containing INSERT ... SELECT actions, per bug report from Joel Burton. Turns out that my simple idea of turning the SELECT into a subquery does not interact well *at all* with the way the rule rewriter works. Really what we need to make INSERT ... SELECT work cleanly is to decouple targetlists from rangetables: an INSERT ... SELECT wants to have two levels of targetlist but only one rangetable. No time for that for 7.1, however, so I've inserted some ugly hacks to make the rewriter know explicitly about the structure of INSERT ... SELECT queries. Ugh :-(
Make DROP TABLE rollback-able: postpone physical file delete until commit. (WAL logging for this is not done yet, however.) Clean up a number of really crufty things that are no longer needed now that DROP behaves nicely. Make temp table mapper do the right things when drop or rename affecting a temp table is rolled back. Also, remove "relation modified while in use" error check, in favor of locking tables at first reference and holding that lock throughout the statement.
Reimplementation of UNION/INTERSECT/EXCEPT. INTERSECT/EXCEPT now meet the SQL92 semantics, including support for ALL option. All three can be used in subqueries and views. DISTINCT and ORDER BY work now in views, too. This rewrite fixes many problems with cross-datatype UNIONs and INSERT/SELECT where the SELECT yields different datatypes than the INSERT needs. I did that by making UNION subqueries and SELECT in INSERT be treated like subselects-in-FROM, thereby allowing an extra level of targetlist where the datatype conversions can be inserted safely. INITDB NEEDED!
Subselects in FROM clause, per ISO syntax: FROM (SELECT ...) [AS] alias. (Don't forget that an alias is required.) Views reimplemented as expanding to subselect-in-FROM. Grouping, aggregates, DISTINCT in views actually work now (he says optimistically). No UNION support in subselects/views yet, but I have some ideas about that. Rule-related permissions checking moved out of rewriter and into executor. INITDB REQUIRED!
First cut at full support for OUTER JOINs. There are still a few loose ends to clean up (see my message of same date to pghackers), but mostly it works. INITDB REQUIRED!
Code cleanup of user name and user id handling in the backend. The current user is now defined in terms of the user id, the user name is only computed upon request (for display purposes). This is kind of the opposite of the previous state, which would maintain the user name and compute the user id for permission checks. Besides perhaps saving a few cycles (integer vs string), this now creates a single point of attack for changing the user id during a connection, for purposes of "setuid" functions, etc.
Remove 'func_tlist' from Func expression nodes, likewise 'param_tlist' from Param nodes, per discussion a few days ago on pghackers. Add new expression node type FieldSelect that implements the functionality where it's actually needed. Clean up some other unused fields in Func nodes as well. NOTE: initdb forced due to change in stored expression trees for rules.
Use a private memory context to store rule information in each relcache entry that has rules. This allows us to release the rule parsetrees on relcache flush without needing a working freeObject() routine. Formerly, the rule trees were leaked permanently at relcache flush. Also, clean up handling of rule creation and deletion --- there was not sufficient locking of the relation being modified, and there was no reliable notification of other backends that a relcache reload was needed. Also, clean up relcache.c code so that scans of system tables needed to load a relcache entry are done in the caller's memory context, not in CacheMemoryContext. This prevents any un-pfreed memory from those scans from becoming a permanent memory leak.
Clean up #include's.
Rename rule CURRENT to OLD in source tree. Add mapping for backward compatiblity with old rules.
Remove unused include files. Do not touch /port or includes used by defines.
Except_Intersect_Rewrite() failed to ignore resjunk targetlist entries, thus causing failure if one sub-select had resjunk entries that the other did not (cf. bug report from Espinosa 4/27/00).
Correct error in rewriter that caused SELECT count(*) FROM view to give wrong results: it should be looking at inJoinSet not inFromCl. Also, make 'modified' flag be local to ApplyRetrieveRule: we should append a rule's quals to the query iff that particular rule applies, not if we have fired any previously-considered rule for the query!
Ye-old pgindent run. Same 4-space tabs.
When rewriting an aggregate introduced into WHERE, allow agg argument to be an expression not just a simple Var, so long as only one table is referenced (so that code isn't really any more difficult than before). This whole thing is still fundamentally bogus, but at least we can accept a few more cases than before.
Fix some (more) problems with subselects in rules. Rewriter failed to mark query as having subselects if a subselect was added from a rule WHERE condition (as opposed to a rule action). Also, fix adjustment of varlevelsup so that it actually has some prospect of working when inserting an expression containing a subselect into a subquery.
Fix performance problem in fireRIRonSubselect: with nested subqueries, fireRIRonSubselect was invoked twice at each subselect, leading to an exponential amount of wasted effort.
Redesign DISTINCT ON as discussed in pgsql-sql 1/25/00: syntax is now SELECT DISTINCT ON (expr [, expr ...]) targetlist ... and there is a check to make sure that the user didn't specify an ORDER BY that's incompatible with the DISTINCT operation. Reimplement nodeUnique and nodeGroup to use the proper datatype-specific equality function for each column being compared --- they used to do bitwise comparisons or convert the data to text strings and strcmp(). (To add insult to injury, they'd look up the conversion functions once for each tuple...) Parse/plan representation of DISTINCT is now a list of SortClause nodes. initdb forced by querytree change...
Add: * Portions Copyright (c) 1996-2000, PostgreSQL, Inc to all files copyright Regents of Berkeley. Man, that's a lot of files.
Pass atttypmod to CoerceTargetExpr, so that it can pass it on to coerce_type, so that the right things happen when coercing a previously- unknown constant to a destination data type.
Fix it's and its to be correct.
Implement subselects in target lists. Also, relax requirement that subselects can only appear on the righthand side of a binary operator. That's still true for quantified predicates like x = ANY (SELECT ...), but a subselect that delivers a single result can now appear anywhere in an expression. This is implemented by changing EXPR_SUBLINK sublinks to represent just the (SELECT ...) expression, without any 'left hand side' or combining operator --- so they're now more like EXISTS_SUBLINK. To handle the case of '(x, y, z) = (SELECT ...)', I added a new sublink type MULTIEXPR_SUBLINK, which acts just like EXPR_SUBLINK used to. But the grammar will only generate one for a multiple-left-hand-side row expression.
Eliminate RewritePreprocessQuery, which was taking an unreasonable amount of time to clean up after a vanished parser problem. Don't call fireRIRonSubselect when we know there are no subselects, either.
Back-patch LIMIT + INTERSECT fix into 6.5.*.
Except_Intersect_Rewrite() forgot to move LIMIT info to new topmost SELECT node after rearranging query tree.
Fix planner and rewriter to follow SQL semantics for tables that are mentioned in FROM but not elsewhere in the query: such tables should be joined over anyway. Aside from being more standards-compliant, this allows removal of some very ugly hacks for COUNT(*) processing. Also, allow HAVING clause without aggregate functions, since SQL does. Clean up CREATE RULE statement-list syntax the same way Bruce just fixed the main stmtmulti production. CAUTION: addition of a field to RangeTblEntry nodes breaks stored rules; you will have to initdb if you have any rules.
Stick finger into a couple more holes in the leaky dike of modifyAggrefQual. This routine really, really needs to be retired, but until we have subselects in FROM there's no chance of doing the job right. In the meantime try to respond to unhandlable cases with elog rather than coredump.
Clean up rewriter routines to use expression_tree_walker and expression_tree_mutator rather than ad-hoc tree walking code. This shortens the code materially and fixes a fair number of sins of omission. Also, change modifyAggrefQual to *not* recurse into subselects, since its mission is satisfied if it removes aggregate functions from the top level of a WHERE clause. This cures problems with queries of the form SELECT ... WHERE x IN (SELECT ... HAVING something-using-an-aggregate), which would formerly get mucked up by modifyAggrefQual. The routine is still fundamentally broken, of course, but I don't think there's any way to get rid of it before we implement subselects in FROM ...
Remove incorrect 'Assert(targetList != NULL)'. An INSERT ... DEFAULT VALUES statement does indeed have a null targetlist, at least during parse and rewrite stages.
Mega-commit to make heap_open/heap_openr/heap_close take an additional argument specifying the kind of lock to acquire/release (or 'NoLock' to do no lock processing). Ensure that all relations are locked with some appropriate lock level before being examined --- this ensures that relevant shared-inval messages have been processed and should prevent problems caused by concurrent VACUUM. Fix several bugs having to do with mismatched increment/decrement of relation ref count and mismatched heap_open/close (which amounts to the same thing). A bogus ref count on a relation doesn't matter much *unless* a SI Inval message happens to arrive at the wrong time, which is probably why we got away with this sloppiness for so long. Repair missing grab of AccessExclusiveLock in DROP TABLE, ALTER/RENAME TABLE, etc, as noted by Hiroshi. Recommend 'make clean all' after pulling this update; I modified the Relation struct layout slightly. Will post further discussion to pghackers list shortly.
Revise implementation of SubLinks so that there is a consistent, documented intepretation of the lefthand and oper fields. Fix a number of obscure problems while at it --- for example, the old code failed if the parser decided to insert a type-coercion function just below the operator of a SubLink. CAUTION: this will break stored rules that contain subplans. You may need to initdb.
Move some system includes into c.h, and remove duplicates.
Final cleanup.
Update #include cleanups
Remove unused #includes in *.c files.
Clean up #include in /include directory. Add scripts for checking includes.
Remove S*I comments from Stephan.
RewritePreprocessQuery tried to match resjunk targets against result relation ... wrong ...
Replace rewriter's checkQueryHasAggs and checkQueryHasSubLink with expression_tree_walker-based code. The former failed to cope with expressions containing SubLinks, and the latter returned TRUE for both SubLinks and Aggrefs (cut-and-paste bug?). There is a lot more scope for using expression_tree_walker in this module, but I'll restrain myself until the 6.6 split occurs from touching not-demonstrably-broken code.
Make functions static or NOT_USED as appropriate.
pgindent run over code.
Bugfix - Range table entries that are unused after rewriting should not be marked inFromCl any longer. Otherwise the planner gets confused and joins over them where in fact it does not have to. Adjust hasSubLinks now with a recursive lookup - could be wrong in multi action rules because parse state isn't reset correctly and all actions in the rule are marked hasSubLinks if one of them has. Jan
Skip junk nodes when comparing UNION target list lengths.
Change resjunk to a boolean.
Rip out QueryTreeList structure, root and branch. Querytree lists are now plain old garden-variety Lists, allocated with palloc, rather than specialized expansible-array data allocated with malloc. This substantially simplifies their handling and eliminates several sources of memory leakage. Several basic types of erroneous queries (syntax error, attempt to insert a duplicate key into a unique index) now demonstrably leak zero bytes per query.
Fixed wrong hasAggs when aggregate columns of view aren't selected. Disabled ability of defining DISTINCT or ORDER BY on views. Jan
Replaced targetlist entry in GroupClause by reference number in Resdom and GroupClause so changing of resno's doesn't confuse the grouping any more. Jan
Rearrange top-level rewrite operations so that EXPLAIN works on queries involving UNION, EXCEPT, INTERSECT.
Final optimizer cleanups.
From: Tatsuo Ishii <t-ishii@sra.co.jp> Ok. I made patches replacing all of "#if FALSE" or "#if 0" to "#ifdef NOT_USED" for current. I have tested these patches in that the postgres binaries are identical.
Change my-function-name-- to my_function_name, and optimizer renames.
Cleanup of source files where 'return' or 'var =' is alone on a line.
Add TEMP tables/indexes. Add COPY pfree(). Other cleanups.
Agg/Aggreg cleanup and datetime.sql patch.
SELECT FOR UPDATE is implemented...
Rename Aggreg to Aggref.
FOR UPDATE is in parser & rules.
Hi! INTERSECT and EXCEPT is available for postgresql-v6.4! The patch against v6.4 is included at the end of the current text (in uuencoded form!) I also included the text of my Master's Thesis. (a postscript version). I hope that you find something of it useful and would be happy if parts of it find their way into the PostgreSQL documentation project (If so, tell me, then I send the sources of the document!) The contents of the document are: -) The first chapter might be of less interest as it gives only an overview on SQL. -) The second chapter gives a description on much of PostgreSQL's features (like user defined types etc. and how to use these features) -) The third chapter starts with an overview of PostgreSQL's internal structure with focus on the stages a query has to pass (i.e. parser, planner/optimizer, executor). Then a detailed description of the implementation of the Having clause and the Intersect/Except logic is given. Originally I worked on v6.3.2 but never found time enough to prepare and post a patch. Now I applied the changes to v6.4 to get Intersect and Except working with the new version. Chapter 3 of my documentation deals with the changes against v6.3.2, so keep that in mind when comparing the parts of the code printed there with the patched sources of v6.4. Here are some remarks on the patch. There are some things that have still to be done but at the moment I don't have time to do them myself. (I'm doing my military service at the moment) Sorry for that :-( -) I used a rewrite technique for the implementation of the Except/Intersect logic which rewrites the query to a semantically equivalent query before it is handed to the rewrite system (for views, rules etc.), planner, executor etc. -) In v6.3.2 the types of the attributes of two select statements connected by the UNION keyword had to match 100%. In v6.4 the types only need to be familiar (i.e. int and float can be mixed). Since this feature did not exist when I worked on Intersect/Except it does not work correctly for Except/Intersect queries WHEN USED IN COMBINATION WITH UNIONS! (i.e. sometimes the wrong type is used for the resulting table. This is because until now the types of the attributes of the first select statement have been used for the resulting table. When Intersects and/or Excepts are used in combination with Unions it might happen, that the first select statement of the original query appears at another position in the query which will be executed. The reason for this is the technique used for the implementation of Except/Intersect which does a query rewrite!) NOTE: It is NOT broken for pure UNION queries and pure INTERSECT/EXCEPT queries!!! -) I had to add the field intersect_clause to some data structures but did not find time to implement printfuncs for the new field. This does NOT break the debug modes but when an Except/Intersect is used the query debug output will be the already rewritten query. -) Massive changes to the grammar rules for SELECT and INSERT statements have been necessary (see comments in gram.y and documentation for deatails) in order to be able to use mixed queries like (SELECT ... UNION (SELECT ... EXCEPT SELECT)) INTERSECT SELECT...; -) When using UNION/EXCEPT/INTERSECT you will get: NOTICE: equal: "Don't know if nodes of type xxx are equal". I did not have time to add comparsion support for all the needed nodes, but the default behaviour of the function equal met my requirements. I did not dare to supress this message! That's the reason why the regression test for union will fail: These messages are also included in the union.out file! -) Somebody of you changed the union_planner() function for v6.4 (I copied the targetlist to new_tlist and that was removed and replaced by a cleanup of the original targetlist). These chnages violated some having queries executed against views so I changed it back again. I did not have time to examine the differences between the two versions but now it works :-) If you want to find out, try the file queries/view_having.sql on both versions and compare the results . Two queries won't produce a correct result with your version. regards Stefan
Add support for the CASE statement in the rewrite handling. Allows (at least some) rules and views. Still some trouble (crashes) with target CASE columns spanning tables, but lots now works.
Implement CASE expression.
The patch does 2 things: Fixes a bug in the rule system that caused a crashing backend when a join-view with calculated column is used in subselect. Modifies EXPLAIN to explain rewritten queries instead of the plain SeqScan on a view. Rules can produce very deep MORE Jan.
Fix for rules system from Jan.
Please apply the patch at the end. Disables use of system columns of views at all (not only oid, cmin etc. too). pgsql=> select cmin from pg_rules; ERROR: system column cmin not available - pg_rules is a view pgsql=> select * from pg_rules where pg_rules.oid = pg_class.oid; ERROR: system column oid not available - pg_rules is a view pgsql=> Jan
Here's a combination of all the patches I'm currently waiting for against a just updated CVS tree. It contains Partial new rewrite system that handles subselects, view aggregate columns, insert into select from view, updates with set col = view-value and select rules restriction to view definition. Updates for rule/view backparsing utility functions to handle subselects correct. New system views pg_tables and pg_indexes (where you can see the complete index definition in the latter one). Enabling array references on query parameters. Bugfix for functional index. Little changes to system views pg_rules and pg_views. The rule system isn't a release-stopper any longer. But another stopper is that I don't know if the latest changes to PL/pgSQL (not already in CVS) made it compile on AIX. Still wait for some response from Dave. Jan
OK, folks, here is the pgindent output.
This is the final state of the rule system for 6.4 after the patch is applied: Rewrite rules on relation level work fine now. Event qualifications on insert/update/delete rules work fine now. I added the new keyword OLD to reference the CURRENT tuple. CURRENT will be removed in 6.5. Update rules can reference NEW and OLD in the rule qualification and the actions. Insert/update/delete rules on views can be established to let them behave like real tables. For insert/update/delete rules multiple actions are supported now. The actions can also be surrounded by parantheses to make psql happy. Multiple actions are required if update to a view requires updates to multiple tables. Regular users are permitted to create/drop rules on tables they have RULE permissions for (DefineQueryRewrite() is now able to get around the access restrictions on pg_rewrite). This enables view creation for regular users too. This required an extra boolean parameter to pg_parse_and_plan() that tells to set skipAcl on all rangetable entries of the resulting queries. There is a new function pg_exec_query_acl_override() that could be used by backend utilities to use this facility. All rule actions (not only views) inherit the permissions of the event relations owner. Sample: User A creates tables T1 and T2, creates rules that log INSERT/UPDATE/DELETE on T1 in T2 (like in the regression tests for rules I created) and grants ALL but RULE on T1 to user B. User B can now fully access T1 and the logging happens in T2. But user B cannot access T2 at all, only the rule actions can. And due to missing RULE permissions on T1, user B cannot disable logging. Rules on the attribute level are disabled (they don't work properly and since regular users are now permitted to create rules I decided to disable them). Rules on select must have exactly one action that is a select (so select rules must be a view definition). UPDATE NEW/OLD rules are disabled (still broken, but triggers can do it). There are two new system views (pg_rule and pg_view) that show the definition of the rules or views so the db admin can see what the users do. They use two new functions pg_get_ruledef() and pg_get_viewdef() that are builtins. The functions pg_get_ruledef() and pg_get_viewdef() could be used to implement rule and view support in pg_dump. PostgreSQL is now the only database system I know, that has rewrite rules on the query level. All others (where I found a rule statement at all) use stored database procedures or the like (triggers as we call them) for active rules (as some call them). Future of the rule system: The now disabled parts of the rule system (attribute level, multiple actions on select and update new stuff) require a complete new rewrite handler from scratch. The old one is too badly wired up. After 6.4 I'll start to work on a new rewrite handler, that fully supports the attribute level rules, multiple actions on select and update new. This will be available for 6.5 so we get full rewrite rule capabilities. Jan
heap_fetch requires buffer pointer, must be released; heap_getnext no longer returns buffer pointer, can be gotten from scan; descriptor; bootstrap can create multi-key indexes; pg_procname index now is multi-key index; oidint2, oidint4, oidname are gone (must be removed from regression tests); use System Cache rather than sequential scan in many places; heap_modifytuple no longer takes buffer parameter; remove unused buffer parameter in a few other functions; oid8 is not index-able; remove some use of single-character variable names; cleanup Buffer variables usage and scan descriptor looping; cleaned up allocation and freeing of tuples; 18k lines of diff;
From: Jan Wieck <jwieck@debis.com> Hi, as proposed here comes the first patch for the query rewrite system. <for details, see archive dated Mon, 17 Aug 1998>
1) Queries using the having clause on base tables should work well now. Here some tested features, (examples included in the patch): 1.1) Subselects in the having clause 1.2) Double nested subselects 1.3) Subselects used in the where clause and in the having clause simultaneously 1.4) Union Selects using having 1.5) Indexes on the base relations are used correctly 1.6) Unallowed Queries are prevented (e.g. qualifications in the having clause that belong to the where clause) 1.7) Insert into as select 2) Queries using the having clause on view relations also work but there are some restrictions: 2.1) Create View as Select ... Having ...; using base tables in the select 2.1.1) The Query rewrite system: 2.1.2) Why are only simple queries allowed against a view from 2.1) ? 2.2) Select ... from testview1, testview2, ... having...; 3) Bug in ExecMergeJoin ?? Regards Stefan
Remove un-needed braces around single statements.
I started adding the Having Clause and it works quite fine for sequential scans! (I think it will also work with hash, index, etc but I did not check it out! I made some High level changes which should work for all access methods, but maybe I'm wrong. Please let me know.) Now it is possible to make queries like: select s.sname, max(p.pid), min(p.pid) from part p, supplier s where s.sid=p.sid group by s.sname having max(pid)=6 and min(pid)=1 or avg(pid)=4; Having does not work yet for queries that contain a subselect statement in the Having clause, I'll try to fix this in the next days. If there are some bugs, please let me know, I'll start to read the mailinglists now! Now here is the patch against the original 6.3 version (no snapshot!!): Stefan
pgindent run before 6.3 release, with Thomas' requested changes.
From: Jan Wieck <jwieck@debis.com> seems that my last post didn't make it through. That's good since the diff itself didn't covered the renaming of pg_user.h to pg_shadow.h and it's new content. Here it's again. The complete regression test passwd with only some float diffs. createuser and destroyuser work. pg_shadow cannot be read by ordinary user.
First step done, below is the patch to have views to override the permission checks for the accessed tables. Now we can do the following: CREATE VIEW db_user AS SELECT usename, usesysid, usecreatedb, usetrace, usecatupd, '**********'::text as passwd, valuntil FROM pg_user; REVOKE ALL ON pg_user FROM public; REVOKE ALL ON db_user FROM public; GRANT SELECT ON db_user TO public;
Make subqueries rewrite properly.
Yohoo UNIONS of VIEWS.
Goodbye ABORT. Hello ERROR for all errors.
Change elog(WARN) to elog(ERROR) and elog(ABORT).
Fix for count(*), aggs with views and multiple tables and sum(3).
Used modified version of indent that understands over 100 typedefs.
Another PGINDENT run that changes variable indenting and case label indenting. Also static variable indenting.
Massive commit to run PGINDENT on all *.c and *.h files.
Remove more (void) and fix -Wall warnings.
Fix access through null pointer info->rule_action. Thanks Darren King.
Postgres95 1.01 Distribution - Virgin Sources
Initial revision