Implement lookbehind constraints in our regular-expression engine.

A lookbehind constraint is like a lookahead constraint in that it consumes
no text; but it checks for existence (or nonexistence) of a match *ending*
at the current point in the string, rather than one *starting* at the
current point.  This is a long-requested feature since it exists in many
other regex libraries, but Henry Spencer had never got around to
implementing it in the code we use.

Just making it work is actually pretty trivial; but naive copying of the
logic for lookahead constraints leads to code that often spends O(N^2) time
to scan an N-character string, because we have to run the match engine
from string start to the current probe point each time the constraint is
checked.  In typical use-cases a lookbehind constraint will be written at
the start of the regex and hence will need to be checked at every character
--- so O(N^2) work overall.  To fix that, I introduced a third copy of the
core DFA matching loop, paralleling the existing longest() and shortest()
loops.  This version, matchuntil(), can suspend and resume matching given
a couple of pointers' worth of storage space.  So we need only run it
across the string once, stopping at each interesting probe point and then
resuming to advance to the next one.

I also put in an optimization that simplifies one-character lookahead and
lookbehind constraints, such as "(?=x)" or "(?<!\w)", into AHEAD and BEHIND
constraints, which already existed in the engine.  This avoids the overhead
of the LACON machinery entirely for these rather common cases.

The net result is that lookbehind constraints run a factor of three or so
slower than Perl's for multi-character constraints, but faster than Perl's
for one-character constraints ... and they work fine for variable-length
constraints, which Perl gives up on entirely.  So that's not bad from a
competitive perspective, and there's room for further optimization if
anyone cares.  (In reality, raw scan rate across a large input string is
probably not that big a deal for Postgres usage anyway; so I'm happy if
it's linear.)

Author: tglsfdc

doc/src/sgml/func.sgml

@@ -4477,13 +4477,27 @@ SELECT foo FROM regexp_split_to_table('the quick brown fox', E'\\s*') AS foo;
        where no substring matching <replaceable>re</> begins
        (AREs only) </entry>
        </row>
+
+       <row>
+       <entry> <literal>(?&lt;=</><replaceable>re</><literal>)</> </entry>
+       <entry> <firstterm>positive lookbehind</> matches at any point
+       where a substring matching <replaceable>re</> ends
+       (AREs only) </entry>
+       </row>
+
+       <row>
+       <entry> <literal>(?&lt;!</><replaceable>re</><literal>)</> </entry>
+       <entry> <firstterm>negative lookbehind</> matches at any point
+       where no substring matching <replaceable>re</> ends
+       (AREs only) </entry>
+       </row>
       </tbody>
      </tgroup>
     </table>
 
    <para>
-    Lookahead constraints cannot contain <firstterm>back references</>
-    (see <xref linkend="posix-escape-sequences">),
+    Lookahead and lookbehind constraints cannot contain <firstterm>back
+    references</> (see <xref linkend="posix-escape-sequences">),
     and all parentheses within them are considered non-capturing.
    </para>
    </sect3>
@@ -5355,7 +5369,7 @@ SELECT regexp_matches('abc01234xyz', '(?:(.*?)(\d+)(.*)){1,1}');
     the lack of special treatment for a trailing newline,
     the addition of complemented bracket expressions to the things
     affected by newline-sensitive matching,
-    the restrictions on parentheses and back references in lookahead
+    the restrictions on parentheses and back references in lookahead/lookbehind
     constraints, and the longest/shortest-match (rather than first-match)
     matching semantics.
    </para>

src/backend/regex/README

@@ -332,10 +332,10 @@ The possible arc types are:
     as "$0->to_state" or "$1->to_state" for end-of-string and end-of-line
     constraints respectively.
 
-    LACON constraints, which represent "(?=re)" and "(?!re)" constraints,
-    i.e. the input starting at this point must match (or not match) a
-    given sub-RE, but the matching input is not consumed.  These are
-    dumped as ":subtree_number:->to_state".
+    LACON constraints, which represent "(?=re)", "(?!re)", "(?<=re)", and
+    "(?<!re)" constraints, i.e. the input starting/ending at this point must
+    match (or not match) a given sub-RE, but the matching input is not
+    consumed.  These are dumped as ":subtree_number:->to_state".
 
 If you see anything else (especially any question marks) in the display of
 an arc, it's dumpnfa() trying to tell you that there's something fishy

src/backend/regex/re_syntax.n

@@ -196,10 +196,18 @@ where a substring matching \fIre\fR begins
 \fB(?!\fIre\fB)\fR
 \fInegative lookahead\fR (AREs only), matches at any point
 where no substring matching \fIre\fR begins
+.TP
+\fB(?<=\fIre\fB)\fR
+\fIpositive lookbehind\fR (AREs only), matches at any point
+where a substring matching \fIre\fR ends
+.TP
+\fB(?<!\fIre\fB)\fR
+\fInegative lookbehind\fR (AREs only), matches at any point
+where no substring matching \fIre\fR ends
 .RE
 .PP
-The lookahead constraints may not contain back references (see later),
-and all parentheses within them are considered non-capturing.
+Lookahead and lookbehind constraints may not contain back references
+(see later), and all parentheses within them are considered non-capturing.
 .PP
 An RE may not end with `\fB\e\fR'.
 
@@ -856,7 +864,8 @@ Incompatibilities of note include `\fB\eb\fR', `\fB\eB\fR',
 the lack of special treatment for a trailing newline,
 the addition of complemented bracket expressions to the things
 affected by newline-sensitive matching,
-the restrictions on parentheses and back references in lookahead constraints,
+the restrictions on parentheses and back references in lookahead/lookbehind
+constraints,
 and the longest/shortest-match (rather than first-match) matching semantics.
 .PP
 The matching rules for REs containing both normal and non-greedy quantifiers

src/backend/regex/regc_lex.c

@@ -582,6 +582,8 @@ next(struct vars * v)
 			{
 				NOTE(REG_UNONPOSIX);
 				v->now++;
+				if (ATEOS())
+					FAILW(REG_BADRPT);
 				switch (*v->now++)
 				{
 					case CHR(':'):		/* non-capturing paren */
@@ -596,12 +598,31 @@ next(struct vars * v)
 						return next(v);
 						break;
 					case CHR('='):		/* positive lookahead */
-						NOTE(REG_ULOOKAHEAD);
-						RETV(LACON, 1);
+						NOTE(REG_ULOOKAROUND);
+						RETV(LACON, LATYPE_AHEAD_POS);
 						break;
 					case CHR('!'):		/* negative lookahead */
-						NOTE(REG_ULOOKAHEAD);
-						RETV(LACON, 0);
+						NOTE(REG_ULOOKAROUND);
+						RETV(LACON, LATYPE_AHEAD_NEG);
+						break;
+					case CHR('<'):
+						if (ATEOS())
+							FAILW(REG_BADRPT);
+						switch (*v->now++)
+						{
+							case CHR('='):		/* positive lookbehind */
+								NOTE(REG_ULOOKAROUND);
+								RETV(LACON, LATYPE_BEHIND_POS);
+								break;
+							case CHR('!'):		/* negative lookbehind */
+								NOTE(REG_ULOOKAROUND);
+								RETV(LACON, LATYPE_BEHIND_NEG);
+								break;
+							default:
+								FAILW(REG_BADRPT);
+								break;
+						}
+						assert(NOTREACHED);
 						break;
 					default:
 						FAILW(REG_BADRPT);

src/backend/regex/regc_nfa.c

@@ -1348,6 +1348,49 @@ cleartraverse(struct nfa * nfa,
 		cleartraverse(nfa, a->to);
 }
 
+/*
+ * single_color_transition - does getting from s1 to s2 cross one PLAIN arc?
+ *
+ * If traversing from s1 to s2 requires a single PLAIN match (possibly of any
+ * of a set of colors), return a state whose outarc list contains only PLAIN
+ * arcs of those color(s).  Otherwise return NULL.
+ *
+ * This is used before optimizing the NFA, so there may be EMPTY arcs, which
+ * we should ignore; the possibility of an EMPTY is why the result state could
+ * be different from s1.
+ *
+ * It's worth troubling to handle multiple parallel PLAIN arcs here because a
+ * bracket construct such as [abc] might yield either one or several parallel
+ * PLAIN arcs depending on earlier atoms in the expression.  We'd rather that
+ * that implementation detail not create user-visible performance differences.
+ */
+static struct state *
+single_color_transition(struct state * s1, struct state * s2)
+{
+	struct arc *a;
+
+	/* Ignore leading EMPTY arc, if any */
+	if (s1->nouts == 1 && s1->outs->type == EMPTY)
+		s1 = s1->outs->to;
+	/* Likewise for any trailing EMPTY arc */
+	if (s2->nins == 1 && s2->ins->type == EMPTY)
+		s2 = s2->ins->from;
+	/* Perhaps we could have a single-state loop in between, if so reject */
+	if (s1 == s2)
+		return NULL;
+	/* s1 must have at least one outarc... */
+	if (s1->outs == NULL)
+		return NULL;
+	/* ... and they must all be PLAIN arcs to s2 */
+	for (a = s1->outs; a != NULL; a = a->outchain)
+	{
+		if (a->type != PLAIN || a->to != s2)
+			return NULL;
+	}
+	/* OK, return s1 as the possessor of the relevant outarcs */
+	return s1;
+}
+
 /*
  * specialcolors - fill in special colors for an NFA
  */