In this section, we first indicate that large character sets with bounded or unbounded repetitions are the leading causes of state explosion. Then, we use several typical examples to illustrate how Offset-FA solves the state explosion problem while keeping semantic equivalence. We also discuss various specific pattern cases for implementation in Offset-FA. Lastly, we provide the formal definition of Offset-FA.
2.1. Main Idea
With the ever-increasing scale and complexity of pattern sets in practice, state explosion is an inevitable problem. State explosion originates from the ambiguity of metacharacters in regular expressions because massive states are needed to distinguish all possible input sequences. Generally speaking, there are two kinds of inflation: when compiling a single expression, and when compiling multiple expressions.
Yu [
34] analyzed the DFA state number for various pattern features. The large character set with counting repetitions in a single pattern may cause quadratic or even exponential state expansion. The DFA state number of pattern
“ˆA+[ˆ∖n]{4}BC”, is 22, a quadratic complexity with respect to the length constraint. Quadratic complexity comes from character class
‘[ˆ∖n]’ overlapping with its prefix
‘A+’. When the first character
‘A’ is matched, input
‘A’s can match either
‘A+’ or
‘[ˆ∖n]{4}’. To avoid a false negative, the DFA must record all the situations that would cause quadratic inflation. For another pattern,
“.*A.{4}BC”, a large character set with counting repetitions
‘.{4}’ also overlaps with its prefix
‘A’. Without starting anchor
‘ˆ’, prefix
‘A’ can appear at any position of the input sequence. Its DFA needs 56 states, which is exponential complexity with respect to the counting constraint.
In practice, a more popular situation for state explosion occurs when compiling multiple patterns into a single DFA, though none of these patterns had state inflation. This explosion is mainly caused by the
‘.*’ feature [
34,
37], where
‘.’ represents any character, and
‘*’ denotes repeating its previous character from zero to an infinite number of times. Thus,
‘.*’ can represent any input sequence. When compiling multiple patterns together, each
‘.*’ in a pattern generates duplications of states from all other patterns. DFA needs to record the prefixes of these
‘.*’s that were matched, and each combinational situation requires a state. Generally speaking, for
m patterns with one
‘.*’ in each pattern, we need
states to record the power set of these prefixes. In fact, a large character class with counting repetitions can cause a similar problem if the counting repetition is large enough.
In practical rule sets, large character sets with bounded or unbounded repetitions are the primary causes of state explosion. To prevent such state explosion, Offset-FA extracts these features from original patterns and compiles the remaining fragments to a DFA, namely, fragment-DFA. Then, a fragment relation table and a reset table are constructed to record the relationships among these fragments. In essence, Offset-FA uses tables rather than a vast number of states to represent these repetitions.
To clarify, we used a simple pattern set to explain how Offset-FA handles the state explosion problem while keeping semantic equivalence. Considering rules
GET[ˆ∖r∖n]*HTTP/,
mailmsg.*google,
stream.{5}ifenguc, and
Uptime[ˆ∖x7D]*Trojan∖x7D as the existing large character sets with unbounded or bounded repetitions, the corresponding DFA requires 996 states to describe these rules. In total, 48 states are enough in Offset-FA, as shown in
Figure 2.
Offset-FA comprises three parts: fragment-DFA, fragment relation table, and reset table. The extracted fragments are compiled to fragment-DFA, which is a DFA for fragment matching. After removing the large character sets with repetitions, the fragments are mainly simple strings. Thus, there is slight state expansion in fragment-DFA. Large character sets with repetitions are compiled to the fragment relation table (FRT) and the reset table (RT) for checking and keeping semantic equivalence. The fragment relation table records the length, previous fragment ID, offset range requirement, and corresponding rule ID for each fragment. Here, the matching of a fragment is checked as valid if its matched position minus the matched position of its previous fragment fulfils the offset range. In addition, we need a reset table to handle ‘[ˆ∖n]’-like large character set, which may invalidate the matching of its previous fragment.
Next, we used several input sequences to explain how Offset-FA works for matching. For each input sequence, the matching engine maintains a matching process table (MPT) to record the current matching status, recording the fragments and the matched positions that were matched. The MPT is checked and updated during the matching process.
For input
GETifengucHTTP/, the matching engine creates an empty MPT and starts from the beginning of the input. After reading
GET, State 3 is active in fragment-DFA, and fragment
“GET” is matched. Then, the matching engine needs to inquire the fragment relation table to obtain the information of the fragment
“GET”. As no previous fragment is required before
“GET”, the matching of
“GET” is confirmed, and the matching information is updated to the MPT. Then, for the following
ifenguc, State 34 is active, and fragment
“ifenguc” is matched. However, from FRT, we know that fragment
“ifenguc” relies on the matching of its previous fragment
“stream”. As fragment
“stream” has not yet appeared in the matching process table, the matching of fragment
“ifenguc” is a false positive and is not updated to the MPT. After reading the remaining sequence,
HTTP/, State 8 is active, and we need to inquire about information on Fragment 2. According to FRT, Fragment 2 relies on Fragment 1. As Fragment 1 was recorded in the MPT, we needed to further check whether the offset requirement was fulfilled. The matched positions of Fragments 1 and 2 were 3 and 15, respectively, and their position offset was 12. This fulfils the offset range [5); thus, the matching of Fragment 2 is valid, and its corresponding rule ID 1 is reported.
Figure 3 shows how the MPT is updated during matching.
Similarly, for input sequence
streamxxxxifenguc, Fragment 5 is marked as matched at Position 6 after the processing
stream. When the last character is processed, State 34 is active, and Fragment 6 is matched at Position 17. From the FRT, we know that Fragment 6 relied on Fragment 5, but their position offset was 11, not equal to the offset requirement 12. Thus, the matching of Fragment 6 is a false positive. For input
streamxxxxxifenguc, Fragments 5 and 6 are matched at Positions 6 and 18, respectively. Their position offset was equal to the offset range 12; thus, Fragment 6 was matched, and its corresponding rule ID 3 was reported. The matching processes for these two sequences are shown in
Figure 4.
In RE pattern
“mailmsg.*google”, fragment
‘mailmsg’ overlapped with fragment
‘google’ on character
‘g’. This is a typical situation that would cause false positives in traditional methods [
44], but Offset-FA could handle it correctly. For input
mailmsgoogle, Fragments 3 and 4 were matched at Positions 7 and 12, respectively. The position offset was 5, not belonging to offset range [6,), thus the matching of fragment 4 is a false positive. For input
mailmsggoogle, Fragments 3 and 4 are matched at Positions 7 and 13. Ttheir position offset 6 fulfills the range [6,). Thus the corresponding rule ID 2 is reported. For this kind of RE pattern, no false positives or false negatives occur in Offset-FA.
Figure 5 displays the corresponding matching process.
For large character set ‘.’, we only needed to consider the offset relationships among the corresponding fragments. However, for the ‘[ˆc]’-like character set, we had to also consider the situations in which excluded character c appears in the input sequence. For this kind of situation, another reset position field is required in the matching process table. The following examples show how these situations may cause potential false positives or false negatives and how Offset-FA handles these situations with the aid of the reset table.
For input
GETxx∖nxxHTTP/, Fragment 1 is matched at Position 3, and this matching status is updated to the MPT. Then, the MPT has no change until the matching engine reads character
∖n at Position 6. The reset table shows that character
∖n would cause the reset operation for Fragment 1. Thus, the matching information of Fragment 1 is reset in MPT, as shown in
Figure 6. Here, −1 denotes that the fragment was matched before but reset later. The reset position value is also updated to the MPT and is used to avoid false positives and false negatives. When the last character was processed in fragment-DFA, State 8 was active. Then, we needed to check the corresponding requirements for Fragment 2. Its previous fragment was reset at Position 6. From the matched position (13) and length (5) of Fragment 2, we know that the reset position was actually before Fragment 2. This means that the reset operation was valid, and the matching of Fragment 2 was a false positive and not recorded.
The situation becomes more complex if the excluded character appears in the following fragment in the RE pattern [
44]. For example, in pattern “
Uptime[ˆ∖x7D]*Trojan∖x7D”, excluded character
‘∖x7D’ appears in fragment
‘Trojan∖x7D’. The matching engine must be able to distinguish between different reset situations. For input
UptimexxxTrojan∖x7D, Fragment 7 is first matched at Position 6. Then, after processing
xxxTrojan∖x7D, State 47 was active in Fragment-DFA. For the last character,
∖x7D, the engine reset the corresponding Fragment 7, and the reset position was set as 16. Then, for matched Fragment 8, the engine checked that Fragment 7 is reset at Position 16. According to the matched position (16) and length (7) of Fragment 8, the reset position was inside Fragment 8, which means that the reset was caused by the matching of Fragment 8. Thus, this reset is an invalid operation, and the matched information of Fragment 8 was updated to the MPT. Further, corresponding rule ID 4 was reported. The matching process is illustrated in
Figure 7a.
For another input,
Uptimexx∖x7DTrojan∖x7D, excluded character
∖x7D appeared twice. To avoid false positives, the matching engine should only execute and record the first reset operation, as shown in
Figure 7b. Similarly, Fragment 7 was set as matched at Position 6 and reset at Position 9. Then, when processing the last character,
∖x7D, Fragment 8 was matched. As the character
∖x7D appeared in the reset table, the matching engine had to also check the corresponding Fragment 7 in the matching process table. Fragment 7 was reset before, and the engine only recorded the first reset position. Thus, there was no change in the MPT. According to similar procedures, the reset operation of Fragment 7 is before Fragment 8. Thus, it was a valid reset operation; the matching of Fragment 8 was invalid and not recorded in the MPT.
These RE patterns and input examples are typical situations for a large character set with unbounded or bounded repetitions. Next, we discuss various general cases about large character sets with repetitions.
2.2. Solutions for Various General Patterns
The main idea of Offset-FA is to divide the patterns at the features of large character sets with repetitions (
LCSR) and reassemble them with the fragment relation table and reset table. An RE pattern can be expressed as in (
1). Here,
LCSRs can be regarded as the split points, and
frags denote the substrings (or fragments) except
LCSRs.
As long as these fragments can be matched and confirmed sequentially, the RE pattern is checked when the last fragment is matched and confirmed. Hence, the matching problem can be reduced to matching a series of .
Table 2 displays various
LCSR features and corresponding configurations in the fragment relation table. For each
, the fragment length field, previous fragment field, and rule ID field for each fragment are clear. Thus, we only discuss the offset range field for each
, as listed in the second column. Here,
denotes the length of the fragment.
In this table, the upper part concludes unbounded LCSR features, especially typical ‘.*’ and ‘[ˆc]*’ features. These two kinds of features are the most common in practice. For ‘.*’ and ‘.+’ features, and being matched sequentially was not sufficient to demonstrate was matched. The matching engine must rely on the offset information to omit false positives when overlaps (e.g., pattern “ab.*bc”). For ‘[ˆc]*’ and ‘[ˆc]+’, except for the above false positives, we also need to consider situations in which the excluded character c appears in (e.g., pattern “ab[ˆc]*cd”), which may cause false positives or false negatives. The matching engine cannot handle such complex features with only matched positions. With the assistance of reset information from the reset table and matching process table, the engine can accurately judge whether the reset of and the matching of are valid.
The lower part of
Table 2 describes bounded
LCSR features, among which ‘.{n}’ and ‘[ˆc]{n}’ are the most common in practice. It is relatively easy to handle ‘.{n}’ features because the engine only needs to compute the offset distance between the matched positions of
and
. Other features, such as ‘.{m,n}’, ‘.{,m}’ and ‘.{n,}’, are similar to ‘.{n}’. For ‘[ˆc]{n}’ features, it is also necessary to consider false positives/negatives caused by reset operations.
Here, we only provide basic configurations for various general
LCSR features. The detailed generation algorithm for Offset-FA and the matching algorithm are discussed in
Section 3. Next, we first introduce the formal model of Offset-FA.