Update meaning of {,5} etc to match update in Perl 5.34.0; refactor quantifier parsing

Author: PhilipHazel

ChangeLog

@@ -109,6 +109,10 @@ the code for extending the heap frames vector. This fixes GitHub issue #275.
 28. Update pcre2_fuzzsupport.c to avoid clang sanitize complaint about shifting 
 left by 16 when there are non-zeros in the top 16 bits.
 
+29. Perl 5.34.0 changed the meaning of (for example) {,3} which did not used to 
+be treated as a quantifier. Now it is interpreted as {0,3} and PCRE2 has 
+changed to match. Note that {,} is still not a quantifier.
+
 
 Version 10.42 11-December-2022
 ------------------------------

doc/html/pcre2compat.html

@@ -46,13 +46,18 @@ <h1>pcre2compat man page</h1>
 any kind of quantifier on non-lookaround assertions.
 </P>
 <P>
-4. Capture groups that occur inside negative lookaround assertions are counted,
+4. If a quantifier appears where there is nothing to repeat (for example, at
+the start of a branch), PCRE2 raises an error whereas Perl treats the
+quantifier characters as literal.
+</P>
+<P>
+5. Capture groups that occur inside negative lookaround assertions are counted,
 but their entries in the offsets vector are set only when a negative assertion
 is a condition that has a matching branch (that is, the condition is false).
 Perl may set such capture groups in other circumstances.
 </P>
 <P>
-5. The following Perl escape sequences are not supported: \F, \l, \L, \u,
+6. The following Perl escape sequences are not supported: \F, \l, \L, \u,
 \U, and \N when followed by a character name. \N on its own, matching a
 non-newline character, and \N{U+dd..}, matching a Unicode code point, are
 supported. The escapes that modify the case of following letters are
@@ -63,7 +68,7 @@ <h1>pcre2compat man page</h1>
 interprets them.
 </P>
 <P>
-6. The Perl escape sequences \p, \P, and \X are supported only if PCRE2 is
+7. The Perl escape sequences \p, \P, and \X are supported only if PCRE2 is
 built with Unicode support (the default). The properties that can be tested
 with \p and \P are limited to the general category properties such as Lu and
 Nd, script names such as Greek or Han, Bidi_Class, Bidi_Control, and the
@@ -75,7 +80,7 @@ <h1>pcre2compat man page</h1>
 to prefix any of these properties with "Is".
 </P>
 <P>
-7. PCRE2 supports the \Q...\E escape for quoting substrings. Characters
+8. PCRE2 supports the \Q...\E escape for quoting substrings. Characters
 in between are treated as literals. However, this is slightly different from
 Perl in that $ and @ are also handled as literals inside the quotes. In Perl,
 they cause variable interpolation (PCRE2 does not have variables). Also, Perl
@@ -96,19 +101,19 @@ <h1>pcre2compat man page</h1>
 by both PCRE2 and Perl.
 </P>
 <P>
-8. Fairly obviously, PCRE2 does not support the (?{code}) and (??{code})
+9. Fairly obviously, PCRE2 does not support the (?{code}) and (??{code})
 constructions. However, PCRE2 does have a "callout" feature, which allows an
 external function to be called during pattern matching. See the
 <a href="pcre2callout.html"><b>pcre2callout</b></a>
 documentation for details.
 </P>
 <P>
-9. Subroutine calls (whether recursive or not) were treated as atomic groups up
+10. Subroutine calls (whether recursive or not) were treated as atomic groups up
 to PCRE2 release 10.23, but from release 10.30 this changed, and backtracking
 into subroutine calls is now supported, as in Perl.
 </P>
 <P>
-10. In PCRE2, if any of the backtracking control verbs are used in a group that
+11. In PCRE2, if any of the backtracking control verbs are used in a group that
 is called as a subroutine (whether or not recursively), their effect is
 confined to that group; it does not extend to the surrounding pattern. This is
 not always the case in Perl. In particular, if (*THEN) is present in a group
@@ -117,20 +122,20 @@ <h1>pcre2compat man page</h1>
 processed as anchored at the point where they are tested.
 </P>
 <P>
-11. If a pattern contains more than one backtracking control verb, the first
+12. If a pattern contains more than one backtracking control verb, the first
 one that is backtracked onto acts. For example, in the pattern
 A(*COMMIT)B(*PRUNE)C a failure in B triggers (*COMMIT), but a failure in C
 triggers (*PRUNE). Perl's behaviour is more complex; in many cases it is the
 same as PCRE2, but there are cases where it differs.
 </P>
 <P>
-12. There are some differences that are concerned with the settings of captured
+13. There are some differences that are concerned with the settings of captured
 strings when part of a pattern is repeated. For example, matching "aba" against
 the pattern /^(a(b)?)+$/ in Perl leaves $2 unset, but in PCRE2 it is set to
 "b".
 </P>
 <P>
-13. PCRE2's handling of duplicate capture group numbers and names is not as
+14. PCRE2's handling of duplicate capture group numbers and names is not as
 general as Perl's. This is a consequence of the fact the PCRE2 works internally
 just with numbers, using an external table to translate between numbers and
 names. In particular, a pattern such as (?|(?&#60;a&#62;A)|(?&#60;b&#62;B)), where the two
@@ -140,34 +145,34 @@ <h1>pcre2compat man page</h1>
 number 1. To avoid this confusing situation, an error is given at compile time.
 </P>
 <P>
-14. Perl used to recognize comments in some places that PCRE2 does not, for
+15. Perl used to recognize comments in some places that PCRE2 does not, for
 example, between the ( and ? at the start of a group. If the /x modifier is
 set, Perl allowed white space between ( and ? though the latest Perls give an
 error (for a while it was just deprecated). There may still be some cases where
 Perl behaves differently.
 </P>
 <P>
-15. Perl, when in warning mode, gives warnings for character classes such as
+16. Perl, when in warning mode, gives warnings for character classes such as
 [A-\d] or [a-[:digit:]]. It then treats the hyphens as literals. PCRE2 has no
 warning features, so it gives an error in these cases because they are almost
 certainly user mistakes.
 </P>
 <P>
-16. In PCRE2, the upper/lower case character properties Lu and Ll are not
+17. In PCRE2, the upper/lower case character properties Lu and Ll are not
 affected when case-independent matching is specified. For example, \p{Lu}
 always matches an upper case letter. I think Perl has changed in this respect;
 in the release at the time of writing (5.34), \p{Lu} and \p{Ll} match all
 letters, regardless of case, when case independence is specified.
 </P>
 <P>
-17. From release 5.32.0, Perl locks out the use of \K in lookaround
+18. From release 5.32.0, Perl locks out the use of \K in lookaround
 assertions. From release 10.38 PCRE2 does the same by default. However, there
 is an option for re-enabling the previous behaviour. When this option is set,
 \K is acted on when it occurs in positive assertions, but is ignored in
 negative assertions.
 </P>
 <P>
-18. PCRE2 provides some extensions to the Perl regular expression facilities.
+19. PCRE2 provides some extensions to the Perl regular expression facilities.
 Perl 5.10 included new features that were not in earlier versions of Perl, some
 of which (such as named parentheses) were in PCRE2 for some time before. This
 list is with respect to Perl 5.34:
@@ -219,7 +224,7 @@ <h1>pcre2compat man page</h1>
 lookarounds are atomic.
 </P>
 <P>
-19. Perl has different limits than PCRE2. See the
+20. Perl has different limits than PCRE2. See the
 <a href="pcre2limit.html"><b>pcre2limit</b></a>
 documentation for details. Perl went with 5.10 from recursion to iteration
 keeping the intermediate matches on the heap, which is ~10% slower but does not
@@ -241,7 +246,7 @@ <h1>pcre2compat man page</h1>
 REVISION
 </b><br>
 <P>
-Last updated: 11 August 2023
+Last updated: 13 September 2023
 <br>
 Copyright &copy; 1997-2023 University of Cambridge.
 <br>

doc/html/pcre2pattern.html

@@ -323,7 +323,7 @@ <h1>pcre2pattern man page</h1>
   *      0 or more quantifier
   +      1 or more quantifier; also "possessive quantifier"
   ?      0 or 1 quantifier; also quantifier minimizer
-  {      start min/max quantifier
+  {      potential start of min/max quantifier
 </pre>
 Part of a pattern that is in square brackets is called a "character class". In
 a character class the only metacharacters are:
@@ -1914,24 +1914,25 @@ <h1>pcre2pattern man page</h1>
 </P>
 <br><a name="SEC17" href="#TOC1">REPETITION</a><br>
 <P>
-Repetition is specified by quantifiers, which can follow any of the following
+Repetition is specified by quantifiers, which may follow any one of these
 items:
 <pre>
   a literal data character
   the dot metacharacter
   the \C escape sequence
   the \R escape sequence
   the \X escape sequence
-  an escape such as \d or \pL that matches a single character
+  any escape sequence that matches a single character
   a character class
   a backreference
   a parenthesized group (including lookaround assertions)
   a subroutine call (recursive or otherwise)
 </pre>
-The general repetition quantifier specifies a minimum and maximum number of
-permitted matches, by giving the two numbers in curly brackets (braces),
-separated by a comma. The numbers must be less than 65536, and the first must
-be less than or equal to the second. For example,
+If a quantifier does not follow a repeatabke item, an error occurs. The
+general repetition quantifier specifies a minimum and maximum number of
+permitted matches by giving two numbers in curly brackets (braces), separated
+by a comma. The numbers must be less than 65536, and the first must be less
+than or equal to the second. For example,
 <pre>
   z{2,4}
 </pre>
@@ -1946,10 +1947,20 @@ <h1>pcre2pattern man page</h1>
 <pre>
   \d{8}
 </pre>
-matches exactly 8 digits. An opening curly bracket that appears in a position
-where a quantifier is not allowed, or one that does not match the syntax of a
-quantifier, is taken as a literal character. For example, {,6} is not a
-quantifier, but a literal string of four characters.
+matches exactly 8 digits. If the first number is omitted, the lower limit is
+taken as zero; in this case the upper limit must be present.
+<pre>
+  X{,4} is interpreted as X{0,4}
+</pre>
+This is a change in behaviour that happened in Perl 5.34.0 and PCRE2 10.43. In
+earlier versions such a sequence was not interpreted as a quantifier. If the
+characters that follow an opening brace do not match the syntax of a
+quantifier, the brace is taken as a literal character. In particular, this
+means that {,} is a literal string of three characters.
+</P>
+<P>
+Note that not every opening brace is potentially the start of a quantifier
+because braces are used in other items such as \N{U+345} or \k{name}.
 </P>
 <P>
 In UTF modes, quantifiers apply to characters rather than to individual code
@@ -1990,11 +2001,11 @@ <h1>pcre2pattern man page</h1>
 </P>
 <P>
 By default, quantifiers are "greedy", that is, they match as much as possible
-(up to the maximum number of permitted times), without causing the rest of the
-pattern to fail. The classic example of where this gives problems is in trying
-to match comments in C programs. These appear between /* and */ and within the
-comment, individual * and / characters may appear. An attempt to match C
-comments by applying the pattern
+(up to the maximum number of permitted repetitions), without causing the rest
+of the pattern to fail. The classic example of where this gives problems is in
+trying to match comments in C programs. These appear between /* and */ and
+within the comment, individual * and / characters may appear. An attempt to
+match C comments by applying the pattern
 <pre>
   /\*.*\*/
 </pre>
@@ -2009,10 +2020,10 @@ <h1>pcre2pattern man page</h1>
 <pre>
   /\*.*?\*/
 </pre>
-does the right thing with the C comments. The meaning of the various
-quantifiers is not otherwise changed, just the preferred number of matches.
-Do not confuse this use of question mark with its use as a quantifier in its
-own right. Because it has two uses, it can sometimes appear doubled, as in
+does the right thing with C comments. The meaning of the various quantifiers is
+not otherwise changed, just the preferred number of matches. Do not confuse
+this use of question mark with its use as a quantifier in its own right.
+Because it has two uses, it can sometimes appear doubled, as in
 <pre>
   \d??\d
 </pre>
@@ -3781,7 +3792,7 @@ <h1>pcre2pattern man page</h1>
 </P>
 <br><a name="SEC32" href="#TOC1">REVISION</a><br>
 <P>
-Last updated: 11 August 2023
+Last updated: 13 September 2023
 <br>
 Copyright &copy; 1997-2023 University of Cambridge.
 <br>

doc/html/pcre2syntax.html

@@ -305,6 +305,9 @@ <h1>pcre2syntax man page</h1>
   {n,}        n or more, greedy
   {n,}+       n or more, possessive
   {n,}?       n or more, lazy
+  {,m}        zero up to m, greedy
+  {,m}+       zero up to m, possessive
+  {,m}?       zero up to m, lazy
 </PRE>
 </P>
 <br><a name="SEC12" href="#TOC1">ANCHORS AND SIMPLE ASSERTIONS</a><br>
@@ -604,7 +607,7 @@ <h1>pcre2syntax man page</h1>
 </P>
 <br><a name="SEC31" href="#TOC1">REVISION</a><br>
 <P>
-Last updated: 11 August 2023
+Last updated: 13 September 2023
 <br>
 Copyright &copy; 1997-2023 University of Cambridge.
 <br>

doc/pcre2-config.txt

@@ -1,3 +1,4 @@
+
 PCRE2-CONFIG(1)             General Commands Manual            PCRE2-CONFIG(1)