Add pcre2_code_copy_with_tables().

Author: PhilipHazel

ChangeLog

@@ -181,6 +181,9 @@ wrong name.
 
 27. In pcre2test, give some offset information for errors in hex patterns.
 
+28. Implemented pcre2_code_copy_with_tables(), and added pushtablescopy to
+pcre2test for testing it.
+
 
 Version 10.22 29-July-2016
 --------------------------
@@ -250,7 +253,7 @@ a report of compiler warnings from Visual Studio 2013 and a few tests with
 gcc's -Wconversion (which still throws up a lot).
 
 15. Implemented pcre2_code_copy(), and added pushcopy and #popcopy to pcre2test
-for testing it.
+for testing it. 
 
 16. Change 66 for 10.21 introduced the use of snprintf() in PCRE2's version of
 regerror(). When the error buffer is too small, my version of snprintf() puts a

Makefile.am

@@ -25,6 +25,7 @@ dist_html_DATA = \
   doc/html/pcre2.html \
   doc/html/pcre2_callout_enumerate.html \
   doc/html/pcre2_code_copy.html \
+  doc/html/pcre2_code_copy_with_tables.html \
   doc/html/pcre2_code_free.html \
   doc/html/pcre2_compile.html \
   doc/html/pcre2_compile_context_copy.html \
@@ -107,6 +108,7 @@ dist_man_MANS = \
   doc/pcre2.3 \
   doc/pcre2_callout_enumerate.3 \
   doc/pcre2_code_copy.3 \
+  doc/pcre2_code_copy_with_tables.3 \
   doc/pcre2_code_free.3 \
   doc/pcre2_compile.3 \
   doc/pcre2_compile_context_copy.3 \

doc/html/NON-AUTOTOOLS-BUILD.txt

@@ -174,7 +174,11 @@ can skip ahead to the CMake section.
 
 (11) If you want to use the pcre2grep command, compile and link
      src/pcre2grep.c; it uses only the basic 8-bit PCRE2 library (it does not
-     need the pcre2posix library).
+     need the pcre2posix library). If you have built the PCRE2 library with JIT
+     support by defining SUPPORT_JIT in src/config.h, you can also define
+     SUPPORT_PCRE2GREP_JIT, which causes pcre2grep to make use of JIT (unless
+     it is run with --no-jit). If you define SUPPORT_PCRE2GREP_JIT without
+     defining SUPPORT_JIT, pcre2grep does not try to make use of JIT.
 
 
 STACK SIZE IN WINDOWS ENVIRONMENTS
@@ -389,4 +393,4 @@ and executable, is in EBCDIC and native z/OS file formats and this is the
 recommended download site.
 
 =============================
-Last Updated: 16 July 2015
+Last Updated: 13 October 2016

doc/html/README.txt

@@ -44,7 +44,7 @@ wrappers.
 
 The distribution does contain a set of C wrapper functions for the 8-bit
 library that are based on the POSIX regular expression API (see the pcre2posix
-man page). These can be found in a library called libpcre2posix. Note that this
+man page). These can be found in a library called libpcre2-posix. Note that this
 just provides a POSIX calling interface to PCRE2; the regular expressions
 themselves still follow Perl syntax and semantics. The POSIX API is restricted,
 and does not give full access to all of PCRE2's facilities.
@@ -58,8 +58,8 @@ renamed or pointed at by a link.
 If you are using the POSIX interface to PCRE2 and there is already a POSIX
 regex library installed on your system, as well as worrying about the regex.h
 header file (as mentioned above), you must also take care when linking programs
-to ensure that they link with PCRE2's libpcre2posix library. Otherwise they may
-pick up the POSIX functions of the same name from the other library.
+to ensure that they link with PCRE2's libpcre2-posix library. Otherwise they
+may pick up the POSIX functions of the same name from the other library.
 
 One way of avoiding this confusion is to compile PCRE2 with the addition of
 -Dregcomp=PCRE2regcomp (and similarly for the other POSIX functions) to the
@@ -204,13 +204,6 @@ library. They are also documented in the pcre2build man page.
   --enable-newline-is-crlf, --enable-newline-is-anycrlf, or
   --enable-newline-is-any to the "configure" command, respectively.
 
-  If you specify --enable-newline-is-cr or --enable-newline-is-crlf, some of
-  the standard tests will fail, because the lines in the test files end with
-  LF. Even if the files are edited to change the line endings, there are likely
-  to be some failures. With --enable-newline-is-anycrlf or
-  --enable-newline-is-any, many tests should succeed, but there may be some
-  failures.
-
 . By default, the sequence \R in a pattern matches any Unicode line ending
   sequence. This is independent of the option specifying what PCRE2 considers
   to be the end of a line (see above). However, the caller of PCRE2 can
@@ -253,13 +246,13 @@ library. They are also documented in the pcre2build man page.
   sizes in the pcre2stack man page.
 
 . In the 8-bit library, the default maximum compiled pattern size is around
-  64K. You can increase this by adding --with-link-size=3 to the "configure"
-  command. PCRE2 then uses three bytes instead of two for offsets to different
-  parts of the compiled pattern. In the 16-bit library, --with-link-size=3 is
-  the same as --with-link-size=4, which (in both libraries) uses four-byte
-  offsets. Increasing the internal link size reduces performance in the 8-bit
-  and 16-bit libraries. In the 32-bit library, the link size setting is
-  ignored, as 4-byte offsets are always used.
+  64K bytes. You can increase this by adding --with-link-size=3 to the
+  "configure" command. PCRE2 then uses three bytes instead of two for offsets
+  to different parts of the compiled pattern. In the 16-bit library,
+  --with-link-size=3 is the same as --with-link-size=4, which (in both
+  libraries) uses four-byte offsets. Increasing the internal link size reduces
+  performance in the 8-bit and 16-bit libraries. In the 32-bit library, the
+  link size setting is ignored, as 4-byte offsets are always used.
 
 . You can build PCRE2 so that its internal match() function that is called from
   pcre2_match() does not call itself recursively. Instead, it uses memory
@@ -339,12 +332,23 @@ library. They are also documented in the pcre2build man page.
 
   Of course, the relevant libraries must be installed on your system.
 
-. The default size (in bytes) of the internal buffer used by pcre2grep can be
-  set by, for example:
+. The default starting size (in bytes) of the internal buffer used by pcre2grep
+  can be set by, for example:
 
   --with-pcre2grep-bufsize=51200
 
-  The value must be a plain integer. The default is 20480.
+  The value must be a plain integer. The default is 20480. The amount of memory 
+  used by pcre2grep is actually three times this number, to allow for "before" 
+  and "after" lines. If very long lines are encountered, the buffer is 
+  automatically enlarged, up to a fixed maximum size.
+  
+. The default maximum size of pcre2grep's internal buffer can be set by, for 
+  example:
+  
+  --with-pcre2grep-max-bufsize=2097152
+  
+  The default is either 1048576 or the value of --with-pcre2grep-bufsize, 
+  whichever is the larger.  
 
 . It is possible to compile pcre2test so that it links with the libreadline
   or libedit libraries, by specifying, respectively,
@@ -368,6 +372,22 @@ library. They are also documented in the pcre2build man page.
   If you get error messages about missing functions tgetstr, tgetent, tputs,
   tgetflag, or tgoto, this is the problem, and linking with the ncurses library
   should fix it.
+  
+. There is a special option called --enable-fuzz-support for use by people who 
+  want to run fuzzing tests on PCRE2. At present this applies only to the 8-bit 
+  library. If set, it causes an extra library called libpcre2-fuzzsupport.a to 
+  be built, but not installed. This contains a single function called
+  LLVMFuzzerTestOneInput() whose arguments are a pointer to a string and the 
+  length of the string. When called, this function tries to compile the string 
+  as a pattern, and if that succeeds, to match it. This is done both with no 
+  options and with some random options bits that are generated from the string.
+  Setting --enable-fuzz-support also causes a binary called pcre2fuzzcheck to
+  be created. This is normally run under valgrind or used when PCRE2 is
+  compiled with address sanitizing enabled. It calls the fuzzing function and
+  outputs information about it is doing. The input strings are specified by 
+  arguments: if an argument starts with "=" the rest of it is a literal input 
+  string. Otherwise, it is assumed to be a file name, and the contents of the 
+  file are the test string.
 
 The "configure" script builds the following files for the basic C library:
 
@@ -543,7 +563,7 @@ script creates the .txt and HTML forms of the documentation from the man pages.
 
 
 Testing PCRE2
-------------
+-------------
 
 To test the basic PCRE2 library on a Unix-like system, run the RunTest script.
 There is another script called RunGrepTest that tests the pcre2grep command.
@@ -757,6 +777,7 @@ The distribution should contain the files listed below.
   src/pcre2_xclass.c       )
 
   src/pcre2_printint.c     debugging function that is used by pcre2test,
+  src/pcre2_fuzzsupport.c  function for (optional) fuzzing support 
 
   src/config.h.in          template for config.h, when built by "configure"
   src/pcre2.h.in           template for pcre2.h when built by "configure"
@@ -814,7 +835,7 @@ The distribution should contain the files listed below.
   libpcre2-8.pc.in         template for libpcre2-8.pc for pkg-config
   libpcre2-16.pc.in        template for libpcre2-16.pc for pkg-config
   libpcre2-32.pc.in        template for libpcre2-32.pc for pkg-config
-  libpcre2posix.pc.in      template for libpcre2posix.pc for pkg-config
+  libpcre2-posix.pc.in     template for libpcre2-posix.pc for pkg-config
   ltmain.sh                file used to build a libtool script
   missing                  ) common stub for a few missing GNU programs while
                            )   installing, generated by automake
@@ -845,4 +866,4 @@ The distribution should contain the files listed below.
 Philip Hazel
 Email local part: ph10
 Email domain: cam.ac.uk
-Last updated: 01 April 2016
+Last updated: 01 November 2016

doc/html/index.html

@@ -94,6 +94,9 @@ <h1>Perl-compatible Regular Expressions (revised API: PCRE2)</h1>
 <tr><td><a href="pcre2_code_copy.html">pcre2_code_copy</a></td>
     <td>&nbsp;&nbsp;Copy a compiled pattern</td></tr>
 
+<tr><td><a href="pcre2_code_copy_with_tables.html">pcre2_code_copy_with_tables</a></td>
+    <td>&nbsp;&nbsp;Copy a compiled pattern and its character tables</td></tr>
+
 <tr><td><a href="pcre2_code_free.html">pcre2_code_free</a></td>
     <td>&nbsp;&nbsp;Free a compiled pattern</td></tr>
 

doc/html/pcre2_code_copy.html

@@ -28,8 +28,9 @@ <h1>pcre2_code_copy man page</h1>
 This function makes a copy of the memory used for a compiled pattern, excluding
 any memory used by the JIT compiler. Without a subsequent call to
 <b>pcre2_jit_compile()</b>, the copy can be used only for non-JIT matching. The
-yield of the function is NULL if <i>code</i> is NULL or if sufficient memory
-cannot be obtained.
+pointer to the character tables is copied, not the tables themselves (see
+<b>pcre2_code_copy_with_tables()</b>). The yield of the function is NULL if
+<i>code</i> is NULL or if sufficient memory cannot be obtained.
 </P>
 <P>
 There is a complete description of the PCRE2 native API in the

doc/html/pcre2_code_copy_with_tables.html

@@ -0,0 +1,44 @@
+<html>
+<head>
+<title>pcre2_code_copy_with_tables specification</title>
+</head>
+<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB">
+<h1>pcre2_code_copy_with_tables man page</h1>
+<p>
+Return to the <a href="index.html">PCRE2 index page</a>.
+</p>
+<p>
+This page is part of the PCRE2 HTML documentation. It was generated
+automatically from the original man page. If there is any nonsense in it,
+please consult the man page, in case the conversion went wrong.
+<br>
+<br><b>
+SYNOPSIS
+</b><br>
+<P>
+<b>#include &#60;pcre2.h&#62;</b>
+</P>
+<P>
+<b>pcre2_code *pcre2_code_copy_with_tables(const pcre2_code *<i>code</i>);</b>
+</P>
+<br><b>
+DESCRIPTION
+</b><br>
+<P>
+This function makes a copy of the memory used for a compiled pattern, excluding
+any memory used by the JIT compiler. Without a subsequent call to
+<b>pcre2_jit_compile()</b>, the copy can be used only for non-JIT matching. 
+Unlike <b>pcre2_code_copy()</b>, a separate copy of the character tables is also
+made, with the new code pointing to it. This memory will be automatically freed
+when <b>pcre2_code_free()</b> is called. The yield of the function is NULL if
+<i>code</i> is NULL or if sufficient memory cannot be obtained.
+</P>
+<P>
+There is a complete description of the PCRE2 native API in the
+<a href="pcre2api.html"><b>pcre2api</b></a>
+page and a description of the POSIX API in the
+<a href="pcre2posix.html"><b>pcre2posix</b></a>
+page.
+<p>
+Return to the <a href="index.html">PCRE2 index page</a>.
+</p>

doc/html/pcre2_set_max_pattern_length.html

@@ -26,8 +26,11 @@ <h1>pcre2_set_max_pattern_length man page</h1>
 DESCRIPTION
 </b><br>
 <P>
-This function sets, in a compile context, the maximum length (in code units) of
-the pattern that can be compiled. The result is always zero.
+This function sets, in a compile context, the maximum text length (in code
+units) of the pattern that can be compiled. The result is always zero. If a
+longer pattern is passed to <b>pcre2_compile()</b> there is an immediate error
+return. The default is effectively unlimited, being the largest value a
+PCRE2_SIZE variable can hold.
 </P>
 <P>
 There is a complete description of the PCRE2 native API in the

doc/html/pcre2api.html

@@ -294,6 +294,9 @@ <h1>pcre2api man page</h1>
 <b>pcre2_code *pcre2_code_copy(const pcre2_code *<i>code</i>);</b>
 <br>
 <br>
+<b>pcre2_code *pcre2_code_copy_with_tables(const pcre2_code *<i>code</i>);</b>
+<br>
+<br>
 <b>int pcre2_get_error_message(int <i>errorcode</i>, PCRE2_UCHAR *<i>buffer</i>,</b>
 <b>  PCRE2_SIZE <i>bufflen</i>);</b>
 <br>
@@ -567,8 +570,9 @@ <h1>pcre2api man page</h1>
 (perhaps waiting to see if the pattern is used often enough) similar logic is
 required. JIT compilation updates a pointer within the compiled code block, so
 a thread must gain unique write access to the pointer before calling
-<b>pcre2_jit_compile()</b>. Alternatively, <b>pcre2_code_copy()</b> can be used
-to obtain a private copy of the compiled code.
+<b>pcre2_jit_compile()</b>. Alternatively, <b>pcre2_code_copy()</b> or 
+<b>pcre2_code_copy_with_tables()</b> can be used to obtain a private copy of the
+compiled code.
 </P>
 <br><b>
 Context blocks
@@ -736,7 +740,8 @@ <h1>pcre2api man page</h1>
 <br>
 This parameter ajusts the limit, set when PCRE2 is built (default 250), on the
 depth of parenthesis nesting in a pattern. This limit stops rogue patterns
-using up too much system stack when being compiled.
+using up too much system stack when being compiled. The limit applies to 
+parentheses of all kinds, not just capturing parentheses.
 <b>int pcre2_set_compile_recursion_guard(pcre2_compile_context *<i>ccontext</i>,</b>
 <b>  int (*<i>guard_function</i>)(uint32_t, void *), void *<i>user_data</i>);</b>
 <br>
@@ -1058,6 +1063,9 @@ <h1>pcre2api man page</h1>
 <br>
 <br>
 <b>pcre2_code *pcre2_code_copy(const pcre2_code *<i>code</i>);</b>
+<br>
+<br>
+<b>pcre2_code *pcre2_code_copy_with_tables(const pcre2_code *<i>code</i>);</b>
 </P>
 <P>
 The <b>pcre2_compile()</b> function compiles a pattern into an internal form.
@@ -1079,9 +1087,22 @@ <h1>pcre2api man page</h1>
 <a href="#jitcompiling">below),</a>
 the JIT information cannot be copied (because it is position-dependent).
 The new copy can initially be used only for non-JIT matching, though it can be
-passed to <b>pcre2_jit_compile()</b> if required. The <b>pcre2_code_copy()</b>
-function provides a way for individual threads in a multithreaded application
-to acquire a private copy of shared compiled code.
+passed to <b>pcre2_jit_compile()</b> if required. 
+</P>
+<P>
+The <b>pcre2_code_copy()</b> function provides a way for individual threads in a
+multithreaded application to acquire a private copy of shared compiled code. 
+However, it does not make a copy of the character tables used by the compiled 
+pattern; the new pattern code points to the same tables as the original code.
+(See
+<a href="#jitcompiling">"Locale Support"</a>
+below for details of these character tables.) In many applications the same
+tables are used throughout, so this behaviour is appropriate. Nevertheless, 
+there are occasions when a copy of a compiled pattern and the relevant tables 
+are needed. The <b>pcre2_code_copy_with_tables()</b> provides this facility. 
+Copies of both the code and the tables are made, with the new code pointing to 
+the new tables. The memory for the new tables is automatically freed when
+<b>pcre2_code_free()</b> is called for the new copy of the compiled code.
 </P>
 <P>
 NOTE: When one of the matching functions is called, pointers to the compiled
@@ -1119,7 +1140,14 @@ <h1>pcre2api man page</h1>
 error code and an offset (number of code units) within the pattern,
 respectively, when <b>pcre2_compile()</b> returns NULL because a compilation
 error has occurred. The values are not defined when compilation is successful
-and <b>pcre2_compile()</b> returns a non-NULL value.
+and <b>pcre2_compile()</b> returns a non-NULL value. 
+</P>
+<P>
+The value returned in <i>erroroffset</i> is an indication of where in the
+pattern the error occurred. It is not necessarily the furthest point in the
+pattern that was read. For example, after the error "lookbehind assertion is
+not fixed length", the error offset points to the start of the failing
+assertion.
 </P>
 <P>
 The <b>pcre2_get_error_message()</b> function (see "Obtaining a textual error
@@ -1215,8 +1243,8 @@ <h1>pcre2api man page</h1>
   PCRE2_AUTO_CALLOUT
 </pre>
 If this bit is set, <b>pcre2_compile()</b> automatically inserts callout items,
-all with number 255, before each pattern item. For discussion of the callout
-facility, see the
+all with number 255, before each pattern item, except immediately before or
+after a callout in the pattern. For discussion of the callout facility, see the
 <a href="pcre2callout.html"><b>pcre2callout</b></a>
 documentation.
 <pre>
@@ -3235,7 +3263,7 @@ <h1>pcre2api man page</h1>
 </P>
 <br><a name="SEC41" href="#TOC1">REVISION</a><br>
 <P>
-Last updated: 17 June 2016
+Last updated: 22 November 2016
 <br>
 Copyright &copy; 1997-2016 University of Cambridge.
 <br>