#- # Copyright 2014-2018 UPLEX Nils Goroll Systemoptimierung # All rights reserved # # This document is licensed under the same conditions as the libvmod-re # project. See LICENSE for details. # # Authors: Geoffrey Simmons # Nils Goroll $Module re 3 "Varnish Module for Regular Expression Matching with Subexpression Capture" $Synopsis manual SYNOPSIS ======== :: import re; # object interface new = re.regex(STRING, [, INT limit] [, INT limit_recursion]) BOOL .match(STRING, [, INT limit] [, INT limit_recursion]) STRING .backref(INT [, STRING fallback]) # function interface BOOL re.match_dyn(STRING, [, INT limit] [, INT limit_recursion]) STRING re.backref_dyn(INT [, STRING fallback]) STRING re.version() DESCRIPTION =========== Varnish Module (VMOD) for matching strings against regular expressions, and for extracting captured substrings after matches. Regular expression matching as implemented by the VMOD is equivalent to VCL's infix operator ``~``. The VMOD is motivated by the fact that backreference capture in standard VCL requires verbose and suboptimal use of the ``regsub`` or ``regsuball`` functions. For example, this common idiom in VCL captures a string of digits following the substring ``"bar"`` from one request header into another:: sub vcl_recv { if (req.http.Foo ~ "bar\d+")) { set req.http.Baz = regsub(req.http.Foo, "^.*bar(\d+).*$", "\1"); } } It requires two regex executions when a match is found, the second one less efficient than the first (since it must match the entire string to be replaced while capturing a substring), and is just cumbersome. The equivalent solution with the VMOD looks like this:: import re; sub vcl_init { new myregex = re.regex("bar(\d+)"); } sub vcl_recv { if (myregex.match(req.http.Foo)) { set req.http.Baz = myregex.backref(1); } } The object is created at VCL initialization with the regex containing the capture expression, only describing the substring to be matched. When a match with the ``match`` method succeeds, then a captured string can be obtained from the ``backref`` method. Calls to the ``backref`` method refer back to the most recent successful call to ``match`` for the same object in the same task scope; that is, in the same client or backend context. For example if ``match`` is called for an object in one of the ``vcl_backend_*`` subroutines and returns ``true``, then subsequent calls to ``backref`` in the same backend scope extract substrings from the matched substring. The VMOD also supports dynamic regex matching with the ``match_dyn`` and ``backref_dyn`` functions:: import re; sub vcl_backend_response { if (re.match_dyn(beresp.http.Bar + "(\d+)", req.http.Foo)) { set beresp.http.Baz = re.backref_dyn(1); } } In ``match_dyn``, the regex in the first argument is compiled when it is called, and matched against the string in the second argument. Subsequent calls to ``backref_dyn`` extract substrings from the matched string for the most recent successful call to ``match_dyn`` in the same task scope. As with the constructor, the regex argument to ``match_dyn`` should contain any capturing expressions needed for calls to ``backref_dyn``. ``match_dyn`` makes it possible to construct regexen whose contents are not fully known until runtime, but ``match`` is more efficient, since it re-uses the compiled expression obtained at VCL initialization. So if you are matching against a fixed pattern that never changes during the lifetime of VCL, use ``match``. $Object regex(STRING, INT limit=1000, INT limit_recursion=1000) Description Create a regex object with the given regular expression. The expression is compiled when the constructor is called. It should include any capturing parentheses that will be needed for extracting backreferences. If the regular expression fails to compile, then the VCL load fails with an error message describing the problem. The optional parameters ``limit`` and ``limit_recursion`` are per-object defaults for the respective parameters of the `vmod_re.regex.match`_ method. Example ``new myregex = re.regex("\bmax-age\s*=\s*(\d+)");`` $Method BOOL .match(STRING, INT limit=0, INT limit_recursion=0) Description Determines whether the given string matches the regex compiled by the constructor; functionally equivalent to VCL's infix operator ``~``. The optional parameter ``limit`` restricts the number of internal matching function calls in a ``pcre_exec()`` execution, analogous to the varnishd ``pcre_match_limit`` parameter. For the default value 0, the ``limit`` given to the constructor `vmod_re.regex`_ is used. The optional parameter ``limit_recursion`` restricts the number of internal matching function recursions in a ``pcre_exec()`` execution, analogous to the varnishd ``pcre_match_limit_recursion`` parameter. For the default value 0, the ``limit_recursion`` given to the constructor `vmod_re.regex`_ is used. Example ``if (myregex.match(beresp.http.Surrogate-Control)) { # ...`` $Method STRING .backref(INT, STRING fallback="**BACKREF METHOD FAILED**") Description Extracts the `nth` subexpression of the most recent successful call of the ``match`` method for this object in the same task scope (client or backend context), or a fallback string in case the extraction fails. Backref 0 indicates the entire matched string. Thus this function behaves like the ``\n`` symbols in ``regsub`` and ``regsuball``, and the ``$1``, ``$2`` ... variables in Perl. After unsuccessful matches, the ``fallback`` string is returned for any call to ``backref``. The default value of ``fallback`` is ``"**BACKREF METHOD FAILED**"``. The VCL infix operators ``~`` and ``!~`` do not affect this method, nor do the functions ``regsub`` or ``regsuball``. If ``backref`` is called without any prior call to ``match`` for this object in the same task scope, then an error message is emitted to the Varnish log using the ``VCL_Error`` tag, and the fallback string is returned. Example ``set beresp.ttl = std.duration(myregex.backref(1, "120"), 120s);`` $Function BOOL match_dyn(PRIV_TASK, STRING, STRING, INT limit=1000, INT limit_recursion=1000) Description Compiles the regular expression given in the first argument, and determines whether it matches the string in the second argument. If the regular expression fails to compile, then an error message describing the problem is emitted to the Varnish log with the tag ``VCL_Error``, and ``match_dyn`` returns ``false``. For parameters ``limit`` and ``limit_recursion`` see `vmod_re.regex.match`_, except that there is no object to inherit defaults from. Example ``if (re.match_dyn(req.http.Foo + "(\d+)", beresp.http.Bar)) { # ...`` $Function STRING backref_dyn(PRIV_TASK, INT, STRING fallback="**BACKREF FUNCTION FAILED**") Description Similar to the ``backref`` method, this function extracts the `nth` subexpression of the most recent successful call of the ``match_dyn`` function in the same task scope, or a fallback string in case the extraction fails. After unsuccessful matches, the ``fallback`` string is returned for any call to ``backref_dyn``. The default value of ``fallback`` is ``"**BACKREF FUNCTION FAILED**"``. If ``backref_dyn`` is called without any prior call to ``match_dyn`` in the same task scope, then a ``VCL_Error`` message is logged, and the fallback string is returned. $Function STRING version() Description Returns the version string for this vmod. Example ``set resp.http.X-re-version = re.version();`` REQUIREMENTS ============ The VMOD requires the Varnish since version 6.0.0 or the master branch. See the project repository for versions that are compatible with other versions of Varnish. INSTALLATION ============ See `INSTALL.rst `_ in the project source repository. LIMITATIONS =========== The VMOD allocates memory for captured subexpressions from Varnish workspaces, whose sizes are determined by the runtime parameters ``workspace_backend``, for calls within the ``vcl_backend_*`` subroutines, and ``workspace_client``, for the other VCL subs. The VMOD copies the string to be matched into the workspace, if it's not already in the workspace, and also uses workspace to save data about backreferences. For typical usage, the default workspace sizes are probably enough; but if you are matching against many, long strings in each client or backend context, you might need to increase the Varnish parameters for workspace sizes. If the VMOD cannot allocate enough workspace, then a ``VCL_error`` message is emitted, and the match methods as well as ``backref`` will fail. (If you're just using the regexen for matching and not to capture backrefs, then you might as well just use the standard VCL operators ``~`` and ``!~``, and save the workspace.) ``backref`` can extract up to 10 subexpressions, in addition to the full expression indicated by backref 0. If a ``match`` or ``match_dyn`` operation would have resulted in more than 11 captures (10 substrings and the full string), then a ``VCL_Error`` message is emitted to the Varnish log, and the captures are limited to 11. SEE ALSO ======== * varnishd(1) * vcl(7) * pcre(3) * source repository: https://code.uplex.de/uplex-varnish/libvmod-re $ABI vrt