Skip to content

L
libvmod-re2
Commit 6d20f3a9 authored by Geoff Simmons's avatar Geoff Simmons
Browse files
Options

Document recent changes in the VMOD. 

- Changed error handling -- VCL failure for unrecoverable errors.
- .compile() is now unnecessary and deprecated.
- Addition of .subroutine() and .check_call().
- Update the REQUIREMENTS and compatibility statements.
- Fix some typos noticed along the way.
parent 04b3e9e0
Hide whitespace changes
Inline Side-by-side
Showing with 503 additions and 278 deletions
README.md
View file @ 6d20f3a9
......@@ -34,8 +34,7 @@
# set object interface
new OBJECT = re2.set([ENUM anchor] [, <regex options>])
VOID <obj>.add(STRING [, BOOL save] [, BOOL never_capture] [, STRING string]
[, BACKEND backend] [, INT integer])
VOID <obj>.compile()
[, BACKEND backend] [, INT integer] [,SUB sub])
BOOL <obj>.match(STRING)
INT <obj>.nmatches()
BOOL <obj>.matched(INT)
......@@ -43,13 +42,15 @@
STRING <obj>.string([INT n,] [ENUM select])
BACKEND <obj>.backend([INT n,] [ENUM select])
INT <obj>.integer([INT n] [, ENUM select])
SUB <obj>.subroutine([INT n] [, ENUM select])
BOOL <obj>.check_call([INT n] [, ENUM select])
STRING <obj>.sub(STRING text, STRING rewrite [, INT n]
[, ENUM select])
STRING <obj>.suball(STRING text, STRING rewrite [, INT n]
[, ENUM select])
STRING <obj>.extract(STRING text, STRING rewrite [, INT n]
[, ENUM select])
BOOL <obj>.saved([ENUM {REGEX, STR, BE, INT} which] [, INT n]
BOOL <obj>.saved([ENUM {REGEX, STR, BE, INT, SUB} which] [, INT n]
[, ENUM select])
VOID <obj>.hdr_filter(HTTP [, BOOL])
......@@ -130,7 +131,6 @@ example:
myset.add("foo"); # Pattern 1
myset.add("bar"); # Pattern 2
myset.add("baz"); # Pattern 3
myset.compile();
}
`myset.match(<string>)` can now be used to match a string against the
......@@ -182,14 +182,12 @@ method:
prefix.add("/bar", string="www.domain2.com");
prefix.add("/baz", string="www.domain3.com");
prefix.add("/quux", string="www.domain4.com");
prefix.compile();
new appmatcher = re2.set(anchor=start);
appmatcher.add("/foo", backend=app1);
appmatcher.add("/bar", backend=app2);
appmatcher.add("/baz", backend=app3);
appmatcher.add("/quux", backend=app4);
appmatcher.compile();
}
After a successful match, the string or backend associated with the
......@@ -280,10 +278,11 @@ the resulting alternation.
- `never_capture`
If true, parentheses in a pattern are interpreted as non-capturing,
and all invocations of the `backref` and `namedref` methods or
functions will fail, including `backref(0)` after a successful
match. Default is **false**, except for set objects, for which
`never_capture` is always true (and cannot be changed), since back
references are not possible with sets.
functions will lead to VCL faillure (see [ERRORS](#errors)),
including `backref(0)` after a successful match. Default is
**false**, except for set objects, for which `never_capture` is
always true (and cannot be changed), since back references are not
possible with sets.
- `case_sensitive`
If true, matches are case-sensitive. A pattern can override this
......@@ -362,10 +361,10 @@ Example:
Returns the nth captured subexpression from the most recent successful
call of the `.match()` method for this object in the same client or
backend, context, or a fallback string in case the capture fails.
Backref 0 indicates the entire matched string. Thus this function
behaves like the `\n` in the native VCL functions `regsub` and
`regsuball`, and the `$1`, `$2` ... variables in Perl.
backend context, or a fallback string in case the capture fails. Backref
0 indicates the entire matched string. Thus this function behaves like
the `\n` notation in the native VCL functions `regsub` and `regsuball`,
and the `$1`, `$2` ... variables in Perl.
Since Varnish client and backend operations run in different threads,
`.backref()` can only refer back to a `.match()` call in the same
......@@ -375,15 +374,16 @@ subroutines -- the backend context -- refers back to a previous
other VCL subroutines -- the client context -- refers back to a
`.match()` in the same client context.
After unsuccessful matches, the `fallback` string is returned for any
call to `.backref()`. The default value of `fallback` is `"**BACKREF
METHOD FAILED**"`. `.backref()` always fails after a failed match, even
if `.match()` had been called successfully before the failure.
`.backref()` may also return `fallback` after a successful match, if no
`.backref()` may return `fallback` after a successful match, if no
captured group in the matching string corresponds to the backref number.
For example, when the pattern `(a|(b))c` matches the string `ac`, there
is no backref 2, since nothing matches `b` in the string.
is no backref 2, since nothing matches `b` in the string. The default
value of `fallback` is `"**BACKREF METHOD FAILED**"`, but you may set
another value (such as the empty string).
After unsuccessful matches, `.backref()` invokes VCL failure (see
[ERRORS](#errors)). `.backref()` always fails after a failed match, even
if `.match()` had been called successfully before the failure.
The VCL infix operators `~` and `!~` do not affect this method, nor do
the functions `regsub` or `regsuball`. Nor is it affected by the matches
......@@ -391,10 +391,9 @@ performed by any other method or function in this VMOD (such as the
`sub()`, `suball()` or `extract()` methods or functions, or the `set`
object's `.match()` method).
`.backref()` fails, returning `fallback` and writing an error message to
the Varnish log with the `VCL_Error` tag, under the following conditions
(even if a previous match was successful and a substring could have been
captured):
`.backref()` invokes VCL failure under the following conditions, even if
a previous match was successful and a substring could have been captured
(see [ERRORS](#errors)):
- The `fallback` string is undefined, for example if set from an unset
header variable.
......@@ -422,7 +421,7 @@ Example:
Returns the captured subexpression designated by `name` from the most
recent successful call to `.match()` in the current context (client or
backend), or `fallback` in case of failure.
backend).
Named capturing groups are written in RE2 as: `(?P<name>re)`. (Note that
this syntax with `P`, inspired by Python, differs from the notation for
......@@ -432,16 +431,14 @@ named capturing groups in PCRE.) Thus when `(?P<foo>.+)bar$` matches
Note that a named capturing group can also be referenced as a numbered
group. So in the previous example, `.backref(1)` also returns `baz`.
`fallback` is returned when `.namedref()` is called after an
unsuccessful match. The default fallback is `"**NAMEDREF METHOD
FAILED**"`.
`fallback` is returned when the named reference did not match. The
default fallback is `"**NAMEDREF METHOD FAILED**"`.
Like `.backref()`, `.namedref()` is not affected by native VCL regex
operations, nor by any other matches performed by methods or functions
of the VMOD, except for a prior `.match()` for the same object.
`.namedref()` fails, returning `fallback` and logging a `VCL_Error`
message, if:
`.namedref()` invokes VCL failure (see [ERRORS](#errors)) if:
- The `fallback` string is undefined.
- `name` is undefined or the empty string.
......@@ -480,8 +477,7 @@ matching text. This method corresponds to the VCL native function
`fallback` is returned if the pattern does not match `text`. The default
fallback is `"**SUB METHOD FAILED**"`.
`.sub()` fails, returning `fallback` and logging a `VCL_Error` message,
if:
`.sub()` invokes VCL failure (see [ERRORS](#errors)) if:
- Any of `text`, `rewrite` or `fallback` are undefined.
- There is insufficient workspace for the rewritten string.
......@@ -600,8 +596,8 @@ Like the `regex.match()` method, return `true` if `pattern` matches
`subject`, where `pattern` is compiled with the given options (or
default options) on each invocation.
If `pattern` fails to compile, then an error message is logged with the
`VCL_Error` tag, and `false` is returned.
If `pattern` fails to compile, then VCL failure is invoked (see
[ERRORS](#errors)).
Example:
......@@ -622,20 +618,20 @@ call of the `match()` function in the current client or backend context,
or a fallback string if the capture fails. The default `fallback` is
`"**BACKREF FUNCTION FAILED**"`.
Similarly to the `regex.backref()` method, `fallback` is returned after
any failed invocation of the `match()` function, or if there is no
captured group corresponding to the backref number. The function is not
affected by native VCL regex operations, or any other method or function
of the VMOD except for the `match()` function.
Similarly to the `regex.backref()` method, `fallback` is returned if
there is no captured group corresponding to the backref number. The
function is not affected by native VCL regex operations, or any other
method or function of the VMOD except for the `match()` function.
The function fails, returning `fallback` and logging a `VCL_Error`
message, under the same conditions as the corresponding method:
The function invokes VCL failure under the same conditions as the
corresponding method (see [ERRORS](#errors)):
- `fallback` is undefined.
- `never_capture` was true in the previous invocation of the `match()`
function.
- `ref` is out of range.
- The `match()` function was never called in this context.
- The `match()` function was never called in this context, or if the
previous `match()` call failed (returned `false`).
- The pattern failed to compile for the previous `match()` call.
- There is insufficient workspace for the captured subexpression.
......@@ -656,20 +652,18 @@ Example:
Returns the captured subexpression designated by `name` from the most
recent successful call to the `match()` function in the current context,
or `fallback` in case of failure. The default fallback is `"**NAMEDREF
FUNCTION FAILED**"`.
or `fallback` if the corresponding group did not match. The default
fallback is `"**NAMEDREF FUNCTION FAILED**"`.
The function returns `fallback` when the previous invocation of the
`match()` function failed, and is only affected by use of the `match()`
function. The function fails, returning `fallback` and logging a
`VCL_Error` message, under the same conditions as the corresponding
method:
The function invokes VCL failure under the same conditions as the
corresponding method (see [ERRORS](#errors)):
- `fallback` is undefined.
- `name` is undefined or the empty string.
- The `never_capture` option was set to `true`.
- There is no such named group.
- `match()` was not called in this context.
- `match()` was not called in this context, or the previous call
failed.
- The pattern failed to compile for the previous `match()` call.
- There is insufficient workspace for the captured expression.
......@@ -708,8 +702,7 @@ used in `rewrite` to substitute captured groups from the pattern.
`fallback` is returned if the pattern does not match `text`. The default
fallback is `"**SUB FUNCTION FAILED**"`.
`sub()` fails, returning `fallback` and logging a `VCL_Error` message,
if:
`sub()` invokes VCL failure (see [ERRORS](#errors)) if:
- `pattern` cannot be compiled.
- Any of `text`, `rewrite` or `fallback` are undefined.
......@@ -817,7 +810,8 @@ Like the `.cost()` method above, return a numeric measurement \> 0 from
the RE2 library for `pattern` with the given options. More complex
regexen have a higher cost than less complex regexen.
Fails and returns -1 if `pattern` cannot be compiled.
Invokes VCL failure if `pattern` cannot be compiled (see
[ERRORS](#errors)).
Example:
......@@ -862,18 +856,26 @@ the resulting composed pattern. However, the `never_capture` option
cannot be set, and is always implicitly true, since backrefs and
namedrefs are not possible with sets.
Sets are compiled automatically when `vcl_init` finishes (or when the
deprecated `.compile()` method is called). Compilation fails if any of
the added patterns cannot be compiled, or if no patterns were added to
the set. It may also fail if the `max_mem` setting is not large enough
for the composed pattern. In that case, the VCL load will fail with an
error message (then consider a larger value for `max_mem` in the set
constructor).
Example:
sub vcl_init {
# Initialize a regex set for partial matches
# Initialize a regex set for partial matches
# with default options
new foo = re2.set();
# Initialize a regex set for case insensitive matches
# Initialize a regex set for case insensitive matches
# with anchors on both ends (^ and $).
new bar = re2.set(anchor=both, case_sensitive=false);
# Initialize a regex set using POSIX syntax, but allowing
# Initialize a regex set using POSIX syntax, but allowing
# Perl character classes, and anchoring at the left (^).
new baz = re2.set(anchor=start, posix_syntax=true,
perl_classes=true);
......@@ -895,12 +897,12 @@ Add the given pattern to the set. If the pattern is invalid, `.add()`
fails, and the VCL will fail to load, with an error message describing
the problem.
If values for the `string`, `backend` and/or `integer` parameters are
provided, then these values can be retrieved with the `.string()`,
`.backend()` and `.integer()` methods, respectively, as described below.
This makes it possible to associate data with the added pattern after it
matches successfully. By default the pattern is not associated with any
such value.
If values for the `string`, `backend`, `integer` and/or `sub` parameters
are provided, then these values can be retrieved with the `.string()`,
`.backend()`, `.integer()` and `.subroutine()` methods, respectively, as
described below. This makes it possible to associate data with the added
pattern after it matches successfully. By default the pattern is not
associated with any such value.
If `save` is true, then the given pattern is compiled and saved as a
`regex` object, just as if the `regex` constructor described above is
......@@ -922,13 +924,10 @@ for the individual regex, even though it is implicitly set to true for
the full set object (default is false).
`.add()` MUST be called in `vcl_init`, and MAY NOT be called after
`.compile()`. If `.add()` is called in any other subroutine, an error
message with `VCL_Error` is logged, and the call has no effect. If it is
called in `vcl_init` after `.compile()`, then the VCL load will fail
with an error message.
In other words, add all patterns to the set in `vcl_init`, and finally
call `.compile()` when you're done.
`.compile()`. VCL failure is invoked if `.add()` is called in any other
subroutine (see [ERRORS](#errors)). If it is called in `vcl_init` after
`.compile()`, then the VCL load will fail with an error message. Note
that `.compile()` is now unneccessary and deprecated.
When the `.matched(INT)` method is called after a successful match, the
numbering corresponds to the order in which patterns were added. The
......@@ -945,7 +944,6 @@ Example:
hostmatcher.add("www.domain1.com");
hostmatcher.add("www.domain2.com");
hostmatcher.add("www.domain3.com");
hostmatcher.compile();
}
# See the documentation of the .string() and .backend() methods
......@@ -953,20 +951,20 @@ Example:
#### VOID xset.compile()
**This method is deprecated**, and will be removed in a future version.
`.compile()` may be omitted, since compilation now happens automatically
when `vcl_init` finishes.
Compile the compound pattern represented by the set -- an alternation of
all patterns added by `.add()`.
`.compile()` fails if no patterns were added to the set. It may also
fail if the `max_mem` setting is not large enough for the composed
pattern. In that case, the VCL load will fail with an error message
(then consider a larger value for `max_mem` in the set constructor).
Compilation may fail for any of the reasons described for automatic
compilation of set objects as described above.
`.compile()` MUST be called in `vcl_init`, and MAY NOT be called more
than once for a set object. If it is called in any other subroutine, a
`VCL_Error` message is logged, and the call has no effect. If it is
called a second time in `vcl_init`, the VCL load will fail.
See above for examples.
than once for a set object. VCL failure is invoked if it is called in
any other subroutine. If it is called a second time in `vcl_init`, the
VCL load will fail.
#### BOOL xset.match(STRING)
......@@ -979,17 +977,12 @@ and match the given string. These can be determined after a successful
match using the `.matched(INT)` and `.nmatches()` methods described
below.
`.match()` MUST be called after `.compile()`; otherwise the match always
fails.
A match may also fail (returning `false`) if the internal memory limit
imposed by the `max_mem` parameter in the constructor is exceeded. (With
the default value of `max_mem`, this ordinarily requires very large
patterns and/or a very large string to be matched.) Since about version
2017-12-01, the RE2 library reports this condition; if so, the VMOD
writes a `VCL_Error` message in the log if it happens, except during
`vcl_init`, in which case the VCL load fails with the error message. If
matches fail due to the out-of-memory condition, increase the `max_mem`
A match may also fail (leading to VCL failure) if the internal memory
limit imposed by the `max_mem` parameter in the constructor is exceeded.
(With the default value of `max_mem`, this ordinarily requires very
large patterns and/or a very large string to be matched.) Since about
version 2017-12-01, the RE2 library reports this condition. If matches
fail due to the out-of-memory condition, increase the `max_mem`
parameter in the constructor.
Example:
......@@ -1010,20 +1003,13 @@ the same object in the same client or backend context. It always returns
`false`, for every value of the parameter, if it is called after an
unsuccessful match (`.match()` returned `false`).
`.matched()` fails and returns `false` if:
`.matched()` invokes VCL failure (see [ERRORS](#errors)) if:
- The `.match()` method was not called for this object in the same
client or backend scope.
- The integer parameter is out of range; that is, if it is less than 1
or greater than the number of patterns added to the set.
On failure, the method writes an error message to the log with the tag
`VCL_Error`; if it fails during `vcl_init`, then the VCL load fails with
the error message. In any other VCL subroutine, the method returns
`false` on failure and processing continues; since `false` is a
legitimate return value, you should consider monitoring the log for the
error messages.
Example:
if (hostmatcher.match(req.http.Host)) {
......@@ -1046,12 +1032,8 @@ backend context. The method always returns 0 after an unsuccessful match
(`.match()` returned `false`).
If `.match()` was not called for this object in the same client or
backend scope, `.nmatches()` fails and returns 0, writing an error
message with `VCL_Error` to the log. If this happens in `vcl_init`, the
VCL load fails with the error message. As with `.matched()`,
`.nmatches()` returns a legitimate value and VCL processing continues
when it fails in any other subroutine, so you should monitor the log for
the error messages.
backend scope, `.nmatches()` invokes VCL failure (see
[ERRORS](#errors)).
Example:
......@@ -1078,7 +1060,7 @@ is returned. The values `FIRST` and `LAST` specify that, of the patterns
that matched, the first or last one added via the `.add()` method is
chosen, and the number for that pattern is returned.
`.which()` fails, returning 0 with a `VCL_Error` message in the log, if:
`.which()` invokes VCL failure (see [ERRORS](#errors)) if:
- `.match()` was not called for the set in the current client or
backend transaction, or if the previous call returned `false`.
......@@ -1158,8 +1140,7 @@ For the pattern selected by these rules, return the string that was set
with the `string` parameter in the `.add()` method that added the
pattern to the set.
`.string()` fails, returning NULL with an a `VCL_Error` message in the
log, if:
`.string()` invokes VCL failure (see [ERRORS](#errors)) if:
- The values of `n` and `select` are invalid:
- `n` is greater than the number of patterns in the set.
......@@ -1169,9 +1150,11 @@ log, if:
- `n` \<= 0, but the previous `.match()` call returned `false`.
- `n` \<= 0 and the `select` ENUM is `UNIQUE` (or default), but
more than one pattern matched in the previous `.match()` call.
This can be avoided by checking for `.nmatches() == 1`.
- No string was associated with the pattern selected by `n` and
`select`; that is, the `string` parameter was not set in the
`.add()` call that added the pattern.
`.add()` call that added the pattern. This can be avoided by
checking the `.saved()` method (see below).
Examples:
......@@ -1255,10 +1238,10 @@ The rules for selecting a pattern from the set and its associated
backend based on `n` and `select` are the same as described above for
`.string()`.
`.backend()` fails, returning NULL with an a `VCL_Error` message in the
log, under the same conditions described for `.string()` above -- `n`
and `select` are invalid, or no backend was associated with the selected
pattern with the `.add()` method.
`.backend()` invokes VCL failure under the same conditions described for
`.string()` above -- `n` and `select` are invalid, or no backend was
associated with the selected pattern with the `.add()` method (see
[ERRORS](#errors)).
Example:
......@@ -1308,18 +1291,7 @@ integer based on `n` and `select` are the same as described above for
`.integer()` invokes VCL failure under the same error conditions
described for `.string()` above -- `n` and `select` are invalid, or no
integer was associated with the selected pattern with the `.add()`
method.
Note that VCL failure differs from the failure mode for `.string()` and
`.backend()`, since there is no distinguished "error" value that could
be returned as the INT. VCL failure has the same effect as if
`return(fail)` were called from a VCL subroutine; usually, control
directs immediately to `vcl_synth`, with the response status set to 503,
and the response reason set to "VCL failed".
You can avoid that, for example, by testing if `.nmatches()==1` after
calling `.match()`, if you need to ensure that calling
`.integer(select=UNIQUE)` will not fail.
method (see [ERRORS](#errors)).
Example:
......@@ -1383,8 +1355,7 @@ the rules given above. If an internal `regex` object was saved for that
pattern, then the result of the `.sub()` method invoked on that object
is returned.
`.sub()` fails, returning NULL with a `VCL_Error` message in the log,
if:
`.sub()` invokes VCL failkure (see [ERRORS](#errors)) if:
- The values of `n` and `select` are invalid, according to the rules
given above.
......@@ -1531,7 +1502,59 @@ Example:
ENUM {FIRST, LAST, UNIQUE} select=UNIQUE
)
XXX ...
Returns the subroutine set by the `sub` parameter for the element of the
set indicated by `n` and `select`, according to the rules given above.
The subroutine may be invoked with VCL `call`.
**Note**: you must ensure that the subroutine may invoked legally in the
context in which it is called. This means that:
- The subroutine may only refer to VCL elements that are legal in the
invocation context. For example, if the subroutine only refers to
headers in `req.http.*`, then it may be called in `vcl_recv`, but
not if it refers to any header in `resp.http.*`. See `vcl-var(7)`
for the specification of which VCL variables may be used in which
contexts.
- Recursive subroutine calls are not permitted in VCL. The subroutine
invocation may not appear anywhere in its own call stack.
For standard subroutine invocations with `call`, the VCL compiler checks
these conditions and issues a compile-time error if either one is
violated. This is not possible with invocations using `.subroutine()`;
the error can only be determined at runtime. So it is advisable to test
the use of `.subroutine()` carefully before using it in production. You
can use the `.check_call()` method described below to determine if the
subroutine call is legal.
`.subroutine()` invokes VCL failure (See [ERRORS](#errors)) if:
- The rules for `n` and `select` indicate failure.
- No subroutine was set with the `sub` parameter in `.add()`.
- The subroutine is invoked with `call`, but the call is not legal in
the invocation context, for the reasons given above.
Example:
# Due to the use of resp.http.*, this subroutine may only be invoked
# in vcl_deliver or vcl_synth, as documented in vcl-var(7). Note
# that subroutine definitions must appear before vcl_init to
# permitted for the sub parameter in .add().
sub resp_sub {
set resp.http.Call-Me = "but only in deliver or synth";
}
sub vcl_init {
new myset = re2.set();
myset.add("/foo", sub=resp_sub);
myset.add("/foo/bar", sub=some_other_sub);
# ...
}
sub vcl_deliver {
if (myset.match(req.url)) {
call myset.subroutine(select=FIRST);
}
}
#### BOOL xset.check\_call(INT n, ENUM select)
......@@ -1540,7 +1563,33 @@ XXX ...
ENUM {FIRST, LAST, UNIQUE} select=UNIQUE
)
XXX ...
Returns `true` iff the subroutine returned by `.subroutine()` for the
element of the set indicated by `n` and `select` may be invoked legally
in the current context. The conditions for legal invocation are
documented for `.subroutine()` above.
`.check_call()` never invokes VCL failure, but rather returns `false`
under conditions for which the use of `.subroutine()` would invoke VCL
failure. In that case, a message is emitted to the Vanrish log using the
`Notice` tag (the same message that would appear with the `VCL_Error`
tag if the subroutine were called).
`Notice` messages in the log produced by this VMOD are always prefixed
with the string `vmod_re2:`.
Example:
# Assume that myset is declared as in the example above.
sub vcl_deliver {
if (myset.match(req.url)) {
if (myset.check_call(select=FIRST)) {
call myset.subroutine(select=FIRST);
}
else {
call do_if_resp_sub_is_illegal;
}
}
}
#### BOOL xset.saved(ENUM which, INT n, ENUM select)
......@@ -1564,13 +1613,13 @@ In other words, `.saved()` returns true:
- for `which=BE` if a backend was stored with the `backend` attribute.
- for `which=INT` if an integer was stored with the `integer`
attribute.
- for `which=SUB` if an integer was stored with the `sub` attribute.
The default value of `which` is `REGEX`.
The pattern in the set is identified by `n` and `select` according to
the rules given above. `.saved()` fails, returning false with a
`VCL_Error` message in the log, if the values of `n` or `select` are
invalid.
the rules given above. `.saved()` invokes VCL failure if the values of
`n` or `select` are invalid (see [ERRORS](#errors)).
Example:
......@@ -1666,8 +1715,8 @@ special characters. This function has a purpose similar to a `\Q..\E`
sequence within a regex, or the `literal=true` setting in a regex
constructor.
The function fails and returns `fallback` if there is insufficient
workspace for the return string.
The function invokes VCL failure if there is insufficient workspace for
the return string (see [ERRORS](#errors)).
Example:
......@@ -1683,14 +1732,70 @@ Example:
std.log("Using VMOD re2 version: " + re2.version());
### ERRORS
Functions and methods of the VMOD may invoke VCL failure under
unrecoverable error conditions. The effects of VCL failure depend on the
VCL subroutine in which it takes place:
- If invoked during `vcl_init`, then the VCL load fails, and an error
message is returned over the CLI (reported by `varnishadm(1)`).
- If invoked during any other subroutine besides `vcl_synth`, then an
error message is recorded in the log with the `VCL_Error` tag,
further processing is aborted immediately, and a response with
status 503 (Service Not Available) is returned with the reason
string "VCL failed".
- If invoked during `vcl_synth`, then further processing is aborted,
the error message is logged with `VCL_Error`, and the client
connection is immediately closed -- the client receives no response.
Errors that lead to VCL failure include:
- Any regex compilation failure.
- Out of workspace errors (see [LIMITATIONS](#limitations)).
- Failures reported by the RE2 library for: matches, backrefs,
namedrefs, the rewrite operations (sub, suball and extract), the
`.cost()` function or method, and the `.quotemeta()` function. The
VMOD detects most common errors that would lead to library errors,
and invokes VCL failure in such cases without calling the library.
But library errors may happen under conditions such as out of
memory.
- Functions and methods that require a previous successful match when
there was no prior match, or when the previous match was
unsuccessful. These include backrefs, namedrefs, and the data
retrieval methods for set objects.
- Any of the following parameters are undefined, for example when set
from an unset header: fallbacks; patterns for the regex functions
(which are compiled at runtime); the text and rewrite parameters for
rewrite operations; the name parameter for namedrefs.
- The name parameter for namedrefs is the empty string.
- Backref number is out of range (greater than the number of backrefs
in the pattern).
- Backref or namedref attempted when the `never_capture` option was
set to `true` for the pattern.
- For set objects:
- Numeric index (parameter `n`) is out of range (greater than the
number of patterns in the set).
- Use of `select=UNIQUE` after more than one pattern was matched.
The `.nmatches()` can be used to check for this condition, to
avoid VCL failure -- `UNIQUE` will fail in `.namtches()` \> 1.
- Retrieval of data from a set (such as a string, backend etc) by
numeric index (`n`) or "associatively" (after a match) when no
such object was saved for the corresponding pattern. Use the
`.saved()` and `.check_call()` methods to check for this.
- Calling the subroutine returned by `.subrooutine()` may be
illegal, if it is not permitted in the subroutine from which it
is called, or if it would lead to recursive calls. Use the
`.check_call()` method to check for this.
### REQUIREMENTS
The VMOD requires the Varnish master branch. See the source repository
for versions of the VMOD that are compatible with other Varnish
versions.
The VMOD requires Varnish since version 6.6, or the master branch. See
the source repository for versions of the VMOD that are compatible with
other Varnish versions.
It requires the RE2 library, and has been tested against RE2 versions
since 2015-06-01 (through 2020-11-01 at the time of writing).
since 2015-06-01 (through 2021-04-01 at the time of writing).
If the VMOD is built against versions of RE2 since 2017-12-01, it uses a
version of the set match operation that reports out-of-memory conditions
......
src/vmod_re2.vcc
View file @ 6d20f3a9
......@@ -46,8 +46,7 @@ SYNOPSIS
# set object interface
new OBJECT = re2.set([ENUM anchor] [, <regex options>])
VOID <obj>.add(STRING [, BOOL save] [, BOOL never_capture] [, STRING string]
[, BACKEND backend] [, INT integer])
VOID <obj>.compile()
[, BACKEND backend] [, INT integer] [,SUB sub])
BOOL <obj>.match(STRING)
INT <obj>.nmatches()
BOOL <obj>.matched(INT)
......@@ -55,13 +54,15 @@ SYNOPSIS
STRING <obj>.string([INT n,] [ENUM select])
BACKEND <obj>.backend([INT n,] [ENUM select])
INT <obj>.integer([INT n] [, ENUM select])
SUB <obj>.subroutine([INT n] [, ENUM select])
BOOL <obj>.check_call([INT n] [, ENUM select])
STRING <obj>.sub(STRING text, STRING rewrite [, INT n]
[, ENUM select])
STRING <obj>.suball(STRING text, STRING rewrite [, INT n]
[, ENUM select])
STRING <obj>.extract(STRING text, STRING rewrite [, INT n]
[, ENUM select])
BOOL <obj>.saved([ENUM {REGEX, STR, BE, INT} which] [, INT n]
BOOL <obj>.saved([ENUM {REGEX, STR, BE, INT, SUB} which] [, INT n]
[, ENUM select])
VOID <obj>.hdr_filter(HTTP [, BOOL])
......@@ -141,7 +142,6 @@ example::
myset.add("foo"); # Pattern 1
myset.add("bar"); # Pattern 2
myset.add("baz"); # Pattern 3
myset.compile();
}
``myset.match(<string>)`` can now be used to match a string against
......@@ -195,14 +195,12 @@ strings or backends with the patterns added to the set with the
prefix.add("/bar", string="www.domain2.com");
prefix.add("/baz", string="www.domain3.com");
prefix.add("/quux", string="www.domain4.com");
prefix.compile();
new appmatcher = re2.set(anchor=start);
appmatcher.add("/foo", backend=app1);
appmatcher.add("/bar", backend=app2);
appmatcher.add("/baz", backend=app3);
appmatcher.add("/quux", backend=app4);
appmatcher.compile();
}
After a successful match, the string or backend associated with the
......@@ -285,11 +283,13 @@ constructor apply to all of the patterns in the resulting alternation.
including newline. Otherwise, ``.`` never matches newline. Default is
**false**.
``never_capture``
If true, parentheses in a pattern are interpreted as non-capturing, and all
invocations of the ``backref`` and ``namedref`` methods or functions will
fail, including ``backref(0)`` after a successful match. Default is **false**,
except for set objects, for which ``never_capture`` is always true (and cannot
be changed), since back references are not possible with sets.
If true, parentheses in a pattern are interpreted as non-capturing,
and all invocations of the ``backref`` and ``namedref`` methods or
functions will lead to VCL faillure (see `ERRORS`_), including
``backref(0)`` after a successful match. Default is **false**,
except for set objects, for which ``never_capture`` is always true
(and cannot be changed), since back references are not possible with
sets.
``case_sensitive``
If true, matches are case-sensitive. A pattern can override this option with
the ``(?i)`` flag, unless ``posix_syntax`` is true. Default is **true**.
......@@ -346,9 +346,9 @@ $Method STRING .backref(INT ref, STRING fallback = "**BACKREF METHOD FAILED**")
Returns the `nth` captured subexpression from the most recent
successful call of the ``.match()`` method for this object in the same
client or backend, context, or a fallback string in case the capture
client or backend context, or a fallback string in case the capture
fails. Backref 0 indicates the entire matched string. Thus this
function behaves like the ``\n`` in the native VCL functions
function behaves like the ``\n`` notation in the native VCL functions
``regsub`` and ``regsuball``, and the ``$1``, ``$2`` ... variables in
Perl.
......@@ -360,17 +360,16 @@ subroutines -- the backend context -- refers back to a previous
the other VCL subroutines -- the client context -- refers back to a
``.match()`` in the same client context.
After unsuccessful matches, the ``fallback`` string is returned for
any call to ``.backref()``. The default value of ``fallback`` is
``"**BACKREF METHOD FAILED**"``. ``.backref()`` always fails after a
failed match, even if ``.match()`` had been called successfully before
the failure.
``.backref()`` may also return ``fallback`` after a successful match,
if no captured group in the matching string corresponds to the backref
``.backref()`` may return ``fallback`` after a successful match, if no
captured group in the matching string corresponds to the backref
number. For example, when the pattern ``(a|(b))c`` matches the string
``ac``, there is no backref 2, since nothing matches ``b`` in the
string.
string. The default value of ``fallback`` is ``"**BACKREF METHOD
FAILED**"``, but you may set another value (such as the empty string).
After unsuccessful matches, ``.backref()`` invokes VCL failure (see
`ERRORS`_). ``.backref()`` always fails after a failed match, even if
``.match()`` had been called successfully before the failure.
The VCL infix operators ``~`` and ``!~`` do not affect this method,
nor do the functions ``regsub`` or ``regsuball``. Nor is it affected
......@@ -378,10 +377,9 @@ by the matches performed by any other method or function in this VMOD
(such as the ``sub()``, ``suball()`` or ``extract()`` methods or
functions, or the ``set`` object's ``.match()`` method).
``.backref()`` fails, returning ``fallback`` and writing an error
message to the Varnish log with the ``VCL_Error`` tag, under the
following conditions (even if a previous match was successful and a
substring could have been captured):
``.backref()`` invokes VCL failure under the following conditions,
even if a previous match was successful and a substring could have
been captured (see `ERRORS`_):
* The ``fallback`` string is undefined, for example if set from an unset
header variable.
......@@ -404,7 +402,7 @@ $Method STRING .namedref(STRING name,
Returns the captured subexpression designated by ``name`` from the
most recent successful call to ``.match()`` in the current context
(client or backend), or ``fallback`` in case of failure.
(client or backend).
Named capturing groups are written in RE2 as: ``(?P<name>re)``. (Note
that this syntax with ``P``, inspired by Python, differs from the
......@@ -416,17 +414,15 @@ Note that a named capturing group can also be referenced as a numbered
group. So in the previous example, ``.backref(1)`` also returns
``baz``.
``fallback`` is returned when ``.namedref()`` is called after an
unsuccessful match. The default fallback is ``"**NAMEDREF METHOD
FAILED**"``.
``fallback`` is returned when the named reference did not match. The
default fallback is ``"**NAMEDREF METHOD FAILED**"``.
Like ``.backref()``, ``.namedref()`` is not affected by native VCL
regex operations, nor by any other matches performed by methods or
functions of the VMOD, except for a prior ``.match()`` for the same
object.
``.namedref()`` fails, returning ``fallback`` and logging a
``VCL_Error`` message, if:
``.namedref()`` invokes VCL failure (see `ERRORS`_) if:
* The ``fallback`` string is undefined.
* ``name`` is undefined or the empty string.
......@@ -460,8 +456,7 @@ native function ``regsub()``.
``fallback`` is returned if the pattern does not match ``text``. The
default fallback is ``"**SUB METHOD FAILED**"``.
``.sub()`` fails, returning ``fallback`` and logging a ``VCL_Error``
message, if:
``.sub()`` invokes VCL failure (see `ERRORS`_) if:
* Any of ``text``, ``rewrite`` or ``fallback`` are undefined.
* There is insufficient workspace for the rewritten string.
......@@ -559,8 +554,8 @@ Like the ``regex.match()`` method, return ``true`` if ``pattern``
matches ``subject``, where ``pattern`` is compiled with the given
options (or default options) on each invocation.
If ``pattern`` fails to compile, then an error message is logged with
the ``VCL_Error`` tag, and ``false`` is returned.
If ``pattern`` fails to compile, then VCL failure is invoked (see
`ERRORS`_).
Example::
......@@ -578,19 +573,19 @@ backend context, or a fallback string if the capture fails. The
default ``fallback`` is ``"**BACKREF FUNCTION FAILED**"``.
Similarly to the ``regex.backref()`` method, ``fallback`` is returned
after any failed invocation of the ``match()`` function, or if there
is no captured group corresponding to the backref number. The function
is not affected by native VCL regex operations, or any other method or
function of the VMOD except for the ``match()`` function.
if there is no captured group corresponding to the backref number. The
function is not affected by native VCL regex operations, or any other
method or function of the VMOD except for the ``match()`` function.
The function fails, returning ``fallback`` and logging a ``VCL_Error``
message, under the same conditions as the corresponding method:
The function invokes VCL failure under the same conditions as the
corresponding method (see `ERRORS`_):
* ``fallback`` is undefined.
* ``never_capture`` was true in the previous invocation of the ``match()``
function.
* ``ref`` is out of range.
* The ``match()`` function was never called in this context.
* The ``match()`` function was never called in this context, or if the
previous ``match()`` call failed (returned ``false``).
* The pattern failed to compile for the previous ``match()`` call.
* There is insufficient workspace for the captured subexpression.
......@@ -607,20 +602,17 @@ $Function STRING namedref(PRIV_TASK, STRING name,
Returns the captured subexpression designated by ``name`` from the
most recent successful call to the ``match()`` function in the current
context, or ``fallback`` in case of failure. The default fallback is
``"**NAMEDREF FUNCTION FAILED**"``.
context, or ``fallback`` if the corresponding group did not match. The
default fallback is ``"**NAMEDREF FUNCTION FAILED**"``.
The function returns ``fallback`` when the previous invocation of the
``match()`` function failed, and is only affected by use of the
``match()`` function. The function fails, returning ``fallback`` and
logging a ``VCL_Error`` message, under the same conditions as the
corresponding method:
The function invokes VCL failure under the same conditions as the
corresponding method (see `ERRORS`_):
* ``fallback`` is undefined.
* ``name`` is undefined or the empty string.
* The ``never_capture`` option was set to ``true``.
* There is no such named group.
* ``match()`` was not called in this context.
* ``match()`` was not called in this context, or the previous call failed.
* The pattern failed to compile for the previous ``match()`` call.
* There is insufficient workspace for the captured expression.
......@@ -646,8 +638,7 @@ groups from the pattern.
``fallback`` is returned if the pattern does not match ``text``. The
default fallback is ``"**SUB FUNCTION FAILED**"``.
``sub()`` fails, returning ``fallback`` and logging a ``VCL_Error``
message, if:
``sub()`` invokes VCL failure (see `ERRORS`_) if:
* ``pattern`` cannot be compiled.
* Any of ``text``, ``rewrite`` or ``fallback`` are undefined.
......@@ -716,7 +707,7 @@ Like the ``.cost()`` method above, return a numeric measurement > 0
from the RE2 library for ``pattern`` with the given options. More
complex regexen have a higher cost than less complex regexen.
Fails and returns -1 if ``pattern`` cannot be compiled.
Invokes VCL failure if ``pattern`` cannot be compiled (see `ERRORS`_).
Example::
......@@ -752,20 +743,28 @@ against the resulting composed pattern. However, the ``never_capture``
option cannot be set, and is always implicitly true, since backrefs
and namedrefs are not possible with sets.
Sets are compiled automatically when ``vcl_init`` finishes (or when
the deprecated ``.compile()`` method is called). Compilation fails if
any of the added patterns cannot be compiled, or if no patterns were
added to the set. It may also fail if the ``max_mem`` setting is not
large enough for the composed pattern. In that case, the VCL load will
fail with an error message (then consider a larger value for
``max_mem`` in the set constructor).
Example::
sub vcl_init {
# Initialize a regex set for partial matches
# Initialize a regex set for partial matches
# with default options
new foo = re2.set();
new foo = re2.set();
# Initialize a regex set for case insensitive matches
# Initialize a regex set for case insensitive matches
# with anchors on both ends (^ and $).
new bar = re2.set(anchor=both, case_sensitive=false);
new bar = re2.set(anchor=both, case_sensitive=false);
# Initialize a regex set using POSIX syntax, but allowing
# Initialize a regex set using POSIX syntax, but allowing
# Perl character classes, and anchoring at the left (^).
new baz = re2.set(anchor=start, posix_syntax=true,
new baz = re2.set(anchor=start, posix_syntax=true,
perl_classes=true);
}
......@@ -776,12 +775,13 @@ Add the given pattern to the set. If the pattern is invalid,
``.add()`` fails, and the VCL will fail to load, with an error message
describing the problem.
If values for the ``string``, ``backend`` and/or ``integer``
If values for the ``string``, ``backend``, ``integer`` and/or ``sub``
parameters are provided, then these values can be retrieved with the
``.string()``, ``.backend()`` and ``.integer()`` methods,
respectively, as described below. This makes it possible to associate
data with the added pattern after it matches successfully. By default
the pattern is not associated with any such value.
``.string()``, ``.backend()``, ``.integer()`` and ``.subroutine()``
methods, respectively, as described below. This makes it possible to
associate data with the added pattern after it matches
successfully. By default the pattern is not associated with any such
value.
If ``save`` is true, then the given pattern is compiled and saved as a
``regex`` object, just as if the ``regex`` constructor described above
......@@ -803,13 +803,10 @@ be set to false for the individual regex, even though it is implicitly
set to true for the full set object (default is false).
``.add()`` MUST be called in ``vcl_init``, and MAY NOT be called after
``.compile()``. If ``.add()`` is called in any other subroutine, an
error message with ``VCL_Error`` is logged, and the call has no
effect. If it is called in ``vcl_init`` after ``.compile()``, then the
VCL load will fail with an error message.
In other words, add all patterns to the set in ``vcl_init``, and
finally call ``.compile()`` when you're done.
``.compile()``. VCL failure is invoked if ``.add()`` is called in any
other subroutine (see `ERRORS`_). If it is called in ``vcl_init``
after ``.compile()``, then the VCL load will fail with an error
message. Note that ``.compile()`` is now unneccessary and deprecated.
When the ``.matched(INT)`` method is called after a successful match,
the numbering corresponds to the order in which patterns were added.
......@@ -827,7 +824,6 @@ Example::
hostmatcher.add("www.domain1.com");
hostmatcher.add("www.domain2.com");
hostmatcher.add("www.domain3.com");
hostmatcher.compile();
}
# See the documentation of the .string() and .backend() methods
......@@ -835,21 +831,20 @@ Example::
$Method VOID .compile()
**This method is deprecated**, and will be removed in a future
version. ``.compile()`` may be omitted, since compilation now happens
automatically when ``vcl_init`` finishes.
Compile the compound pattern represented by the set -- an alternation
of all patterns added by ``.add()``.
``.compile()`` fails if no patterns were added to the set. It may also
fail if the ``max_mem`` setting is not large enough for the composed
pattern. In that case, the VCL load will fail with an error message
(then consider a larger value for ``max_mem`` in the set constructor).
Compilation may fail for any of the reasons described for automatic
compilation of set objects as described above.
``.compile()`` MUST be called in ``vcl_init``, and MAY NOT be called
more than once for a set object. If it is called in any other
subroutine, a ``VCL_Error`` message is logged, and the call has no
effect. If it is called a second time in ``vcl_init``, the VCL load
will fail.
See above for examples.
more than once for a set object. VCL failure is invoked if it is
called in any other subroutine. If it is called a second time in
``vcl_init``, the VCL load will fail.
$Method BOOL .match(STRING)
......@@ -862,19 +857,13 @@ and match the given string. These can be determined after a successful
match using the ``.matched(INT)`` and ``.nmatches()`` methods
described below.
``.match()`` MUST be called after ``.compile()``; otherwise the match
always fails.
A match may also fail (returning ``false``) if the internal memory
A match may also fail (leading to VCL failure) if the internal memory
limit imposed by the ``max_mem`` parameter in the constructor is
exceeded. (With the default value of ``max_mem``, this ordinarily
requires very large patterns and/or a very large string to be
matched.) Since about version 2017-12-01, the RE2 library reports
this condition; if so, the VMOD writes a ``VCL_Error`` message in the
log if it happens, except during ``vcl_init``, in which case the VCL
load fails with the error message. If matches fail due to the
out-of-memory condition, increase the ``max_mem`` parameter in the
constructor.
this condition. If matches fail due to the out-of-memory condition,
increase the ``max_mem`` parameter in the constructor.
Example::
......@@ -894,7 +883,7 @@ for the same object in the same client or backend context. It always
returns ``false``, for every value of the parameter, if it is called
after an unsuccessful match (``.match()`` returned ``false``).
``.matched()`` fails and returns ``false`` if:
``.matched()`` invokes VCL failure (see `ERRORS`_) if:
* The ``.match()`` method was not called for this object in the same
client or backend scope.
......@@ -902,13 +891,6 @@ after an unsuccessful match (``.match()`` returned ``false``).
* The integer parameter is out of range; that is, if it is less than 1
or greater than the number of patterns added to the set.
On failure, the method writes an error message to the log with the tag
``VCL_Error``; if it fails during ``vcl_init``, then the VCL load
fails with the error message. In any other VCL subroutine, the method
returns ``false`` on failure and processing continues; since ``false``
is a legitimate return value, you should consider monitoring the log
for the error messages.
Example::
if (hostmatcher.match(req.http.Host)) {
......@@ -931,12 +913,7 @@ backend context. The method always returns 0 after an unsuccessful
match (``.match()`` returned ``false``).
If ``.match()`` was not called for this object in the same client or
backend scope, ``.nmatches()`` fails and returns 0, writing an error
message with ``VCL_Error`` to the log. If this happens in
``vcl_init``, the VCL load fails with the error message. As with
``.matched()``, ``.nmatches()`` returns a legitimate value and VCL
processing continues when it fails in any other subroutine, so you
should monitor the log for the error messages.
backend scope, ``.nmatches()`` invokes VCL failure (see `ERRORS`_).
Example::
......@@ -964,8 +941,7 @@ the patterns that matched, the first or last one added via the
``.add()`` method is chosen, and the number for that pattern is
returned.
``.which()`` fails, returning 0 with a ``VCL_Error`` message in the log,
if:
``.which()`` invokes VCL failure (see `ERRORS`_) if:
* ``.match()`` was not called for the set in the current client or
backend transaction, or if the previous call returned ``false``.
......@@ -1045,8 +1021,7 @@ For the pattern selected by these rules, return the string that was
set with the ``string`` parameter in the ``.add()`` method that added
the pattern to the set.
``.string()`` fails, returning NULL with an a ``VCL_Error`` message in
the log, if:
``.string()`` invokes VCL failure (see `ERRORS`_) if:
* The values of ``n`` and ``select`` are invalid:
......@@ -1059,10 +1034,12 @@ the log, if:
* ``n`` <= 0 and the ``select`` ENUM is ``UNIQUE`` (or default), but
more than one pattern matched in the previous ``.match()`` call.
This can be avoided by checking for ``.nmatches() == 1``.
* No string was associated with the pattern selected by ``n`` and
``select``; that is, the ``string`` parameter was not set in the
``.add()`` call that added the pattern.
``.add()`` call that added the pattern. This can be avoided by
checking the ``.saved()`` method (see below).
Examples::
......@@ -1142,10 +1119,10 @@ The rules for selecting a pattern from the set and its associated
backend based on ``n`` and ``select`` are the same as described above
for ``.string()``.
``.backend()`` fails, returning NULL with an a ``VCL_Error`` message
in the log, under the same conditions described for ``.string()``
above -- ``n`` and ``select`` are invalid, or no backend was
associated with the selected pattern with the ``.add()`` method.
``.backend()`` invokes VCL failure under the same conditions described
for ``.string()`` above -- ``n`` and ``select`` are invalid, or no
backend was associated with the selected pattern with the ``.add()``
method (see `ERRORS`_).
Example::
......@@ -1190,18 +1167,7 @@ for ``.string()``.
``.integer()`` invokes VCL failure under the same error conditions
described for ``.string()`` above -- ``n`` and ``select`` are invalid,
or no integer was associated with the selected pattern with the
``.add()`` method.
Note that VCL failure differs from the failure mode for ``.string()``
and ``.backend()``, since there is no distinguished "error" value that
could be returned as the INT. VCL failure has the same effect as if
``return(fail)`` were called from a VCL subroutine; usually, control
directs immediately to ``vcl_synth``, with the response status set to
503, and the response reason set to "VCL failed".
You can avoid that, for example, by testing if ``.nmatches()==1``
after calling ``.match()``, if you need to ensure that calling
``.integer(select=UNIQUE)`` will not fail.
``.add()`` method (see `ERRORS`_).
Example::
......@@ -1259,8 +1225,7 @@ to the rules given above. If an internal ``regex`` object was saved
for that pattern, then the result of the ``.sub()`` method invoked on
that object is returned.
``.sub()`` fails, returning NULL with a ``VCL_Error`` message in the
log, if:
``.sub()`` invokes VCL failkure (see `ERRORS`_) if:
* The values of ``n`` and ``select`` are invalid, according to the
rules given above.
......@@ -1399,11 +1364,92 @@ Example::
$Method SUB .subroutine(INT n=0, ENUM {FIRST, LAST, UNIQUE} select=UNIQUE)
XXX ...
Returns the subroutine set by the ``sub`` parameter for the element of
the set indicated by ``n`` and ``select``, according to the rules
given above. The subroutine may be invoked with VCL ``call``.
**Note**: you must ensure that the subroutine may invoked legally in
the context in which it is called. This means that:
* The subroutine may only refer to VCL elements that are legal in the
invocation context. For example, if the subroutine only refers to
headers in ``req.http.*``, then it may be called in ``vcl_recv``,
but not if it refers to any header in ``resp.http.*``. See
``vcl-var(7)`` for the specification of which VCL variables may be
used in which contexts.
* Recursive subroutine calls are not permitted in VCL. The subroutine
invocation may not appear anywhere in its own call stack.
For standard subroutine invocations with ``call``, the VCL compiler
checks these conditions and issues a compile-time error if either one
is violated. This is not possible with invocations using
``.subroutine()``; the error can only be determined at runtime. So it
is advisable to test the use of ``.subroutine()`` carefully before
using it in production. You can use the ``.check_call()`` method
described below to determine if the subroutine call is legal.
``.subroutine()`` invokes VCL failure (See `ERRORS`_) if:
* The rules for ``n`` and ``select`` indicate failure.
* No subroutine was set with the ``sub`` parameter in ``.add()``.
* The subroutine is invoked with ``call``, but the call is not legal
in the invocation context, for the reasons given above.
Example::
# Due to the use of resp.http.*, this subroutine may only be invoked
# in vcl_deliver or vcl_synth, as documented in vcl-var(7). Note
# that subroutine definitions must appear before vcl_init to
# permitted for the sub parameter in .add().
sub resp_sub {
set resp.http.Call-Me = "but only in deliver or synth";
}
sub vcl_init {
new myset = re2.set();
myset.add("/foo", sub=resp_sub);
myset.add("/foo/bar", sub=some_other_sub);
# ...
}
sub vcl_deliver {
if (myset.match(req.url)) {
call myset.subroutine(select=FIRST);
}
}
$Method BOOL .check_call(INT n=0, ENUM {FIRST, LAST, UNIQUE} select=UNIQUE)
XXX ...
Returns ``true`` iff the subroutine returned by ``.subroutine()`` for
the element of the set indicated by ``n`` and ``select`` may be
invoked legally in the current context. The conditions for legal
invocation are documented for ``.subroutine()`` above.
``.check_call()`` never invokes VCL failure, but rather returns
``false`` under conditions for which the use of ``.subroutine()``
would invoke VCL failure. In that case, a message is emitted to the
Vanrish log using the ``Notice`` tag (the same message that would
appear with the ``VCL_Error`` tag if the subroutine were called).
``Notice`` messages in the log produced by this VMOD are always
prefixed with the string ``vmod_re2: ``.
Example::
# Assume that myset is declared as in the example above.
sub vcl_deliver {
if (myset.match(req.url)) {
if (myset.check_call(select=FIRST)) {
call myset.subroutine(select=FIRST);
}
else {
call do_if_resp_sub_is_illegal;
}
}
}
$Method BOOL .saved(ENUM {REGEX, STR, BE, INT, SUB} which=REGEX, INT n=0,
ENUM {FIRST, LAST, UNIQUE} select=UNIQUE)
......@@ -1427,12 +1473,14 @@ In other words, ``.saved()`` returns true:
* for ``which=INT`` if an integer was stored with the ``integer``
attribute.
* for ``which=SUB`` if an integer was stored with the ``sub``
attribute.
The default value of ``which`` is ``REGEX``.
The pattern in the set is identified by ``n`` and ``select`` according
to the rules given above. ``.saved()`` fails, returning false with a
``VCL_Error`` message in the log, if the values of ``n`` or ``select``
are invalid.
to the rules given above. ``.saved()`` invokes VCL failure if the
values of ``n`` or ``select`` are invalid (see `ERRORS`_).
Example::
......@@ -1528,8 +1576,8 @@ any special characters. This function has a purpose similar to a
``\Q..\E`` sequence within a regex, or the ``literal=true`` setting in
a regex constructor.
The function fails and returns ``fallback`` if there is insufficient
workspace for the return string.
The function invokes VCL failure if there is insufficient workspace
for the return string (see `ERRORS`_).
Example::
......@@ -1545,15 +1593,87 @@ Example::
std.log("Using VMOD re2 version: " + re2.version());
ERRORS
======
Functions and methods of the VMOD may invoke VCL failure under
unrecoverable error conditions. The effects of VCL failure depend on
the VCL subroutine in which it takes place:
* If invoked during ``vcl_init``, then the VCL load fails, and an
error message is returned over the CLI (reported by
``varnishadm(1)``).
* If invoked during any other subroutine besides ``vcl_synth``, then
an error message is recorded in the log with the ``VCL_Error`` tag,
further processing is aborted immediately, and a response with
status 503 (Service Not Available) is returned with the reason
string "VCL failed".
* If invoked during ``vcl_synth``, then further processing is aborted,
the error message is logged with ``VCL_Error``, and the client
connection is immediately closed -- the client receives no response.
Errors that lead to VCL failure include:
* Any regex compilation failure.
* Out of workspace errors (see `LIMITATIONS`_).
* Failures reported by the RE2 library for: matches, backrefs,
namedrefs, the rewrite operations (sub, suball and extract), the
``.cost()`` function or method, and the ``.quotemeta()``
function. The VMOD detects most common errors that would lead to
library errors, and invokes VCL failure in such cases without
calling the library. But library errors may happen under conditions
such as out of memory.
* Functions and methods that require a previous successful match when
there was no prior match, or when the previous match was
unsuccessful. These include backrefs, namedrefs, and the data
retrieval methods for set objects.
* Any of the following parameters are undefined, for example when
set from an unset header: fallbacks; patterns for the regex functions
(which are compiled at runtime); the text and rewrite parameters
for rewrite operations; the name parameter for namedrefs.
* The name parameter for namedrefs is the empty string.
* Backref number is out of range (greater than the number of backrefs
in the pattern).
* Backref or namedref attempted when the ``never_capture`` option was
set to ``true`` for the pattern.
* For set objects:
* Numeric index (parameter ``n``) is out of range (greater than the
number of patterns in the set).
* Use of ``select=UNIQUE`` after more than one pattern was matched.
The ``.nmatches()`` can be used to check for this condition, to
avoid VCL failure -- ``UNIQUE`` will fail in ``.namtches()`` > 1.
* Retrieval of data from a set (such as a string, backend etc) by
numeric index (``n``) or "associatively" (after a match) when no
such object was saved for the corresponding pattern. Use the
``.saved()`` and ``.check_call()`` methods to check for this.
* Calling the subroutine returned by ``.subrooutine()`` may be
illegal, if it is not permitted in the subroutine from which it is
called, or if it would lead to recursive calls. Use the
``.check_call()`` method to check for this.
REQUIREMENTS
============
The VMOD requires the Varnish master branch. See the source repository
for versions of the VMOD that are compatible with other Varnish
versions.
The VMOD requires Varnish since version 6.6, or the master branch. See
the source repository for versions of the VMOD that are compatible
with other Varnish versions.
It requires the RE2 library, and has been tested against RE2 versions
since 2015-06-01 (through 2020-11-01 at the time of writing).
since 2015-06-01 (through 2021-04-01 at the time of writing).
If the VMOD is built against versions of RE2 since 2017-12-01, it uses
a version of the set match operation that reports out-of-memory
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment