These tests had syntax errors besides the error handling actually being
tested, which triggered with future changes to VCC.

Originally from #3163

Requested by phk.

And get a glimpse at how we can simplify VMODs making use of regular
expressions without breaking existing VCL.

When they take a regular expression as an argument, they can benefit
from compile time checks and avoid PRIV_CALL or on-the-fly regex
compilation.

Now that we have a proper type for regular expressions, use it in the
VRT API where it makes sense. The VRT_re_{init,fini}() functions were
moved to VPI since they are here strictly to support libvcc.

Summary: in VRT we use VCL_REGEX but in VPI we use struct vre.

By introducing a proper REGEX type in VCL we can centralize the regular
expressions literals parsing. The type itself is currently not very
potent since VMODs aren't aware of it, but as a first step it allows
error messages to be improved at compile time.

We check for the next storage element under the boc mutex, but reference
the boc state in relation to it after releasing the lock.

Take a copy of the boc state to close this (potential) race.

Fixes #3335, hopefully

Follow-up to #3125 :

That PR added VDP_END, but always issued another VDP_bytes() call with
no data.

We now make it the responsibility of the object iterator to send
VDP_END and opportunistically try to issue it with the last bytes in
order to avoid the additional VDP_bytes() call with no data.

As noted by Martin in
https://github.com/varnishcache/varnish-cache/pull/3125#issuecomment-600620905
the advantage is that "in H/2 there is the need to send a frame with a
last bit set to mark the end of the stream" and in H/1 we could generate
the end chunk right after the last bit of data.

This change does not implement these optimizations yet, but prepares the
grounds.

Dridi noticed that my reasoning about this was flawed: One transaction
ending does not imply no other transactions coming.

Ref #3249

Ref #3249

This is the VTC interface for the "fail list" facility added in a
previous commit. Commands added:

* fail add <vxid> <tag> <regex>

  Add en entry to the fail list at this point in the test.

  From now on, logexpect failure is triggered if this entry matches

* fail clear

  Clear the fail list

Closes #3249

Ref #3249

We add a list of expects which, if matched, will trigger failure.

When we encounter an expect with skip_max == LE_FAIL, we put it onto the
"fail list". For skip_max == LE_CLEAR, we clear that list.

For every VSL line, all expects on the fail list are tested, if any.

When the list of (positive) expects has been completed, logexpect
continues testing the fail list until the transaction is complete. This
obviously does not work for raw grouping, so in this case the test
author needs to ensure that the fail list be empty when the logexpect is
expected to complete.

Ref #3249

See #3498 for an unresolved issue noticed on the way.

Ref: 6b3e26c7

I do not understand how this has slipped through.

Ref 6b3e26c7

Adjust our PRIV_TOP test to use VRT_priv_top_get() for read operation
and copy the saved value onto the top request's workspace if need to
(workspaces differ and the string to be saved is on the current
request's workspace).

Ref 06cf60a0

test the issue

This bug triggered when a PRIV_TOP was requested only from esi_level >
0 and used by a "sibling" ESI request, that is, another request on the
same ESI level or its descendants.

The bug was introduced by someone(tm) in
542bf9b8: I removed req =
req->top->topreq but continued to use the workspace from req->ws.

Fixes #3496

Use a per-instance PRIV_TOP for the oo based test

merge a sentence which I had laying around unpushed. I think it might
help explaining what is happening.

Fixes #3491

Based on anecdotal evidence that it is not often used, and the concern
of lock contention it might contribute to since such records are emitted
unbuffered for every single expiry event, it is now disabled by default.

ExpKill records are still available if needed for example for debugging
purposes, and are enabled in the 3 test cases where we use them. They
were also getting in the way of 1 test.

Closes #3475

I overlooked use of an uninialized mutex.

Fixes https://github.com/vtest/VTest/issues/26

Fix in next commit

This reverts commit f855fa46.

... where log order is undeterministic (I mean, not really, but from the
perspective of our tests it can not be determined).

These efforts lead nowhere. The problem needed to be solved at the root.

This reverts commit 05e79574.
This reverts commit b02c7843.

This was another great idea by Dridi, thank you!

We see cases in VSL where the order of some events can not be (is hard
to) fix. For these cases, we add "all of these have to match, but we do
not care about the order" semantics through "?" in the skip column of
"expect" lines.

Example:

        expect 0 =    Debug             1
        expect ? =    Debug             a
        expect ? =    Debug             b
        expect ? =    Debug             c
        expect 0 =    Debug             2

After a line has matched "1", lines matching a, b or c in any order must
follow and then a line matching "2".

In the log, an "alt" line is emitted whenever an alternative expect is
evaluated.

When an alternative matches, matching will continue with the same
"expect" as before.

Implementation:

The skip_max field of (struct logexp_test) was already used for LE_ANY
if not a non-negative integer.

We add LE_ALT and LE_SEEN. The former denotes an expect with
alternatives (that is, the instruction to look for alternatives after
this expect) and the latter denotes an LE_ALT that has been matched.
Obviously, LE_SEEN is internal and can not result from the logexpect
configuration.

When a match on an LE_ALT fails, we do not advance le->test, but rather
look forward in the expect list for any other altnertives and try them
until we either have a match or run out of alternatives.

pull out the match code

It might not make a difference, except for the disappearance of a magic
number.