But paced, because they may repeat in a semi-tight loop.

For high-throughput scenarios we might be collecting irrelevant
transactions, especially sessions that are never buffered, and
drastically increase the risk of overruns with long-running
sessions (typically when varnishd is behind a load balancer or
a TLS terminator that we can entrust with high client connection
reuse).

The -b and -c options have an effect on the output, which is way
too late for such a setup. These options primarily work with the
MSB backend and client markers but that is not enough to rule out
sessions since they are marked as client transactions. The same
goes for incomplete transactions for which we are lacking a Begin
tag, that would still be ruled out at the output stage.

We also need to collect irrelevant transactions for the sake of
grouping, because even when we do not wish to output them we have
to maintain the hierarchy.

Therefore, we can identify candidate transactions early and not
waste memory footprint and churn keeping track of transactions
that would have no effect besides increasing the risk of overruns.

Otherwise the combination of -E and -b only outputs backend requests.

we take the chance of the breaking change in
681c1199 to also add a VRT_CTX
argument to vmod_priv_fini_f. This allows a vmod_priv fini function,
for example, access to the request or simpler and better logging
(because the task's vsl buffer can be used where otherwise only
unbuffered vsl was possible).

    Implementation:

The meat of this commit really only is the change to the
vmod_priv_fini_f typedef and VRT_priv_fini().

All other changes are either to bundled vmods or the straight forward
infrastructure to make available a ctx to VRT_priv_fini(), which, for
client and backend context, is called via VCL_TaskLeave().

Consequently, it made sense to also change the signature
Req_Rollback() and Bereq_Rollback().

Instead of triggering a WRONG() panic, we now fail the VCL when any vmod
function with PRIV_TOP argument(s) is present in a sub called from
outside client context.

Note that the VCL failure happens not when the vmod function with the
PRIV_TOP argument is called, but rather when the containing SUB is
called. This is because we prepare PRIV_TOP arguments in the function
preamble.

Fixes #3498

This is how 08b642d7 should have looked
like.

and add vtc code to avoid it from happening again.

The mentioned commit lead to resolve=NOW modify the shard's parameter
PRIV_TASK when only parameters on the stack should have been modified.

We will adjust this to use shard_param_task_r() in a follow-up

Spotted thanks to Coverity, ref CID 1472212

Propagate upwards any workspace allocation failure of the
recursive shard_param_task_l() case.

We use shard_param_task_{r,l} on (struct sharddir *) and (struct
vmod_directors_shard_param *), so the id parameter is (const void *).
Yet we still want to make sure that we always retrieve the intended
PRIV_TASK, and we can by simply asserting that the vcl_name attribute
matches.

Note that a simple pointer comparison is sufficient here because,
unltimately, the PRIV_TASK vcl_name is always that of a vmod object,
assigned by the constructor.

This brings back the "who" parameter from
9ea2f29f, but used differently.

When a VUT is slow enough, it might very well be overrun while it is
scanning logs. For our built-in VUTs like varnishncsa or varnishlog
this can happen if writing the output can block waiting for IO ops
or when the output is piped to a slow consumer.

There's only one call site for vtx_dispatch() and it guarantees that len
is greater than zero.

Allow the user to log the request time in milliseconds through the new
format specifier: %{ms}T .
Signed-off-by: Asad Sajjad Ahmed <asadsa@varnish-software.com>

7da412fa at work

This removes logging from xid 0 and makes clearer what is happening

see previous commit: to hopefully help avoid the same oversight
with mylog()

seems to be a real benefit of FreeBSD 13 vtest over all the other
vtesters. Not even the compulsively obsessive Flexelint complained.

- use a custom fini function for all priv* tests
- unify naming to _fini
- move common code to macro

Mostly taken from #3472 to slim that PR

gc unreferenced macro

We lacked error handling in the call to ensure that VDP_END is being sent.

Now we return the first error seen, which can be the one from the last
resort VDP_END call.

Flexelint c2d45aef

Ref #3298

These tests had syntax errors besides the error handling actually being
tested, which triggered with future changes to VCC.

Originally from #3163

Requested by phk.

And get a glimpse at how we can simplify VMODs making use of regular
expressions without breaking existing VCL.

When they take a regular expression as an argument, they can benefit
from compile time checks and avoid PRIV_CALL or on-the-fly regex
compilation.

Now that we have a proper type for regular expressions, use it in the
VRT API where it makes sense. The VRT_re_{init,fini}() functions were
moved to VPI since they are here strictly to support libvcc.

Summary: in VRT we use VCL_REGEX but in VPI we use struct vre.

By introducing a proper REGEX type in VCL we can centralize the regular
expressions literals parsing. The type itself is currently not very
potent since VMODs aren't aware of it, but as a first step it allows
error messages to be improved at compile time.

We check for the next storage element under the boc mutex, but reference
the boc state in relation to it after releasing the lock.

Take a copy of the boc state to close this (potential) race.

Fixes #3335, hopefully

Follow-up to #3125 :

That PR added VDP_END, but always issued another VDP_bytes() call with
no data.

We now make it the responsibility of the object iterator to send
VDP_END and opportunistically try to issue it with the last bytes in
order to avoid the additional VDP_bytes() call with no data.

As noted by Martin in
https://github.com/varnishcache/varnish-cache/pull/3125#issuecomment-600620905
the advantage is that "in H/2 there is the need to send a frame with a
last bit set to mark the end of the stream" and in H/1 we could generate
the end chunk right after the last bit of data.

This change does not implement these optimizations yet, but prepares the
grounds.

Dridi noticed that my reasoning about this was flawed: One transaction
ending does not imply no other transactions coming.

Ref #3249