When a session is continuously used, for example in a multi-tier setup,
it can take a very long time before an HTTP/1 session runs out of tasks
and unwinds back to the point where the worker might update its stats.

This phenomenon is amplified by thread_stats_rate, so it might take even
longer to get statistics published. This is famously observed with load
testing campaigns where a certain set of VSCs may not increase at all
until the very end of a load run.

This change makes it possible for HTTP/1 sessions to at least publish
their statistics into the pool every thread_stats_rate requests.

To avoid a potential performance cost, this behavior (technically
complying with the documentation) is guarded by a feature flag. The
flag is generic so this behavior may be extended to other busy loops,
present (if any) or future.

When gauges are incremented and decremented by different threads, there
is a chance that decrements are published first, provoking an underflow
of the VSC. Values with the MSB set are considered underflowed, adjusted
to zero instead.

If the DBG_VCLREL is in effect and there are unpushed pool stats, make
sure that some thread is waking up to perform the late push. This should
help with stability for VTC test cases using the DBG_VCLREL flag.

Often in test cases the siutation comes where multiple threads go idle at
the same time, and no new activity happens after that. This would
potentially create the situation where the last thread to go idle sees
that some thread is currently pushing stats to global, so it leaves its
stats contribution on the pool stats to avoid contention. But without new
activity in the future, the pool stats may stay in the pool accumulator
indefinitely.

This patch adds a check to see if there are unaccumulated statistics
lingering in the pool accumulator when a thread wakes after idling to
release its VCL reference. If there are, the thread takes on the task of
pushing the pool stats to the global stats. This should help with test
case stability, and potentially help with underflowed stats reported by
some users.

For a red-black tree insert in vrt_priv_dynamic(), we performed a lookup
and then an insert, resulting in basically the same tree traversal
twice.

We now prepare a new struct vrt_priv on the workspace and release it if
the respective priv already existed on the tree.

For a full workspace, we fall back to a lookup via
vrt_priv_dynamic_get().

polishes 06cf60a0

I should have taken more time for that merge.

just to clarify for the next commit.

VRT_priv_task() inserts a priv_task if it does not exist.

In some cases, it is more efficient to only insert a priv_task under
certain conditions. These require an API function to only return a
priv_task if it exists. This is VRT_priv_task_lookup().

A TODO comment has been added for a sensible performance optimization,
avoiding one VRBT traversal for the VRT_priv_task() case. I would
address this after this suggestion got accepted, if so.

as suggested by Dridi

Originally suggested by @rezan in
https://github.com/varnishcache/varnish-cache/pull/3408/commits/4caad2c1858afb399f75230916183d31f57fc908

With the previous commits, as the bereq uses the req workspace, we
cannot roll back the req without also rolling back the bereq.

As the use case is rather exotic, we spare the additional complications
and just outlaw rollbacks from vcl_pipe.

std.rollback(bereq) is already forbidden by the compiler because the
bereq symbol is not available in vcl_pipe {}.

We could, alternative to the runtime failure for std.rollback(req),
remove the req symbol from vcl_pipe {}, but that could break use of req
to vmod functions also. So we chose the more compatible route of
selectively failing rollback rather than making impossible other use
cases.

Also make sure we didn't overflow before entering vcl_pipe. This would
mean we have lost important connection headers.

instead of overriding ctx.ws spefically, just let the Req2Ctx settings
win

This also moves pipe logging to the client side where it belongs.

Previously bo->ws was used which led to workspace mixing. req->ws safely
outlives both the bo and req object when piping.

fixes a regression introduced by c1352b0b:

by not using a suckaddr, VTCP_myname() would also work on UDS, but none
of the VTCP was ever intended to (and there are other places where that
is asserted). So one could argue that 9837045a
was already wrong, but I do not want to deflect from my own mistake in
any way.

Fixes #3492

Refs #3134

exercise both IPv4 and IPv6

Coverity CID 1362617, CID 1362625, CID 1362626, CID 1362627

introduced with 1c19f3fe

CID 1471724

Ref #3490

Ref #3490

For 127.0.0.1, use "None", "${localhost}" or "${bind_addr}" as appropriate.

Instead of ${Xaddr}:${Xport}, use ${Xsock}

For some tests, we need to add a Host header to make them stable for
differing server addresses.

Ref #3490

to support IPv6 only hosts, we try to resolve "localhost" if resolving
"127.0.0.1" fails.

Whatever address we determine is stored in the ${localhost}
macro. Because of the different address:port vs. [address]:port
formats for IPv4 vs. IPv6, we store "${localhost}:0" /
"[${localhost}]:0" in the ${listen_addr} macro.

Likewise we unify the *_sock macros to ${Xaddr}:${Xport} (the colon
used to be missing) for IPv4 and [${Xaddr}]:${Xport} for IPv6.

We also save ${listen_addr} in default_listen_addr for use from within
the vtest code whenever we want to bind to or listen on an emphemeral
port.

Ref #3490

32 bytes are not enough to hold the colons of a full length IPv6
address.

- write to stderr
- output errno for EAI_SYSTEM

for "backend default none;"

Ref #3067

	Connection failed (fec0:: 19925): (null)

The's not an IPv6 address.

*This* is an IPv6 address:

	fec0::205

Before c1352b0b, the function asserted
that getsockname() succeed. We now restore this behaviour in a way which
Flexelint understands (we are returning the memory which we allocated).

When we reserve a session attribute, we now check that the reserved
space is of the expected size.