Since the removal of dry signals, pools will spin when they run out of
threads and increment MAIN.threads_limited at a very high rate. That
spike in CPU consumption will also have detrimental effects on useful
tasks.

This change introduces a 1s delay when the pool is saturated. This
allows to periodically exercise the watchdog check. We could wait until
the watchdog times out but we would also miss potential updates to the
thread_pool_watchdog parameter.

To avoid spurious increments of MAIN.threads_limited we only take task
submissions into account and ignore timeouts of the condvar.

Refs #2942
Refs #3531

it was already accepted on Solaris and NetBSD, now we have seen it on
Linux and I think it does not make sense to keep the exception for
Apple.

Fixes #3532 (hopefully)

Spotted by: Rasmus Villemoes

Spotted by: Rasmus Villemoes

Spotted by: Rasmus Villemoes

Spotted by: Rasmus Villemoes

like the sign is wrong, we should never get EINVAL if the deadline is
in the future.

Fix the test to make sense, and hope for the best (or a M1 :-)

Spotted by: Rasmus Villemoes

Spotted by:	Rasmus Villemoes

Move the description of this feature from user to reference.

In pool_herder(), we do as much work as possible outside the pool
mutex. In particular, we check if we need to breed more threads, then
we do more checks, and finally we aquire the lock and either go sleep
or increase the threads_limited counter if, at that point, lqueue is
non-zero.

Yet it could have increased since we last checked, and there is
nothing wrong with that, we will breed a new thread in the next
iteration of the loop.

Increase threads_limited only if we would not bread a new thread right
away.

Fixes #3531

to handle correctly use of the "gunzip" filter at esi_level > 0

Fixes #3529

The bo fields err_code and err_reason need to be reset on a retry otherwise the values are kept.

Fixes #3525

suggested by phk

7ec28f1a did not work with object
methods because we need a partial symbol table lookup to identify them,
otherwise the symbol kind we find is just SUB for the method's return
value.

Additions to the test case and vmod_debug by @slimhazard, thank you

Fixes #3521

For the next commit, we need a partial lookup without error.

Alternatively, we could also a an error mode.

Diff best viewed with -b option

- avoid the second call to VCC_SymbolGet(), unless necessary
- reorder from simple to complex

Ref 7ec28f1a

    sub vcl_synth {
        if (resp.status == 12404) {
            [...]       // this specific 404
        } else if (resp.status % 1000 == 404) {
            [...]       // all other 404's
        }
    }

that is,

	call vmod.returning_sub();

in VCL.

Background:

The existing

	call sub;

is special in that it instantiates "sub", if it does not exist
already, enabling use-before-definition semantics.

The doctrine of this change is to preserve this behaviour and to not
make existing, static calls any less efficient.

Implementation:

To support calling both literal SUBs as well as SUB expressions, we
peek into the symbol table to check if the called expression is a
function, based on the knowledge that, as of now, only functions can
have a non-literal SUB return. We also know that no other expression
can yield a SUB. So if peeking ahead tells us that a function follows,
we call into expression evaluation.

Otherwise, we expect a literal SUB as before.

Null check:

For dynamic SUB calls from vmods via VRT_call(), we require the SUB
argument be non-NULL. But we can not for call: To signal error with a
SUB return, a vmod function needs to be allowed to return NULL,
otherwise it would need to have a dummy return value available for the
VRT_fail() case. So we need to check for NULL in VGC.

Alternatives considered:

- We could make expression evaluation return SUB also for literal SUB
symbols, turning all calls into sub->func(ctx, ...) C statements. I
tried this path and it resulted in a change of behaviour with
cc_command="clang -g -O2 ...": Where, previously, sub code was
inlined, it would now generate an explicit C function call always,
despite the fact that the C function is known at compile time (because
all named VCL_SUB structs are).

So this option was dismissed for violating the doctrine.

- The null check could go to VPI, via a VPI_call(), basically identical
to VRT_call(), but checking for NULL. This option was dismissed because
it would foreclose the requirement to allow for SUB arguments in the
future.

Acknowledgements:

reza mentioned the idea of call SUB at one point, and I can not remember
if I had it before or not, so when in doubt, it was his idea.

Dridi helped with ideas on the implementation and spotted a bug.

To control the threshold used to warn.

As a recurring pitfall sometimes hard to identify, it deserves its own
notice. The accounting of variants reuses the worker's strangelove field
since it doesn't conflict where other parts of the code base use it.

and protect this pattern in the test-case.