It's one more indirection than vcl_recv because the same logic applies
to multiple cases, so we first have subs for the different conditions
and they all call the shared vcl_beresp_hitmiss sub.

The vcl_beresp_hitmiss sub returns on purpose, that doesn't change the
default VCL behavior and makes it more usable from VCL code:

    if (some user condition applies) {
            call vcl_beresp_hitmiss;
    }

No need to force all call sites to return(deliver) when the desired
outcome is explicit. Extracting this sub also enables VCL authors to
only tweak the hitmiss TTL in one place.

For better compliance, we might also introduce a vcl_beresp_hitpass sub
for some of the built-in cases where a hit-for-miss object might be less
appropriate. The compliance effort will however change some semantics
and is outside of the scope of this split.

There is a very slight breaking change in the sense that the host header
check is grouped with its normalization which swaps its order with the
PRI method check.

In practice that only means that if you have both an unattended PRI
request missing a host header, you'll get a 400 instead of a 405. You
have to get both wrong in the first place so I don't consider this a
concern.

Now that we know in advance the available suboutines in the built-in VCL
we can use that as the condition to create an append-able subroutine
when the vcl_ prefix is encountered.

This work is currently incomplete, at least because non-state built-in
subroutines are currently not listed when an unknwon subroutine is
encountered.

Also, instead of proper test coverage this currently uses b00000.vtc as
a strawman to show how to skip a specific processing.

This is a lookahead parsing to learn about built-in subroutines that are
not tied to a VCL state. Instead of maintaining a mapping of the other
subroutines the builtin.vcl file itself becomes authoritative.

I removed the else keyword from vcl_backend_response because it doesn't
add any value.

Better diff with the --ignore-all-space option.

It's a bit all over the place right now and '#' is the most "scripty" of
the 3 types of comments we have.

Better diff with the --word-diff --word-diff-regex=. options.

Salavaged from #3503

On their own, those changes improve the documentation and don't mention
the reverted "hash_data() in vcl_recv" feature.

Refs e98e8e64

This reverts the following commits:

- e98e8e64.
  "Documentation updates for changed `vcl_hash{}` / `hash_data()`"
- 001279eb.
  "Document proper design pattern for using hash_data() in vcl_recv,"
- e36573e2.
  "Add a test-case for hash_data() in vcl_recv{}"
- 03fe0cee.
  "Allow hash_data() in vcl_recv{}"
- 4ebc3cfe.
  "Make it possible to override the initial digest, and explain in"
- d6ad52f5
  "Change the way we calculate the hash key for the cache."

Conflicts:
	doc/sphinx/reference/dp_vcl_recv_hash.rst
	doc/sphinx/reference/index.rst

Concerns were raised regarding a change of the way we compute the hash
key outside of the dot-zero release where we would expect such breaking
changes (among other things, vmod_shard relies on hash stability).

There is also no definite consensus of how to handle hashing from
vcl_recv.

... documented on Linux as POSIX.1

The exception here is ECONNREFUSED which so far we only tolerate for
Solaris and which seems to make sense for connect() only.

To be discussed in #3539

The watchdog mechanism currently triggers when any queueing is happening,
regardless of the priority. Strictly speaking it is only the backend
fetches that are critical to get executed, and this prevents the thread
limits to be used as limits on the amount of work the Varnish instance
should handle.

This can be especially important for instances with H/2 enabled, as these
connections will be holding threads for extended periods of time, possibly
triggering the watchdog in benign situations.

This patch limits the watchdog to only trigger for no queue development
on the highest priority queue.

When accepting new incoming connections in the acceptor thread, it would
schedule, they would be registered with the VCA priority. This priority is
reserved for the acceptor thread itself, and specifically is not included
in the TASK_QUEUE_CLIENT categorisation. This would interfere with the
thread reserve pools.

t02011.vtc had to be adjusted to account for the new priority
categorisation of the initial request.

in a place more likely to be noticed

I also played with other spatch options to replace ``--dir .``, but they
resulted in more clumsy shell commands.

Ref a98c1489

double quotes ourselves.

FreeBSD's code is undergoing some compatibility adaptations
so we go directly to the source from here.

PS: Please keep cochinelle out of libvgz.

Until now we'd send a zero-length DATA frame with the END_STREAM flag
set, masquerading fetch streaming failures as completed responses. In
the absence of a content-length header a user agent couldn't notice
there was a problem.

This does not apply to ESI sub-requests, they aren't streamed during
delivery.

Since the removal of dry signals, pools will spin when they run out of
threads and increment MAIN.threads_limited at a very high rate. That
spike in CPU consumption will also have detrimental effects on useful
tasks.

This change introduces a 1s delay when the pool is saturated. This
allows to periodically exercise the watchdog check. We could wait until
the watchdog times out but we would also miss potential updates to the
thread_pool_watchdog parameter.

To avoid spurious increments of MAIN.threads_limited we only take task
submissions into account and ignore timeouts of the condvar.

Refs #2942
Refs #3531

it was already accepted on Solaris and NetBSD, now we have seen it on
Linux and I think it does not make sense to keep the exception for
Apple.

Fixes #3532 (hopefully)

Spotted by: Rasmus Villemoes

Spotted by: Rasmus Villemoes

Spotted by: Rasmus Villemoes

Spotted by: Rasmus Villemoes