Restructured so that:

* 'Upgrading' is limited to work that has to be done to upgrade from
  a current deployment to the new version.

* 'Changes' is a comprehensive, user-level description of changes and
  new features.

Conflicts:
	doc/sphinx/whats-new/index.rst

Committer Edit:
	Added Copyright header by example of
	fe8cfe84 and
	93439884

Noted by Simon Vikström, thank you

Closes #3551

The order matches the order of commits, except where I thought it made
more sense to join topics into contiguous sections.

We can incorrectly reference resp.reason from other sources when
jumping into vcl_synth. This also covers passing in a reason in
vcl_backend_error.

Provide a set of predictable subroutines that will be called by the
built-in vcl. Only transactional built-in subroutines are concerned,
which excludes vcl_init and vcl_fini.

Closes #3548

Better diff with the --word-diff --ignore-all-space options.

When it was in its own file it worked fine...

And fix the h2_req VDP error handling as per the VDP contract.

Test case inspired by Simon. Since this is one of those test cases that
explicitly mix two features I wasn't sure whether I wanted to make this
an h2 test case or a range test case. Since this was ultimately a range
bug I decided to register it in a range test case.

It's not obvious what should have been authoritative here. The range VDP
was rightfully latching an error via SC_RANGE_SHORT that is defined as an
error-type session close reason, but VDP_DeliverObj() doesn't take that
into account. While SC_RANGE_SHORT isn't a session/protocol error for h2
but rather a stream error it is not obvious what VDP_DeliverObj() should
do in the absence of a negative retval and the presence of a non-null
sess_close.

Maybe another way could be to turn enum sess_close into a struct and
embed http1 and h2 specificities directly in struct fields. We already
have somewhat structured information in the sess_close.h table.

Refs 03f71c6e

to support more realistic practical test cases like JWTs

Only copy the host header if needed

It's one more indirection than vcl_recv because the same logic applies
to multiple cases, so we first have subs for the different conditions
and they all call the shared vcl_beresp_hitmiss sub.

The vcl_beresp_hitmiss sub returns on purpose, that doesn't change the
default VCL behavior and makes it more usable from VCL code:

    if (some user condition applies) {
            call vcl_beresp_hitmiss;
    }

No need to force all call sites to return(deliver) when the desired
outcome is explicit. Extracting this sub also enables VCL authors to
only tweak the hitmiss TTL in one place.

For better compliance, we might also introduce a vcl_beresp_hitpass sub
for some of the built-in cases where a hit-for-miss object might be less
appropriate. The compliance effort will however change some semantics
and is outside of the scope of this split.

There is a very slight breaking change in the sense that the host header
check is grouped with its normalization which swaps its order with the
PRI method check.

In practice that only means that if you have both an unattended PRI
request missing a host header, you'll get a 400 instead of a 405. You
have to get both wrong in the first place so I don't consider this a
concern.

Now that we know in advance the available suboutines in the built-in VCL
we can use that as the condition to create an append-able subroutine
when the vcl_ prefix is encountered.

This work is currently incomplete, at least because non-state built-in
subroutines are currently not listed when an unknwon subroutine is
encountered.

Also, instead of proper test coverage this currently uses b00000.vtc as
a strawman to show how to skip a specific processing.

This is a lookahead parsing to learn about built-in subroutines that are
not tied to a VCL state. Instead of maintaining a mapping of the other
subroutines the builtin.vcl file itself becomes authoritative.

I removed the else keyword from vcl_backend_response because it doesn't
add any value.

Better diff with the --ignore-all-space option.

It's a bit all over the place right now and '#' is the most "scripty" of
the 3 types of comments we have.

Better diff with the --word-diff --word-diff-regex=. options.

Salavaged from #3503

On their own, those changes improve the documentation and don't mention
the reverted "hash_data() in vcl_recv" feature.

Refs e98e8e64

This reverts the following commits:

- e98e8e64.
  "Documentation updates for changed `vcl_hash{}` / `hash_data()`"
- 001279eb.
  "Document proper design pattern for using hash_data() in vcl_recv,"
- e36573e2.
  "Add a test-case for hash_data() in vcl_recv{}"
- 03fe0cee.
  "Allow hash_data() in vcl_recv{}"
- 4ebc3cfe.
  "Make it possible to override the initial digest, and explain in"
- d6ad52f5
  "Change the way we calculate the hash key for the cache."

Conflicts:
	doc/sphinx/reference/dp_vcl_recv_hash.rst
	doc/sphinx/reference/index.rst

Concerns were raised regarding a change of the way we compute the hash
key outside of the dot-zero release where we would expect such breaking
changes (among other things, vmod_shard relies on hash stability).

There is also no definite consensus of how to handle hashing from
vcl_recv.

... documented on Linux as POSIX.1

The exception here is ECONNREFUSED which so far we only tolerate for
Solaris and which seems to make sense for connect() only.

To be discussed in #3539

The watchdog mechanism currently triggers when any queueing is happening,
regardless of the priority. Strictly speaking it is only the backend
fetches that are critical to get executed, and this prevents the thread
limits to be used as limits on the amount of work the Varnish instance
should handle.

This can be especially important for instances with H/2 enabled, as these
connections will be holding threads for extended periods of time, possibly
triggering the watchdog in benign situations.

This patch limits the watchdog to only trigger for no queue development
on the highest priority queue.

When accepting new incoming connections in the acceptor thread, it would
schedule, they would be registered with the VCA priority. This priority is
reserved for the acceptor thread itself, and specifically is not included
in the TASK_QUEUE_CLIENT categorisation. This would interfere with the
thread reserve pools.

t02011.vtc had to be adjusted to account for the new priority
categorisation of the initial request.

in a place more likely to be noticed

I also played with other spatch options to replace ``--dir .``, but they
resulted in more clumsy shell commands.

Ref a98c1489

double quotes ourselves.

FreeBSD's code is undergoing some compatibility adaptations
so we go directly to the source from here.

PS: Please keep cochinelle out of libvgz.