Feel free to improve further!

in ban_lurker_test_ban() we remove completed bans from the obans list
so the ban list tail might not be present in the obans list.

To ensure that we stop at the tail, we now find our obans iteration
termination ban walking the bans list backwards until we find an
active ban, the head or the first obans element. In the latter two
cases we are done.

If there are any bans left to mark completed, the last active ban must
be contained in the obans list so we should terminate correctly.

Fixes #2556

Closes #2598

Fixes #2597

Merges #2600 slightly modified

depends on the client-varnish connection happening over
127.0.0.1.  Fake it for jails and IPv6-oid hosts.

We introduce a shard_param object to hold the shard director lookup
parameters which until now could only be passed to the .backend()
method.

By associating a parameter object with a shard director, we enable
LAZY lookups as with the other directors. Parameter objects are
defined with VCL scope (normal vmod objects), but can be overridden
per backend request using a task priv.

We use the same concept to carry shard.backend() parameters to
vdi resolve for LAZY mode: They get saved in a per-director task
scope parameter object.

Each object points to another object providing defaults for
values which are not defined.

Actual resolution of the various parameter objects does not happen
before they are used to allow changing them independently (ie,
shard .backend() parameters have precedence over an associated
parameter object, which by itself can be overridden).

Overview of parameter objects (pointers are alternatives)

shard() director	shard_param() object	default praram

	     --------------------------------->	  vmod static
  VCL obj   /				     ->
  .param  -+--------->	  VCL obj	    /  _
  			  .default  --------   /|
					      /
			     ^		     /
			     |		    /
					   /
			  .default	  /
	------------->	  TASK priv	 /
       /				/
  .default -----------------------------
  TASK priv

Properly implementing vmod_storage requires the VMOD to
sanction symbols at VCC compile time, and I'd rather not
face that right now.

Fixes #2596

Spotted by: Dridi

Add assert to make sure it stays that way.

Ref: #2500

see doc/changes.rst for details

Closes #2509

This basically reverts a29fca70,
except that we are using python instead of make-inlined shell code to
do the work.

Reason: vcs_version.h needs to be up-to-date under all circumstances,
otherwise we will end up with wrong version information in binary
builds, which would divert developer resources into unproductive
confusion compensation.

With an all-in-one generate.py, we basically have the choice between:

- running it with each build, which breaks incremental builds as
  generate.py rewrites central includes like vcl.h

- adding version information to all generate.py-build files or similar
  mechanisms to avoid re-writing them unless they have actually
  changed.

- contradicting the argument given above

I think that, unless there are strong reasons for a single
generate.py, avoiding these issues by splitting functions is the best
option.

This got lost with a29fca70

By imposing a fixed ordering on our levels structs, we ensure that pointers
to them are also ordered such that comparing two of them will compare as level
ordering.

Fixes #2587

If the listen() system call were to report problems, they are now reported
back to the management process, failing the child startup (instead of
asserting). This can happen on many widely used Linux kernels if the
listen socket already has another process listening on that port with the
SO_REUSEPORT flag set.

Newer Linux kernels will report the problem at bind() time (causing error
to be reported upon management process startup).

Fixes: #2551

These were logged with a vxid==socket, which is wrong (leftover from old
logging). They are now logged with vxid==0, which is for non-transactional
messages.

This moves the first time initialization of the listening sockets to the
CLI "start" command handler. This enables errors to be reported to the
management process if we were unable to comply.

As the code was, we would run into an assert immediately upon starting the
child when we handed it a socket==-1. Better to exit with an error
message.

To keep running with a degraded set might catch people by surprise, so the
only option left is to exit the process.

This makes sure that any panic message from the child is handled and shown
before continuing (or possibly failing).

This is so that errors in either will be reported under the right heading.

... so log it under the Debug tag.

FetchErrors should be actual errors which can be addressed. In this case,
nothing is wrong in any way, the fact that we abort a fetch if we don't
need the body is a varnish internal optimization (which makes sense, but
comes at the cost of closing a connection).

Merges #2450

They add up to a considerable chunk of the VSL traffic, especially when
cookies are involved, while providing little value when troubleshooting.
In an effort to reduce VSL cycles on heavy workloads I found these to be
good candidates for several reasons:

- obj.* fields are immutable in VCL
- after a hit, we initially get the same Resp* records in vcl_deliver
- before vcl_backend_response returns, we can infer them from Beresp*

The main difference would be ObjHeader:X-Varnish, but it would be the
same as BerespHeader:X-Varnish, and be part of the RespHeader counterpart
in the case of a hit. That, and cache-related headers adjusted to the
current time.

One case where they wouldn't appear is for example on a hit followed by
a restart. I have yet to find a use case where I'd need to query them
with varnishlog or some other VUT.