imples the following changes:

* VDI_Finish() is now always conditional on bo->director_state !=
  DIR_S_NULL, making it idempotent

* introduces additional calls to VFP_Close() from startfetch and
  for the filter_list / VCL_StackVFP error in vbf_stp_fetch(),
  but VFP_Close() is idempotent.

* adds VFP_Close() for VFP_Open() failure in vbf_stp_fetch() which
  I think is actually missing (for the case that some VFPs could
  get opened before the open failure)

* calls VDI_Finish() earlier in vbf_stp_fetchend: I checked the
  code and can not see any issue with this.

motivated by #3009

 Conflicts:
	bin/varnishd/cache/cache_fetch.c

required for #3009

VGZ_NewGzip will either assert or succeed.

Turn assertion into VFP error

The vtc is based upon r02645.vtc and reliably reproduces the panic
without the patch by sweeping through possible amounts of free workspace
ranging from 4 to 400 bytes.

Fixes #3253

This reverts commit 79536662.

It shouldn't be exposed in libvarnishapi the way it is in this change
because it creates two diverging symbol versions (for systems with
linker support for symbol versioning) without changing the soname,
which of course we want to avoid in the first place in a stable branch.

If the requirement comes up we can revisit this topic and ensure both
binary stability and compatility (upgrade path).

Refs #3268

Conflicts:
	doc/sphinx/reference/vcl_var.rst

Test inspired by Carlos Abalde

conflicts:
    bin/varnishd/cache/cache_fetch.c

And update the comment explaining what happens. This is to make it
apparent that a VFP has changed C-E in the stored object, so C-E
should not be changed by an IMS fetch, and a new ETag from IMS has
to be weakened. The flag is appropriate for use by any VFP that
needs to manipulate C-E this way (not just gzip).

Closes #2910

Fix a small memory leak when failing to spawn a new thread.

When getidleworker signals the pool herder to spawn a thread, it increases
the dry counter, and the herder resets dry again when spawning a single
thread. This will in many cases only create a single thread even though
the herder was signaled dry multiple times, and may cause a situation
where the cache acceptor is queued and no new thread is created. Together
with long lasting tasks (ie endless pipelines), and all other tasks having
higher priority, this will prevent the cache acceptor from being
rescheduled.

c00096.vtc demonstrates how this can lock up.

To fix this, spawn threads if we have queued tasks and we are below the
thread maximum level.

Conflicts:
    bin/varnishd/cache/cache_wrk.c

Conflicts:
    bin/varnishd/http2/cache_http2_deliver.c

Copy paste the text from beresp.status to resp.status about the special
handling of status codes.

Some status codes are illegal and will cause VRT_fail() when executed as
normal set instructions in VCL. But this test is bypassed when status is
set as a side effect of a `return (synth(code))` statement.

This patch applies the same rules as when executing a set-instruction to
the return(synth()) handling.

Fixes second part of: #3301

The compound statement doing return() in VCL may have failed for some
reason, for example running out of workspace while putting together the
reason field. The fail handling would be overwritten by the forced
handling at the end of the generated code for a return statement.

This patch adds a test for the failure condition before forcing the
handling to the result of the return statement.

Conflicts:
    lib/libvcc/vcc_action.c

In VCL we allow status codes larger then 999, and do modulo 1000 when
sending it to the client. But the code to apply the modulo was missing in
H/2.

This issue was uncovered and reported by @Francois-v3.

Fixes first case of: #3301

Conflicts:
    bin/varnishd/http2/cache_http2_deliver.c

Fixes: #3266

Fixes: #3273

Automated with Coccinelle, so the semantic patch could be reused in the
vtest project.

Closes #3238

Conflicts:
    lib/libvcc/vcc_compile.c
    bin/varnishd/proxy/cache_proxy_proto.c

Closes #3238

Conflicts:
    lib/libvarnishapi/libvarnishapi.map

it is not before we have parsed parameters

Conflicts:
    bin/varnishd/mgt/mgt_main.c

mgt_cli_askchild() as opposed to directly through mcf_askchild()),
a truncated CLI response is not a failure.

Fixes #3038

Conflicts:
    bin/varnishd/common/common_vsmw.c

Two new paramters allow the limit and than handling (ignore, warn, fail)
to be configured.

Fixes #2713

Wait for the stale object to become fully fetched, so that we can catch
fetch errors, before we unbusy the new object. This serves two
purposes. First it helps with request coalescing, and stops long chains of
IMS-updated short-TTL objects all streaming from a single slow body
fetch. Second it makes sure that all the object attributes are complete
when we copy them (this would be an issue for ie OA_GZIPBITS).

This patch OBE's r01646.vtc, and slightly patches r01648.vtc to expect a
503 instead of a 200 and a broken connection on the failing client.

Fixes: #3089

Conflicts:
    bin/varnishd/cache/cache_fetch.c

This VSL tag will be used for informational messages related to
exceptional handling of requests.

Conflicts:
    doc/sphinx/reference/vsl.rst

Log a notice message when delaying a conditional fetch and the stale
template object is still streaming.

Conflicts:
    bin/varnishd/cache/cache_fetch.c
    doc/sphinx/reference/vsl.rst

When doing a pass, we would remove the Age header from the backend, and
create a new one based on the time the fetch was initiated. This creates
problems when calculating the time to live in downstream caches (browser
cache or layered varnishes).

With this patch, the RFC_2616_Ttl calculation routine is run also for
passes, where the t_origin field of the object is adjusted for an incoming
Age header. This makes sure that the Age header generated during delivery
is correct. The rest of the Ttl calculation is skipped for passes,
including the logging of SLT_TTL "RFC".

Fixes: varnishcache/varnish-cache#3221

One time element function to rule and parse them all.

Change the return value to unsigned, to match with the expected data type
where it is used.

Handle very large numbers consistently. Currently it was converting from
unsigned long to int, which would throw away the most significant
bits. Now overly large integers will be capped at UINT_MAX.

Implement the "allow and ignore decimal point" behaviour that the Age
header parsing incorporated in rfc2616_time(). This way we will allow a
decimal points also in max-age and stale-while-revalidate parsing of
Cache-Control directives.

The change in u00011.vtc is the result of having two commands starting
with "pi", breaking auto-completion. Fortunately "pin\t" still does the
trick.

The assert on WS_ReserveSize() in ses_handle() can not trip because
sizeof (struct pool_task) is less than sizeof (struct waited). But to safe
guard against future problems if that were to change, this patch makes
sure that the session workspace can hold the largest of them before
entering the waiter, erroring out if not.

Currently, with the 505b7bd9 patch, when
calling WS_ReserveSize with bytes equal to the amount of workspace that is
currently available, it will return zero and mark overflow.

This patch redoes the patch, and changes it to return zero and overflow
only when the requested number of bytes is larger than what is available.