Also move the lock up to cover more operations.

Previously we would read the response Content-Length from a failed oc,
which would make the error response valid. Now, if this is detected,
we don't touch the Content-Length.

VRT_delete_backend() sets be->cooled to non-zero as the only place where
that is done. Assert that it is zero on entry as a check that
VRT_delete_backend isn't called multiple times.

We refuse to accept new dynamic backends while the VCL is cooling, and
drop adding the attempted backend on the VCL's backend list when that
condition is found. But that would cause an assert later when it is picked
off the cool_backends list for destruction. Fix this by directly
destroying the backend instead of going through the cooling list.

Note that this patch removes the ASSERT_CLI() macro in vbe_destroy().

Several functions (VBE_Poll and vbe_destroy) tests be->cooled == 0 to
determine which of the two lists backends and cool_backends a specific
instance currently lives on. If the flag is in the process of being
changed, then the wrong list head may be used and will result in strange
bugs.

 Conflicts:
	bin/varnishd/cache/cache_backend.c

The last three commits already made configure recommend installing
Python 3 packages and look for versioned executables, however with a low
priority.

This is a problem on systems such as OpenBSD 6.5 with a default Python
version at 2.7, where 3.7 flavored Python packages get installed with
a "-3" binary suffix.  That is, when both rst2man and rst2man-3 are
installed at configure time, the lower version will be picked unless
explicitly passed through `--with-feature' arguments.

Regardless of this specific case, trying more specificly versioned tool
names first seems correctly in line with recent development and less
error prone, so change it accordingly.

 Conflicts:
	configure.ac

For a given definition of "future" or "now".

Until the naked "python" executable refers to python3 (currently it is
still python2) it now takes lower precedence.

The assertion that the stale objcore of a conditional fetch cannot be
failed unless it was streaming is incorrect. Between the moment when
we grab the stale objcore in HSH_Lookup and the moment we try to use
it after vcl_backend_response, the backend fetch may have completed or
failed.

Instead, we need to treat an ongoing fetch and a failed fetch as
separate checks since the latter may happen with or without a boc.

 Conflicts:
	bin/varnishd/cache/cache_fetch.c

Test case by Reza, thank you

Fixes #3433
Closes #3434

Once we ask the backend to close the connection after a given request
there is no benefit from putting the backend connection back in the
pool. It's actually a surefire way to force a subsequent backend fetch
to fail its first attempt and go straight to its extra chance.

Since we try to maximize connection reuse this would have to come from
VCL and a user asking for the backend to close the connection should
have a good reason to do so, for example when the backend is known to
misbehave under certain circumstances.

Closes #3400
Refs #3405

Original author: @dridi

This commit is from ba78ebeb but
with no analogous file in 6.0, it is added to changes-6.0.rst instead of changes-trunk.rst

Whether the header was set by the backend or directly in VCL, it is now
possible to signal that a backend connection should not be added back to
the pool after a successful fetch with a Connection:close header.

Pooling such a connection would be counter-productive if closing the
session was requested by the backend itself, because it would then be
likely that reusing the connection would result in busting the extra
chance.

Setting the Connection:close directly in VCL can help mitigating against
a misbehaving backend.

Refs #3400

When resolve requests race, we were not guaranteed to consider all
backends because we updated a shared nxt variable.

Fixes #3474

We do not hold a reference, the magic can be unstable.

We can incorrectly reference resp.reason from other sources when
jumping into vcl_synth. This also covers passing in a reason in
vcl_backend_error.

The bo fields err_code and err_reason need to be reset on a retry otherwise the values are kept.

Fixes #3525

 Conflicts:
	doc/sphinx/installation/install_source.rst

If given a build parameter called 'dist-url', the build script downloads
a tarball from the given URL instead of doing a 'make dist' step.

This reverts commit 4f99d164.

This was merged before review, by mistake. I am reverting this so that
we can do the quality assurence before the actual merge. Sorry.

Also make sure we didn't overflow before entering vcl_pipe. This would
mean we have lost important connection headers.

- Complete the missing HTTP response codes / messages from :
 https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml

 patch from master@d686166ad469cad4043ed24328f4377e0d5f71a7