The script remains until its refcount drops to zero.

Refs 0b4a90c0

When retrying after a rollback, we might have unread bytes of the
backend response body. As we need to finalize the director for rollback,
all we can do in this case is to close the connection.

Spotted by reza, who also provided the test case, thank you.

Closes #3353

Ref: #3348, #741

Refs #3345

Refs #3345

This is a tool for VMOD authors for the use case of building a regular
expression partially from arbitrary input, where the input is intended
for an exact match.

For example, one could implement a dispatch feature depending on the
request's host header, building something like:

    "\.?\Q" + req.http.host + "\E$"

A malicious client could however hijack the regular expression with a
\E sequence in the host header. To get safely to this result you can
do this instead in pseudo-code before compiling the regex:

    VSB_cat(vsb, "\\.?");
    VRE_quote(vsb, req.http.host);
    VSB_putc(vsb, '$');

The input is enclosed with PCRE's \Q and \E escape sequences, ensuring
that \E sequences in the input string don't allow Little Bobby Tables'
cousin to mess with your regular expressions.

Once HSH_Unbusy() has been called there is a possibility for
EXP_Remove() to be called before the fetch thread has had a chance to call
EXP_Insert(). By adding a OC_EF_NEW flag on the objects during
HSH_Unbusy(), that is removed again during EXP_Insert(), we can keep track
and clean up once EXP_Insert() is called by the inserting thread if
EXP_Remove() was called in the mean time.

This patch also removes the AZ(OC_F_DYING) in EXP_Insert(), as that is no
longer a requirement.

Fixes: #2999

This makes the order of events the same as on real cache insertions.

The OC_EF_REFD flag indicates whether expiry has a ref on the
OC. Previously, the flag was only gained during the call to
EXP_Insert. With this patch, and the helper function EXP_RefNewObjcore(),
the flag is gained while holding the objhead mutex during
HSH_Unbusy(). This enables the expiry functions to test on missing
OC_EF_REFD and quickly return without having to take the main expiry
mutex.

When posting to the expiry thread, we wrongly counted exp_mailed also if
the OC in question was already on the mail queue. This could cause a
discrepency between the exp_mailed and exp_received counters.

This enables doing extra handling while holding the mutex specific to
EXP_Insert/EXP_Remove before/after calling exp_mail_it.

See 8e45ba9f

This reverts commit cc42bd31.

For some reason Travis CI jobs use autogen.sh and don't build with all
the bells and whistles we expect developers to work with. Despite the
explicit mention in autogen.des:

    # Use this when doing code development

Some of our continuous integration doesn't emulate this properly. At
least on the VTEST and CircleCI sides we do things as expected. I will
reintroduce this change later when Travis CI is ready to take it.

We shouldn't need to care about all the details when autoreconf(1) can
do that for us. The autoworld has changed over the last decade and this
only affects Varnish developers since we ship release archives with a
turnkey configure script. So let's see how much wreckage this change will
cause.

The script remains until its refcount drops to zero.

In order to get early feedback for MacOS the relevant Travis CI job is
already updated.

When we release a dist archive, it comes with the documentation but
since some of it is generated by programs we end up it makes no real
difference, and actually it makes things worse: rebuilding from the
dist archive ends up with a rebuild of the documentation because some
sources ended up being rebuilt. In other words, the documentation we
ship is dead weight in the dist archive and doesn't reduce the number
of build dependencies downstream.

From now on, rst2man remains mandatory to build our manual pages but
can safely be omitted by packaging scripts.

taken from varnishtop. NOOP because #if 0'd

This is understandably no longer present on the buster base docker
image, but I'm not adding this to installation instructions in the
documentation because I hope to remove that dependency before the
next release.

At this point I'm wondering whether we should talk about hacks when we
really add missing dependencies. In some cases it boils down to RST
being generated by programs we need to compile, what I call "regular"
targets here.

One exception could be the changes builder, but if we ever reach the
point where we can automate the changelog initial creation we can bring
it back.

The previous commit added symlinks for source files.

Fixes #3309

... if they differ.

Ref #3309

Partially reverts commit 883fddfe.

Ref #3309

This was brought up by Dridi in an email response to
b90b60d0:

Initially, I also thought that we should VRT_fail() for an invalid
parameter, but on second thought I realized that, as the shard director
supports request-time reconfiguration, graceful error handling should be
possible, so VRT_fail() is too hard.

I think that even returning false for .add_backend() is too harsh, as an
invalid weight is probably not too much of an issue to abort the
reconfiguration in the case of caller error handling.

That said, this might all be over the top. But I really do not want to
run into the (still unfixed) case I saw recently trying to base64 decode
an invalid input, which is not possible to handle gracefully.

... to be more in line with other bundled directors

from flexelint review

We implement weights by scaling the number of replicas of each backend.
The replicas parameter of .reconfigure() remains a minimum.

For existing vtcs, the Debug hashcircle output has been compared
before/after this change to ensure that behaviour is exactly equivalent.

For for wighted backends, it has been checked that the number of
instances per host on the hashcircle matches the expectation.

Also refactor and clean up some of the code:

- consistently make the number of ring points a uint32_t
- some constification

Ref #3276