It's about time...

The --enable-maintainer-mode option has been used in autogen.des for a
while now and it has allowed automatic reconfiguration and hot reload of
makefiles.

For the next step, we can turn certain programs into maintainer programs
and only require them when they are absolutely needed. This is what we
currently do with SVG files generated from dot files, checked in git and
optionally required, with an error message when rebuild is not possible.

It turns out we only need 3 lines in configure.ac to implement all that,
with the following differences:

- no more `./configure --with-dot[=...]` option
  - use `./configure DOT=...` to override the default `dot` command
  - or alternatively run configure with `DOT=...` in the environment
- generic missing error message (but referencing dot) from make
- no need to reconfigure once `dot` is in the PATH

The DOT override is actually documented in `./configure --help` like the
`--with-dot[=...]` option was.

While dot is our most emblematic maintainer tool, with its build targets
checked in all the way in the git repository, this could be applied to
other maintainer programs like sphinx-build and rst2man that should not
be required when building from a dist archive.

We don't need to render witness.svg in CI, even if dot(1) is available.

Should we give it an explicit ETIMEDOUT in the dripping write case?

Better diff with the --word-diff --word-diff-regex=. options.

This is to ensure we don't lose write errors by the time we make the
final checks.

Instead of trying to allocate storage for the entire request body when
the content length is known, let it go in fetch_chunksize steps. This
may prevent spurious cache evictions for large request bodies for pass
transactions when we are merely streaming the body.

On the other hand, when the content length is known and the goal is to
cache the request body, we may still attempt a single allocation and all
that entails.

Also motivated by #3935

motivated by #3935

https://editorconfig.org/

Spotted by Coverity Scan.

Refs #3914

Refs #3929

With that we lose the ability to rebuild the manual from the doc/sphinx
directory if the sources were modified, just like any other dependency
on $(top_builddir) sources. This one is linked once and for all, but
doesn't fit the link_srcdir target.

Suggested by @nigoroll.

This guarantees that VSC code is generated before compiling anything.

Fixes #3928

restriction to port 443 in haproxy config inflexible and unnecessary

The VSB bits in here come with a very significant performance penalty
for H2 request body processing.

And make vsc.am expect it as one of the input variables.

This is the convention for command line programs used in makefiles, like
CC, LD and plenty others. This is also what varnish.m4 exposes to out of
tree projects.

The new vsc.am file takes care of most of the logic, and include sites
only need to reference sources as such.

This change adds a new /usr/share/varnish/vcc (or equivalent) directory
where the built-in VMOD descriptors are installed. Only vmod_debug.vcc
is omitted since vmod_debug is not installed. Shipping VCC files can be
useful to end users, for example, to implement VCL autocompletion in a
text editor.

This change does not even begin to manage this for third-party VMODs.

With this, the documentation becomes authoritative for the VCL state
machine, just like VCL variables.

This also belongs in a reference manual and should not be disjoint. It
also avoids broken references with rst2man that only sphinx can resolve
across multiple pages.

Instead of documenting built-in subroutines in the user guide, a new
vcl-step(7) manual is added. It is structured with a vcl-step.rst file
that includes a vcl_step.rst file containing the actual list of steps.

This is similar to how the vcl-var(7) manual is constructed to later
make it authoritative.

The 304 handling and initial values for beresp.ttl, beresp.grace and
beresp.keep belong in a reference manual.

Transports should be free to keep a reference on the object to be
delivered until after their transport function returns, but
HSH_Cancel() in cnt_transmit() prevented the object from being of any
use for the case that it is final (pass/hfm/hfp).

We solve this by moving the HSH_Cancel() close to VDP_Close, which
also makes sense from the perspective of the VDP design: Until the VDP
close, filters could still reference object data.

HSH_Cancel() needs the objcore, which could be reachable also via
vdc->req. But that member is unset in VDP_DeliverObj(), presumably to
make it clear that VDP .bytes callbacks should not access request
data. Thus, we pass it as a new argument to VDP_Close() as well as any
boc being held by the caller.

The objcore can also be NULL for the case where a transport generates
the body without holding an objcore at all.