This introduces two namespaces MAIN and TYPE to allow top-level symbol
collisions for type methods.

For example, `backend` is a reserved keyword but `backend.resolve` is a
type method. It is impossible to declare such a method symbol because it
would require its parent `backend` symbol to be a type symbol.

There would be a conflict between two `backend` symbols of different
kinds. The same would apply if the BLOB type had methods and vmod_blob
was imported.

At this point only the MAIN namespace is used but the split enables the
symbolization of types and type methods down the line.

This is a first step towards making expression parsing more agnostic and
type methods more prevalent.

For now, this is a copy of struct vcc_methods used in expression parsing
and the list includes type properties as well.

Some assertions or expectations needed to loosen to accomodate $OBJ,
$FUNC and $METHOD entries under the same roof. There is a SYM_METHOD
symbol kind that is currently not used, and shouldn't be until VMOD
methods are per-object symbols instead of per-instance.

Once we reach that point, it becomes possible to simplify further and
move more things in the func_sym() function.

Instead, register all the functions and object constructors at import
time.

It might not be installed as /usr/bin/clang, and there's a mismatch
anyway between the check and the outcome.

For example on my system's $PATH clang is currently driven by ccache
is available at /usr/lib64/ccache/clang.

In order to bootstrap a $(builddir) outside of $(srcdir) until now the
first step needed was to populate the environment. If for example one
wants to work from a build/ sub-directory the shortest autogen.des
invocation would look like:

    SRCDIR=.. ../autogen.des [configure options...]

Since we can infer $(srcdir) from autogen.des' relative path, it is now
possible to simply do this:

    ../autogen.des [configure options...]

It is still possible to provide a specific SRCDIR, even to a different
directory but the default should now work out of the box.

We can get rid of autogen.des' egrep command used to silence this
warning too. In fact we can even remove the subshell and simply
call autoreconf directly. The problem with egrep is the loss of
a meaningful exit status for the autoreconf invocation.

I also enabled autoreconf's verbose output.

We override vmod_debug's LDFLAGS, so that warning is gone.

Instead of calling all VCC files vmod.vcc they were renamed to
vmod_foo.vcc and instead of generating vcc_if.[ch] we do the same
thing with vcc_foo_if.[ch].

This a manual mechanical change that will then be picked up by
vmodtool's automake boilerplate generator.

Better diff with the --word-diff --word-diff-regex=. options.

Ref
https://github.com/varnishcache/varnish-cache/pull/3315#discussion_r429796022

OKed by phk on irc.

VRT_DirectorResolve() was the name originally proposed in #2680, which
was renamed in #3311 and accepted by me to get ahead. Not that Dridi
prefers the original name and phk goes along, we change it back before
the next release.

recover a minor improvement from #2680:

This commit is to emit slightly different error messages to
differentiate between the cases "no backend set in vcl" and
"none returned from the director".

Includes minor polishing

Ref #2860 #3311

With a separate autoconf macro to preserve existing behavior. While at
it, I got rid of some intermediate processing in the generated configure
script that should result in slightly smaller scripts for third party
VMODs.

note: on SmartOS, it should be part of the base install

Ref #3309

This reverts commit 970c59ed.

this has been discussed on irc.

I agree that not rebuilding man pages when they are already there makes
sense.

But not building man pages when we could, does not make any sense to me.
And enforcing maintainer mode to build from git does not seem to make
any sense to me either.

This brings back commit cc42bd31:

When we release a dist archive, it comes with the documentation but
since some of it is generated by programs it makes no real difference,
and actually it makes things worse: rebuilding from the dist archive
ends up with a rebuild of the documentation because some documentation
sources ended up being rebuilt. In other words, the documentation we
ship is dead weight in the dist archive and doesn't reduce the number
of build dependencies downstream.

From now on, rst2man remains mandatory to build our manual pages but
can safely be omitted by packaging scripts.

As documented in autogen.des, this is what we should use for development
and it should encompass continuous integration. Until autogen.sh goes
away we need to explicitly enable maintainer mode at configure time.

Two thirds of our CI setup rely on autogen.des today.

Better diff with the --word-diff option.

The script remains until its refcount drops to zero.

Refs 0b4a90c0

When retrying after a rollback, we might have unread bytes of the
backend response body. As we need to finalize the director for rollback,
all we can do in this case is to close the connection.

Spotted by reza, who also provided the test case, thank you.

Closes #3353

Ref: #3348, #741

Refs #3345

Refs #3345

This is a tool for VMOD authors for the use case of building a regular
expression partially from arbitrary input, where the input is intended
for an exact match.

For example, one could implement a dispatch feature depending on the
request's host header, building something like:

    "\.?\Q" + req.http.host + "\E$"

A malicious client could however hijack the regular expression with a
\E sequence in the host header. To get safely to this result you can
do this instead in pseudo-code before compiling the regex:

    VSB_cat(vsb, "\\.?");
    VRE_quote(vsb, req.http.host);
    VSB_putc(vsb, '$');

The input is enclosed with PCRE's \Q and \E escape sequences, ensuring
that \E sequences in the input string don't allow Little Bobby Tables'
cousin to mess with your regular expressions.

Once HSH_Unbusy() has been called there is a possibility for
EXP_Remove() to be called before the fetch thread has had a chance to call
EXP_Insert(). By adding a OC_EF_NEW flag on the objects during
HSH_Unbusy(), that is removed again during EXP_Insert(), we can keep track
and clean up once EXP_Insert() is called by the inserting thread if
EXP_Remove() was called in the mean time.

This patch also removes the AZ(OC_F_DYING) in EXP_Insert(), as that is no
longer a requirement.

Fixes: #2999

This makes the order of events the same as on real cache insertions.

The OC_EF_REFD flag indicates whether expiry has a ref on the
OC. Previously, the flag was only gained during the call to
EXP_Insert. With this patch, and the helper function EXP_RefNewObjcore(),
the flag is gained while holding the objhead mutex during
HSH_Unbusy(). This enables the expiry functions to test on missing
OC_EF_REFD and quickly return without having to take the main expiry
mutex.

When posting to the expiry thread, we wrongly counted exp_mailed also if
the OC in question was already on the mail queue. This could cause a
discrepency between the exp_mailed and exp_received counters.

This enables doing extra handling while holding the mutex specific to
EXP_Insert/EXP_Remove before/after calling exp_mail_it.

See 8e45ba9f

This reverts commit cc42bd31.

For some reason Travis CI jobs use autogen.sh and don't build with all
the bells and whistles we expect developers to work with. Despite the
explicit mention in autogen.des:

    # Use this when doing code development

Some of our continuous integration doesn't emulate this properly. At
least on the VTEST and CircleCI sides we do things as expected. I will
reintroduce this change later when Travis CI is ready to take it.

We shouldn't need to care about all the details when autoreconf(1) can
do that for us. The autoworld has changed over the last decade and this
only affects Varnish developers since we ship release archives with a
turnkey configure script. So let's see how much wreckage this change will
cause.

The script remains until its refcount drops to zero.

In order to get early feedback for MacOS the relevant Travis CI job is
already updated.

When we release a dist archive, it comes with the documentation but
since some of it is generated by programs we end up it makes no real
difference, and actually it makes things worse: rebuilding from the
dist archive ends up with a rebuild of the documentation because some
sources ended up being rebuilt. In other words, the documentation we
ship is dead weight in the dist archive and doesn't reduce the number
of build dependencies downstream.

From now on, rst2man remains mandatory to build our manual pages but
can safely be omitted by packaging scripts.

taken from varnishtop. NOOP because #if 0'd

This is understandably no longer present on the buster base docker
image, but I'm not adding this to installation instructions in the
documentation because I hope to remove that dependency before the
next release.