I was initially going to align the output only if the nonstandard
column(1) command was present.

As I was comparing the output of two varnishstat executions that were
captured after a fresh start of varnishd followed by the workloads to
compare, I realized diff(1) was giving me a hard time, and git-diff(1)
barely improved the situation.

Looking for generic command line utilities to compare metrics I wasn't
able to find anything. So instead I came up with the output format I
thought would help me spot interesting differences and came up with a
format inspired by the unified diff, with a twist. I wanted metrics to
be vertically aligned to easily see differences in orders of magnitude
and reduce the noise to a minimum.

The result is that taking this detour to script varnishstatdiff sped
my research up ultimately.

This should hopefully be portable to POSIX systems.

This will add scripts we ship to the PATH, so they remain reachable for
VPATH builds.

This removes the ability to run in -i mode without finding a makefile to
derive ${topbuild} from, which shouldn't exist in the first place.

In the top makefile we make the check target depend on the all target,
so we ensure that everything is built before we start using varnishtest.

I accidentally broke the installation in 9f996c1f

Stumpled over (unsigned:1)var = FEATURE(...) not being true
when it should have been...

Related: We might want to remove some redundancy from common_param.h...

Motivated by and in preparation of #3826

The filter check does not make sense in the context of bereq
and the macros became overloaded with two different cases.

Motivated by and in preparation of #3826

The goal is to prevent grace mode from adding load to a saturated
Varnish server.

A background fetch entering the queue will block the client task that
triggerred it until it starts its execution, and reaches the point
where it no longer needs to hold onto its req.

On a saturated system this can result in significant client latency
despite a grace hit. A stale object can be served until the end of its
grace period, at which point a regular fetch would be attempted, and
eligible to queuing if that is still necessary.

This turns the Pool_Task() failure dead branch from VBF_Fetch() into a
reachable one.

A fetch transaction is always successfully scheduled or queued, so the
fetch_no_thread counter can only ever stay at zero. The plan is to relax
this for bgfetch tasks.

The new macro is added to better reflect operations that cater to the
thread reserve. It also enables the allocation the right number of
reserve heads in struct pool, instead of having one for each priority.
It grants the possibility of priority classes so low that they wouldn't
be eligible to queuing, without wasting space with needless queue heads.

This better conveys the role of this macro, and leaves the door open to
non-client tasks that would be subject to thread_queue_limit.

It doesn't do half of the things I have in mind, but it's good enough
as a starting point to remove error-prone manual copy-pasta of commands
from the README.

Eventually it should also become usable for out-of-tree code.

It turns out declaring typedefs in coccinelle works better with the
typedef keyword. Reusing the same keyword for the same purposes seems
so obvious in hindsight.

What happened before was that a generic type would be created, making
VCL_BOOL or VCL_VOID match virtually any type, completely defeating
the isomorphism purpose. With this out of the way, we can actually
resume the effort of adding more typedefs.

To summarize:

- we should declare a typedef as an isomorphism
- we should define macros as such

I'm considering renaming tools/coccinelle/vdef.h to varnish.h to match
the varnish.iso file name and wrapping the README instructions in a new
shell script.

This reverts commit c722eb77.

It didn't change anything and it certainly didn't fix anything either.

It does not work, and it is inaccurate since it doesn't include the ctx
identifier that is normally part of the definition (it can't anyway).

Refs f5b0b201

Successfully tested with a dummy semantic patch:

    @@
    idexpression ctx, caller;
    @@

    -CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
    +AN(ctx);
     CHECK_OBJ_NOTNULL(caller, DEBUG_CALLER_MAGIC);
    -AN(caller->sub);
    +assert(caller->sub);

I decided to cram all definitions in the existing vdef.h file to keep
our relatively simple setup. We can add more macros as we learn what
coccinelle trips over.

Refs f5b0b201

As I was relying on the lack of -E option to discard another type of
non-ESI sub-requests I recently misled myself with this assumption. It
took me a while to realize that I was also seeing transactions I was not
interested in, corrupting the statistics I was deriving from a log dump.

Now that vmod_pesi [1] sets the precedent, we should expect new types of
sub-requests to materialize and expecting the reason for the transaction
to be "esi" is no longer appropriate.

We could also track this field as vtx::sub_level for example, to make it
available as VSL query LHS operand. The focus here was to adjust the
behavior of an existing feature, not add a new one.

[1] https://code.uplex.de/uplex-varnish/libvdp-pesi

Same thing for Link tags since they share the same parsing code.

After #3515 merge

I do not understand
     Warning 552: Symbol 'vpi_wrk_len' (line 50, file cache/cache_vpi.c) not accessed

It is used for array sizing, so I have silenced the warning.

Now that we have the vpi ref as set by VPI_Count() available via struct
wrk_vpi, we can dump our position in VCL at the time of the panic.

Additional changes:

* move handling info from pan_wrk() to the new VPI_Panic()
* output the source index of vcl sources with the name in
  VCL_Panic()

The additional panic section looks like this:

vpi = 0x7f5478be93b0 {
  handling (VCL::return) = 0x0 (none),
  ref = 6,
  vpi_ref = 0x7f54753fb530 {
    source = 0 (\"<vcl.inline>\"),
    offset = 365,
    line = 20,
    pos = 25,
    src = \"vtc.panic(\\\"Had Panic header: \\\" + resp.ht[...]\"
  },
},

The source line is truncated (and truncation denoted by [...]) if longer
than 40 characters.

If the in-memory vcl data source string looks truncated, tok = "..." may
be output instead of src "...".

to be able to use our standard miniobj facilities on it, PAN_dump_struct
in this case.

As vpi_ref is static const in VGC, there is no real reason for magic
checks other than that.

So we have it available for Panics in a follow-up commit

(struct vrt_ctx).handling is a special animal: As seen by VGC, struct
vrt_ctx is const (and rightly so), yet VCL needs to have a way to
communicate up to core code the return value from subs.

So, handling was a (non-const) pointer into struct worker. This is all
good and well, with the minor drawback that VMODs would get to see it,
while we should make it very clear that they have to go through
VRT_handling() / VRT_handled() and know the rules (no reset of
handling).

As we now want to move also the vpi count into "the ctx", before adding
another unsigned pointer, we introduce struct wrk_vpi, provided by the
worker, but owned by vpi.

With this change, we thus formalize the vpi count requirement and also
remove the handling variable from vmods' direct line of sight.

This commit comprises the manual changes which either could not be or
were not worth being automated.

The next two commits contain a coccinelle script to do the rest of the
work, and the reult of it.

Add a comment so nobody makes that mistake again.

Also: Dont trust brain to work in heat-wave.