Instead of adding hoops to the documentation, in particular to keep it
in sync, improve the only location where we emit HTC status logs.

We could also consider replacing the HTC enum with a struct, similar to
what we did in other places. The struct symbols would be named after the
UPPER name from the table, have a name and description fields, possibly
replace the error number with a simple is_err field.

Capturing the long description like this is less intrusive.

Refs #4042

Spotted by Darryl Rodden.

Submitted by: @AlveElde in Vtest repo on github

This is an extended variable that has the same format as the default one
used when no format is specified.  The variable is useful for appending
or prepending other formatters to the default format.

If VRT_AddDirector() was called from handling a VCL_COLD event or,
indirectly, from another thread which the VCL_COLD event handler was
waiting for, varnishd would deadlock and prevent any CLI or director
changes, because VRT_AddDirector() requires the vcl_mtx, which is held
during vcl_BackendEvent() to ensure a consistent view of the director
list. Because of the early return from VRT_AddDirector() this likely
only happened in VTC mode, but the underlying race existed
nevertheless.

This patch _almost_ fixes the issue with the intend of making it
highly unlikely to occur without getting too involved with the vcl
temperature controls: We now check the same conditions under which
vcl_set_state() would transition the temperature to COOLING and, if
they apply, use Lck_Trylock() in a try/wait loop instead of
Lck_Lock(), avoiding the deadlock.

The patch presumably still does not fix the problem entirely, because
the reads of vcl->busy and vcl->temp before the Lck_Trylock() could
still be outdated. With the temperature controls otherwise unchanged,
the only alternative idea I could come up with was to always use a
try/wait loop, which I dismissed due to the performance impact
(overhead and added latency).

Ref https://github.com/nigoroll/libvmod-dynamic/issues/110

Otherwise poll(2) returns EINVAL on FreeBSD when both pipe_timeout and
pipe_task_deadline are disabled.

Fixes #4043

It takes a vtim_dur and returns a number of milliseconds. NAN means no
timeout and negative values are considered expired deadlines and get a
chance to succeed a non-blocking poll.

This test case has been failing for a while in continuous integration
and trying to bisect the change that triggered a stack overflow for our
Ubuntu 18.04 job I wasn't able to make it pass, even for the commit
where it first landed.

Something must have happened outside of Varnish to consume more stack.

The other timeouts should eventually become unset-able as well, where
an unset variable falls back to a parameter and zero means no timeout,
overriding the fallback parameter.

Refs #4043

... and minor touch up of the file.

No, this does not imply me checking all our RST files for adherence with
this standard.

Ref 0d663a5e

The [be]req.task_deadline should eventually be generalized to all
subroutines with read-write access to [be]req. When that happens,
bereq.task_deadline will inherit req.task_deadline during the pipe
transition.

Related tasks should otherwise have independent deadlines, except
sub-request tasks like ESI where there is a hierarchical dependency.

This takes care of documenting the meaning of a zero timeout value in
a central location, since none of the timeout parameters have a mention
for the effect it produces.

This is written as if there was a separate varnish-param(7) manual,
which should eventually be enacted. This manual extraction was initially
implemented in the adjacent #3817 timeout nomenclature draft.

in VRT_AddDirector, we create the new vcldir with an initial
reference, which we need to drop if we can not add it.

Compare:

	VRT_AddDirector()
		...
		vdir->refcnt++;

	vcldir_free()
		...
		AZ(vdir->refcnt);

Noticed when testing other experimental changes while working on
https://github.com/nigoroll/libvmod-dynamic/issues/110

 #5  0x000055820c8cb845 in VAS_Fail (func=0x55820c904559 "vcldir_free", file=0x55820c903a47 "cache/cache_vrt_vcl.c",
     line=150, cond=0x55820c90459a "(vdir->refcnt) == 0", kind=VAS_ASSERT) at vas.c:67
 #6  0x000055820c83a442 in vcldir_free (vdir=0x7f662aa53140) at cache/cache_vrt_vcl.c:150
 #7  0x000055820c839fe1 in VRT_AddDirector (ctx=0x7f662befe250, m=0x55820c965260 <vbe_methods_noprobe>,
     priv=0x7f662aa20780, fmt=0x55820c900f7f "%s") at cache/cache_vrt_vcl.c:219
 #8  0x000055820c7c7c4d in VRT_new_backend_clustered (ctx=0x7f662befe250, vc=0x0, vrt=0x7f662befdd10, via=0x0)
    at cache/cache_backend.c:737
 #9  0x000055820c7c8632 in VRT_new_backend (ctx=0x7f662befe250, vrt=0x7f662befdd10, via=0x0)
     at cache/cache_backend.c:755

Motivated by #4042

So the n_expired, n_purges and n_obj_purged counters should reflect
that. Likewise, it shouldn't be logged as "EXP_Expired".

And while at it document it.

It is generally useful to know when an object leaves the cache and while
it may sometimes be logged by call sites (like bans) it is more helpful
centralize it in ExpKill records to get a (more) complete view of what
goes in and out of (durable) storage.

The hits attribute is especially useful, as it can give an indication of
the popularity of objects being removed before they expire.

This makes it easier to estimate the popularity of expired objects.

vcldir_retire() only sends a VCL_EVENT_COLD when the vcl is warm. In
other words, it (rightly) asserts that a COLD event (if any) has
alrady been posted when a director is deleted on a COLD vcl.

Yet, when the director is deleted upon a COLD vmod event, this assertion
was wrong, because the COLD events for directors were only posted after
vmod events.

Given that vmods do things like deleting directors, it appears (more)
correct to post VDI COLD events before VMOD COLD events.

https://github.com/vtest/VTest/pull/32

This should be a no-op on all contemporary UNIX implementations, but
let's see what vtest tells us...