Fix the probe scheduler heap comparison function to be consistent with
regard to different running state of the two arguments. With this fix,
probes that are not running will always bubble to the top before those
that are already running.

Otherwise you might run into this:

    Message from VCC-compiler:
    Unused backend nil, defined:
    ('<vcl.inline>' Line 4 Pos 17)
            backend nil none;
    ----------------###------

    (That was just a warning)
    Message from C-compiler:
    vgc.c:1476:20: error: unused variable 'vgc_backend_nil' [-Werror,-Wunused-variable]
    static VCL_BACKEND vgc_backend_nil;
                       ^
    1 error generated.
    Running C-compiler failed, exited with 1
    VCL compilation failed

This is done in both init and discard code to maintain the balance.

As discussed during bugwash, we should be consistent about the manual
temperature controls and not transition cold->warm, but rather fail.

Conceptually, the auto state was a variant of the warm state which
would automatically cool the vcl. Yet, cooling did not only transition
the temperature, but also the state, so 'auto' only worked one way -
except that vcl.use or moving a label (by labeling another vcl) would
also set 'auto', so a manual warm/cold setting would get lost.

Now the auto-state will remain no matter the actual temperature or
labeling, so when a vcl needs to implicitly change temperature (due to
being used or being labeled), an auto vcl will remain auto, and a
cold/warm vcl will change state, but never become 'auto' implicitly.

The vcl state/temperature test v00003.vtc, besides testing the new
auto semantics, now also checks for the right vcl.list output and has
been reduced by a duplicate check (warm event check has been
integrated into an existing warm event).

On other code changes:

* mgt_vcl_setstate

  is now only concerned with the state, the temperature will be
  changed implicitly if so required. The state will either end up
  changed or restored, depending on success.

  owner of changes to the (struct vclprog).state member

* mgt_vcl_settemp

  responsible for the right action to change the temperature. For auto,
  it will only change the temperature, for non-auto, also the state.

  owner of changes to the (struct vclprog).warm member

* mgt_vcl_tellchild

  Inform the child about a change and/or temperature change

* mgt_vcl_set_cooldown

  Update the cooldown (go_cold) appropriately, should be called after
  a change/temperature change.

Fixes #2834
Closes #2801

Conflicts:
	bin/varnishtest/tests/v00003.vtc

the child process.

Inspired by: #2471

Ref #3067

Ref #3067

When making other changes which require additional workspace, this test
may fail for too big a request in the first place.

This additional logging facilitates before/after analysis

Spotted via vtest.

Fixes: #3086

req->err_code and req->err_reason are set when going to synthetic
handling. From there the resp.reason HTTP field is set from
req->err_reason if set, or the generic code based on req->err_code is used
if it was NULL. This patch clears these members so that a value from the
handling of a previous request doesn't linger.

Fixes: VSV00004

Ref previous commit to remove the use of kill(signull) on the pids of the
Varnish processes to test liveness.

A bug was uncovered in the VSM code that checks if kill(SIGNULL) would
work as a test of liveness of the master and worker processes. If the user
running the utility has the required permissions to send signal to the
worker process, but not the management process, the code would wrongly
assume it could do kill on both. It would then end up in a connect loop
never succeeding.

Because kill() can not always be successfully run (a common scenario when
the user running varnishlog is not the same UID as the cache processes),
there is a fall back to using fstat to check for Varnish restarts. Since
this fallback is active for most use cases anyways, it was decided to
retire the kill() mechanism rather than to fix it. This way the behaviour
does not change depending on what user the utility is run as.

This change was OK'd by PHK after discussing it on IRC.

Clusters VSM allocations start at offset 16. Make sure that the published
segment includes that adjustment.

This speeds up vsm_findseg by keeping a direct pointer to the struct
vsm_seg it points to in the vsm_fantom.

To prevent stale vsm_fantoms in an application from causing segfaults, the
last serial number seen is also kept on the fantom. If this serial number
does not match, the slow path of iterating all segs to find the right
match is done, and the fantom is updated to make any subsequent calls on
the same fantom take the fast path.

With this patch the fantoms store 2 serial numbers and a direct pointer to
the seg. To fit this in struct vsm_fantom without breaking the ABI, the
unused chunk pointer value has been repurposed, giving us two uintptr_t
priv variables to work with. The direct segment pointer occupies one, and
the two serial numbers occupy one half each of the other. This limits the
bits for each serial to only 16 bits on 32 bit systems. But only direct
matches is done, and it is unlikely that a stale fantom should have the
exact right value to be accepted as valid.

When upgrading from when a previous release that doesn't have the latest
VSM index file changes, the utilities will assert when trying to parse the
index file from the previous version. Fix that by ignoring lines not
starting with either '#', '+' or '-'.

This moves the exposed flag handling to the segment level, rather than the
point level. This enables not having to iterate over each point when there
is no change from the previous.

The vsm_set vg pointer points to the last matched segment insertion, and
is used as an optimization when looking for existing entries in the
list. varnishd guarantees that insertions are always ordered, so there is
no need to search the preceeding entries on a successful match. When
parsing an index containg elements that included entries that are added
and then subsequently removed in a later entry, this pointer would be
advanced to the very end, failing to match the entries in between when
parsing the rest of the file.

Fix this mechanism by only updating the pointer on a successful match, and
also advancing it one step it if the entry it is pointing to is removed.

Note that these types of events are not supposed to be possible, because
varnishd will when rewriting the index only include the active ones,
removing any add-then-remove pairs. But if for whatever reason an attempt
is made to reread a list, with this patch it should not cause problems.

Now that we incrementally read the file, the check if the mtime of the
index file has checked has to go away. If not, the index file will always
be marked as new, forcing a reread of the entire index.

Several places in the code lacked checking that these were still valid
objects (ie not free'd).

When a cluster is marked stale, there is no mechanism to actually remove
it once all its containing segments (which would also be marked stale)
goes away.

Fix this by executing vsm_delseg on the cluster in VSM_Unmap if it is marked
stale.

If a segment will be marked stale, it should not have its cluster flag
removed until its actually deleted.

This problem was observed in varnishtest as an assert. What happened was
that a VSM_Map() was executed on a stale segment inside a cluster, which
caused an assert when the cluster segment it pointed to was no longer
marked VSM_FLAG_CLUSTER.

file is recreated by varnishd.

Martin spotted that I lost this in the previous commit.

for clients what the _index entry is about.