req->err_code and req->err_reason are set when going to synthetic
handling. From there the resp.reason HTTP field is set from
req->err_reason if set, or the generic code based on req->err_code is used
if it was NULL. This patch clears these members so that a value from the
handling of a previous request doesn't linger.

Fixes: VSV00004

Ref previous commit to remove the use of kill(signull) on the pids of the
Varnish processes to test liveness.

A bug was uncovered in the VSM code that checks if kill(SIGNULL) would
work as a test of liveness of the master and worker processes. If the user
running the utility has the required permissions to send signal to the
worker process, but not the management process, the code would wrongly
assume it could do kill on both. It would then end up in a connect loop
never succeeding.

Because kill() can not always be successfully run (a common scenario when
the user running varnishlog is not the same UID as the cache processes),
there is a fall back to using fstat to check for Varnish restarts. Since
this fallback is active for most use cases anyways, it was decided to
retire the kill() mechanism rather than to fix it. This way the behaviour
does not change depending on what user the utility is run as.

This change was OK'd by PHK after discussing it on IRC.

Clusters VSM allocations start at offset 16. Make sure that the published
segment includes that adjustment.

This speeds up vsm_findseg by keeping a direct pointer to the struct
vsm_seg it points to in the vsm_fantom.

To prevent stale vsm_fantoms in an application from causing segfaults, the
last serial number seen is also kept on the fantom. If this serial number
does not match, the slow path of iterating all segs to find the right
match is done, and the fantom is updated to make any subsequent calls on
the same fantom take the fast path.

With this patch the fantoms store 2 serial numbers and a direct pointer to
the seg. To fit this in struct vsm_fantom without breaking the ABI, the
unused chunk pointer value has been repurposed, giving us two uintptr_t
priv variables to work with. The direct segment pointer occupies one, and
the two serial numbers occupy one half each of the other. This limits the
bits for each serial to only 16 bits on 32 bit systems. But only direct
matches is done, and it is unlikely that a stale fantom should have the
exact right value to be accepted as valid.

When upgrading from when a previous release that doesn't have the latest
VSM index file changes, the utilities will assert when trying to parse the
index file from the previous version. Fix that by ignoring lines not
starting with either '#', '+' or '-'.

This moves the exposed flag handling to the segment level, rather than the
point level. This enables not having to iterate over each point when there
is no change from the previous.

The vsm_set vg pointer points to the last matched segment insertion, and
is used as an optimization when looking for existing entries in the
list. varnishd guarantees that insertions are always ordered, so there is
no need to search the preceeding entries on a successful match. When
parsing an index containg elements that included entries that are added
and then subsequently removed in a later entry, this pointer would be
advanced to the very end, failing to match the entries in between when
parsing the rest of the file.

Fix this mechanism by only updating the pointer on a successful match, and
also advancing it one step it if the entry it is pointing to is removed.

Note that these types of events are not supposed to be possible, because
varnishd will when rewriting the index only include the active ones,
removing any add-then-remove pairs. But if for whatever reason an attempt
is made to reread a list, with this patch it should not cause problems.

Now that we incrementally read the file, the check if the mtime of the
index file has checked has to go away. If not, the index file will always
be marked as new, forcing a reread of the entire index.

Several places in the code lacked checking that these were still valid
objects (ie not free'd).

When a cluster is marked stale, there is no mechanism to actually remove
it once all its containing segments (which would also be marked stale)
goes away.

Fix this by executing vsm_delseg on the cluster in VSM_Unmap if it is marked
stale.

If a segment will be marked stale, it should not have its cluster flag
removed until its actually deleted.

This problem was observed in varnishtest as an assert. What happened was
that a VSM_Map() was executed on a stale segment inside a cluster, which
caused an assert when the cluster segment it pointed to was no longer
marked VSM_FLAG_CLUSTER.

file is recreated by varnishd.

Martin spotted that I lost this in the previous commit.

for clients what the _index entry is about.

segments dominate the _.index file.

6233b088
4d8e4b9c
c7267466
cda19210
dee47cc4
e34de5c1
a0ad9025
54be487b
1dc891a5
0582753d
78694521
3c04f5c7
f2357c88
740ee39c
cb1b5899
458306fa
0e32f166
9e1e0972
a82d8552
4bbf6df6
d8421dcd
3c8ecb4c
a0b8a734
8c5118d1
3d170c0d
d869640b
1506c5cf
5ecbde38
e1a1fdc7
64c0e1ae
93e6dd70
f3e9ca6a
07ad2fb8Co-authored-by: Dag Haavi Finstad <daghf@varnish-software.com>
Co-authored-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>
Co-authored-by: Federico G. Schwindt <fgsch@lodoss.net>
Co-authored-by: Nils Goroll <nils.goroll@uplex.de>
Co-authored-by: Poul-Henning Kamp <phk@FreeBSD.org>
Co-authored-by: Shohei Tanaka(@xcir) <kokoniimasu+git@gmail.com>

0ad32622Co-authored-by: Poul-Henning Kamp <phk@FreeBSD.org>

b23dddd3Co-authored-by: Guillaume Quintard <guillaume@varnish-software.com>

This makes waitinglist rushes happen also in HSH_Lookup when encountering
cache hits. This helps to get the requests on the waitinglist restarted
when doing streaming delivery. Fixes #2977. Backported from @mbgrydeland
master commits.

Based on: 37368496

93d80501 made execution of
ban_lurker_test_ban() conditional on bd != b, which effectively caused
objects hanging off bans below request bans to not get tested against
relevant bans.

Because object bans (from the obans list) are being marked completed,
the objects which were skipped would also be missed to get evaluated
against the relevant bans at lookup time unless they were evaluated in
request context. So, in effect, we would simply miss to test bans.

Fixes #3007

Maybe related to #3006