It can definitely be used in third party VMOD projects.

vcc_compile.h is generated and included by other compilation units, so
we need to make sure that this happens in sequence.

Seen on vtest:

make[5]: Entering directory
'/tmp/vtest.32_sun12.4/varnish-cache/lib/libvcc'
  CC       libvcc_a-vcc_acl.o
  CC       libvcc_a-vcc_action.o
  CC       libvcc_a-vcc_backend.o
  CC       libvcc_a-vcc_backend_util.o
  CC       libvcc_a-vcc_compile.o
  CC       libvcc_a-vcc_expr.o
mkdir -p ../../include/tbl
/opt/local/bin/python3.4 ../../lib/libvcc/generate.py \
    ../.. ../..
  CC       libvcc_a-vcc_parse.o
  CC       libvcc_a-vcc_storage.o
  CC       libvcc_a-vcc_symb.o
"vcc_compile.h", line 194: undefined symbol: VCL_RET_MAX
"vcc_compile.h", line 194: can not declare variably modified type at file scope
cc: acomp failed for vcc_parse.c
Makefile:668: recipe for target 'libvcc_a-vcc_parse.o' failed

it seems I wrongly assumed that SIZE_MAX would be defined as a positive
integer, in fact it looks like (size_t)-1 was used.

Ref: 2d890940

Have seen ExpKill interfere with the Debug messages in vtest

The great witness test introduced by @bsdphk and reworked relatively
recently by @Dridi reported a hypothetical lock cycle here.

This was not a real issue, because the scheduling code would not re-enter
the probe code, yet we might still drop the probe lock while
scheduling.

At this point I have no idea what I'm doing and no motivation to learn
further YAML so I'll do something I hate: trigger a CI build and resume
what I was working on until it completes.

Refs 0c645dbf

(see previous)

We used lck_backend both for vbp_mtx and for the mtx on individual
backends.

This lead to confusing results of the brinch hansons arrows produced by
make witness.

spotted by @Dridi

This reverts commit 0c3ec169.

as decided by bugwash

Ref #3218

this might need more love in places, but I wanted to get my first
pass done and pushed before an ICE train arrives in Hamburg.

I suggest we aim for a complete and correct changelog first before we
rewrite/reformat it for the release- and upgrading notes.

Please review and feel free to fix/improve or add additional changes
which I left out or might have missed.

The longest part is the test suite, so in order to release build
machines sooner I hope that 8 concurrent jobs will strick a good
balance between speed and resource consumption, considering that
test cases mostly wait.

That is still below what we do for Circle CI builds with 12 jobs
for distcheck tasks.

went through commits and dropped notes in running order, stopped at
f402e5ff for now (had to switch tasks)

Better late than never.

Closes #3238

The havemsg variable is purely local, and only written-then-read once.

Previously, we only checked `v1l->deadline` (which gets initialized
from the `send_timeout` session attribute or parameter) only for short
writes, so for successful "dripping" http1 writes (streaming from a
backend busy object with small chunks), we did not respect the
timeout.

This patch restructures `V1L_Flush()` to always check the deadline
before a write by turning a `while() { ... }` into a `do { ... }
while` with the same continuation criteria: `while (i != v1l->liov)`
is turned into `if (i == v1l->liov) break;` and `while (i > 0 || errno
== EWOULDBLOCK)` is kept to retry short writes.

This also reduces the `writev()` call sites to one.

Fixes #3189

As @Dridi and myself concluded, the send_timeout had no effect on
backend connections anyway because we never set SO_SNDTIMEO (aka
idle_send_timeout on the client side) on backend connections.

With the next commit, we will fix the send_timeout on the client side and
thus would also enable it for "dripping" writes on the backend side.

To preserve existing behavior for the time being, we explicitly disable
the timeout (actually deadline) on the backend side. There is ongoing
work in progress to rework all of our timeouts for 7.x.

Implementation note: if (VTIM_real() > v1l->deadline) evaluates to false
for v1l->deadline == NaN

Ref #3189

and check our WS_Snapshot() / WS_Reset() debug messages

We use workspace overflows to signal to bail out for example after a
failing `VRT_SetHdr()`. This is a guarantee that if some serious issue
occurred during processing, we rather send an error downstream than an
incomplete response or the result of incomplete processing.

We use the `WS_Snapshot() ...  WS_Reset()` pattern as some kind of
second order workspace allocation where the called code itself uses
`WS_Reserve()`.

With this usage pattern, `WS_Reset()` called `ws_ClearOverflow(ws)`,
potentially clearing the overflow bit from a previous relevant
failure.

We now avoid any other unintended clears of the overflow bit by
splitting two functions:

* WS_Rollback() is now what WS_Reset() used to be: It clears overflows
  and accepts the zero cookie for a reset-to-start

  It is only intended for use within varnishd and is thus declared
  in cache_varnishd.h

* WS_Reset() does not touch the overflow bit any longer, ensuring that
  a once-overflowed workspace stays overflowed

`WS_Snapshot()` now returns a magic value which gets recognized by
`WS_Reset()` to ensure that the overflowed marker is still present.
This serves two purposes:

- better debugging and

- a safety measure against passing a cookie from an already overflowed
  workspace to WS_Rollback()

Fixes #3194