fixes https://github.com/varnishcache/varnish-cache/issues/3958

Ref #15

and check that the setting is actually applied (it is).

Now that only the top thread delivers, we can remove the
support for multiple unpending and delivering threads.

This concludes the fix for !11

Some bits were already implemented in previous commits.

See https://github.com/varnishcache/varnish-cache/issues/3938 for why
this took so long.

Also implemented a fix for https://github.com/varnishcache/varnish-cache/issues/3939
(which is not yet in varnish-cache)

In order to properly handle vped_include() failing, we need a way to
insert a node only when it's ready and free it when not.

Thus, we link to the parent in node_new(), but do not insert the
child to the parent's list. As before, this happens in node_insert.

On the way, we also add node_free() to free a node without finalizing
it.

Partly fixes !11

Related prep-work for !6

found by flexelint

Now that we got rid of front unpending, we can also remove all
the complications due to the ST_OPEN state.

Begin with removing that state itself.

It was effectively removed in 913c4653 but
dead code was left until now.

found by flexelint

Found by flexelint

Ever since the first release of vmod_pesi, we knew that this feature
was probably not useful: As explained in the THREADS section of the
vcc / man page, we can not push to VDPs, so the only case where this
could work was when there are no VDPs. The only case pESI itself does
not need any is non-esi, non-gzip uncacheable streaming. Also, the
only case where it made a significant difference from pushing from the
level 0 / front thread was when there are no threads available and the
front thread runs the current include.

Since then, we never encountered a situation where we would have
needed this feature.

Now we can keep references also to private leaf objects.

For ESI objects, which we also need to inspect when building the
delivery tree, we still need to make copies in pesi_buf_bytes because,
if the final flag to ObjIterate() is set, the storage engine can (and
usually will) free segments "behind" delivery.

vmod_pesi works by saving the resulting data from a sub request to a
tree structure, which gets delivered to the client in the top
request's thread, once it is ready.

For cacheable objects which do not require ESI processing, we simply
keep the original request with an additional reference to the object.
So basically we hand delivery from one worker to another.

subreq_fixup() is responsible for converting the saved request to a
state as if it was handled by the request handling the top level
request, so one of the changes it applies is to change the wrk pointer
to the worker of the top level request.

Yet that change was incomplete and we missed an additional pointer in
struct vdp_ctx.

This should hopefully fix #14

AM_CFLAGS are set by the recursive make, but it's CFLAGS
which we need to carry over.

A follow-up issue has been reported in #13:

Assert error in Lck_Delete(), cache/cache_lck.c line 309:
  Condition((pthread_mutex_destroy(&ilck->mtx)) == 0) not true.

triggered from Lck_Delete(&bytes_tree->nodes_lock) at the bottom of
bytes_tree_fini().

Assuming everything else working correctly, the only scenario I
can see the moment is that we see the node->subreq.done == 1
earlier than Lck_Unlock() returned in vped_task(). In this
case, we could advance to destroying the lock while the other
thread still holds it.

The other use case of the shared lock is in fini_final(), where
we already go through an explicit lock/unlock.

Hopefully fixes #13 for real

The BackendReuse tag no longer exists.

This looks like a trivial oversight in 3370a4c4,
which was meant to fix #10: We need to use the corresponding lock to signal
when a subreq node is done.

Hopefully fixes #13

Avoids "Illegal :path pseudo-header" errors