Production system experience with buddy has shown dissatisfactory occupancy
rates with ~20% free storage in page sizes smaller than the chunk size.

It is suspected that this might be related to too aggressive rounddown with
cram=1, so we introduce a tunable to configure the threshold (pivot) at which we
decide to use a crammed vs. an uncrammed allocation.

The default is 1.5 such that allocations which also need the next smaller page
size will not be rounded down. For example, an allocation request for 3072 bytes
(0t110000000000) gets allocated from a 4096 bytes page returning a 1024 bytes
page, while one for 3071 bytes (0t101111111111) gets allocated from a 2048 bytes
page because of the rounddown in the first allocation and a 1024 bytes page
because of the roundup in the second allocation.

It remains to be seen if cram_pivot is helpful and, if so, which value.

Fix regression introduced with 44d5dd9b

Polish the allocation policy once again: For BODY data greater than the chunk
size, pass the original size with the right cram to return no allocation smaller
than the chunk size, but opportunistically make use of available larger pages.

Fixes #96

In SLASH/fellow, we want to minimize the amount of memory needed and thus want
to make objects eligible to LRU eviction whenever possible. The Varnish-Cache
ObjGetAttr() API call requires the returned pointer to remain valid as long as
Varnish-Cache code holds a reference to the object. Therefore, we need to keep
object attributes available in memory for as long as the object is in core. This
is trivial for all the object attributes embedded in struct fellow_disk_obj,
because, by definition, they remain in memory for as long as the object itself
does.

But the auxiliary attributes used for ESI data can be of arbitrary size, which
also does not need to be pre-declared, and are thus managed as a separate
segment.

To keep the API promise, we previously chose a simple approach to never return
references taken on auxiliary attribute segments (of type FCAA), assuming that
FCAA segments would never get LRU'ed individually, but rather only together with
their object.

This was not implemented correctly: If a new ESI data segment was written but
not accessed, it would never gain a reference from ObjGetAttr() and thus end up
on LRU once completely written.

Rather than just fixing this specifically, we make use of an implementation
detail to make FCAA segments LRU'able: We know that Varnish-Cache normally only
issues ObjGetAttr(..., OA_ESIDATA) calls while iterating over the object, and we
already have a pthread key telling us if we are called during an iteration. So,
when iterating over an object, we take/release a reference explicitly, falling
back to the "no return" reference otherwise.

Fixes #95

Ref #95

Background
~~~~~~~~~~

When we receive an object event (OEV_*) for an object which is not in memory,
for example because the ban or timers (ttl etc) have changed, we only need to
submit a new log entry of type DLE_OBJ_CHG to update the object on disk, which
only takes up 72 bytes in the log. Very nice. But if the object is not in
memory, all we know about it is (struct objcore). And because some objects are
not persisted in the log, we need to know if the object even is contained in the
log.

Now one could argue that objects which are not in the log will always be
resident in memory, and that might be true, but the whole object event system is
super racy and making such an assumption could open a can of worms.

So to avoid these races and also be able to add a couple of useful assertions at
other places, we (ab)use one otherwise unused bit in struct objcore, which, so
far, has been the highest bit of the oa_present field.

oa_present is used by varnish-cache to avoid de-tours to the storage engine just
to determine if a certain object attribute is present, in other words, it acts
as an efficient cache of "attribute is set".

The bug
~~~~~~~

Now when we load the storage, we flag all objects as being in the log (which
they are, otherwise they would not get loaded), which makes oa_present be
non-zero (because we (ab)use the highest bit to mark log presence).

When we load the object, we restore the other bits of oa_present to what they
were when the object was written.

So before the object is loaded for the first time, oa_present is non zero, but
not yet correct.

Now when an object is looked up, HSH_Lookup() calls this:

                if (!req->hash_ignore_vary && ObjHasAttr(wrk, oc, OA_VARY)) {
                        vary = ObjGetAttr(wrk, oc, OA_VARY, NULL);

and ObjHasAttr has:

        if (oc->oa_present)
                return (oc->oa_present & (1 << attr));

So because oa_present is not zero but also not yet correct, we return that the
object had no Vary, and, consequently, we deliver the wrong object.

A(nother) hacky solution and a suggestion to solve this sustainably
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To avoid this problem for now, we move the "in log" marker bit to a different,
currently unused bit of (struct objcore).flags, which is equally hacky as using
oa_present, and possibly even worse, because oa_present sill has 8 of 16 bits
currently unused, while (struct objcore).flags has just that _one_ bit currently
unused that we are now "repurposing".

So the proper solution is to get an official place to put our marker bit. This
is https://github.com/varnishcache/varnish-cache/pull/4290

If this PR gets denied, an alternative option is to use one of the lower bits of
the (struct stobj).priv pointer, but that is a more involved change.

Fixes #92

for all but one case we had an LRU change batch open already, and it does not
make sense to apply one batch just to open another.

Regarding the actual LRU change this should not result in reducing the number of
operations, because all cases where fellow_cache_obj_free() is called would
already have removed the fco from LRU.

Also use fellow_cache_obj_lock() / fellow_cache_obj_unlock() for consistency

The idea to embed the fco in the fdo allocation never materialized,

Ref fcf13605

LRU'able FCOs have exactly one reference owned by varnish-cache or, more
specifially, the priv pointer being not NULL.

Now that we take additional references durcing delivery, we avoid having to
reinsert hot objects (which are delivered at least once at any given time) into
the LRU list.

Inspired by #88

so basically this referenecs it while we are delivering the body

There was an implicit assumption that increasing the refcnt would not change
LRU occupancy, and we might want to change that...

Besides getattr calls for OA_VARY, the hash code also runs ban evaluations,
which are equally critical. When we are called from HSH_Lookup(), cur_methods is
still VCL_MET_HASH | 1 from the method call before.

Inspired by #91

When compiling on Centos, following errors appears

```
unknown type name 'uint8_t'
```

Fixing by adding the right import.

(until we learn how to cancel I/O) we need to wait for outstanding I/O anyway
when we slim an object, so do this before we potentially try to free segments.

Fixes #90

Ref 20193268

This usually results from local .rst to .html renders

sizeof is not the correct way to determine the "size" of the struct "header"
before the flexarray, see https://gustedt.wordpress.com/2011/03/14/flexible-array-member/

Coverity complained about an underflow, but it can not know that LRU_NukeOne()
unconditionally decrements wrk->strangelov, so here we know that we will, at
most, increment it again to INT_MAX.

CID#531911

CID#531910

Again it does not understand that the unconditional decrement only applies to
code where it is guaranteed not to underflow: If the value of b was 0, we would
re-set it four lines above, so the underflowed value would never be used.

But anyway ... CID#531907

Yes, formally, the unconditional post-decrement caused an integer underflow, but
it was intentional and without consequence. Coverity does not understand this
pattern...

CID#531903

... understand that there is no overflowed integer used as an array index here.

CID#531909

Motivated by CID#531908, which I am pretty sure is a false positive.

This is utterly unimportant, but adresses a formally correct overity complaint.

CID#531904