- 24 Jul, 2023 19 commits
-
-
Nils Goroll authored
varnish-cache does not touch objects for OA_VARY, but we need to keep FCOs in memory which are frequently used during lookup. Thoughts on why this should not race LRU: - lru_list is owned by lru_mtx - object can't go away, because - for call from hash, we hold the oh->mtx - otherwise, we hold a ref
-
Nils Goroll authored
... which happens potentially under the cache lock
-
Nils Goroll authored
upfront: This is not the segment allocation, which uses parts of the busy obj region allocation, and is mostly motivated by how much data we need to have in RAM at minimum. For the region allocation, we have conflicting goals: - To keep the log short, we want to use the least number of regions - To reduce fragmentation, we want to use the largest possible allocations - To use space efficiently, we want to split regions into power of two allocations. Also, for chunked encoding, we do not have an upper limit of how much space we are going to need, so we have to use the estimate provided by fellow_busy_obj_getspace(). It can not guess more than objsize_max. The new region alloc algorithm takes this compromise: - For the base case that we ran out of available regions (220), we allocate all we need without cramming. - Otherwise if we need less than a chunk, we request it - Otherwise if we know the size, we round down to a power of two - Otherwise we round up We then allow any cramming down to the chunk size, because that is what our LRU reservation uses.
-
Nils Goroll authored
-
Nils Goroll authored
-
Nils Goroll authored
-
Nils Goroll authored
Ref #10
-
Nils Goroll authored
-
Nils Goroll authored
-
Nils Goroll authored
-
Nils Goroll authored
Ref #10
-
Nils Goroll authored
-
Nils Goroll authored
adjust dsk size if mem allocation was smaller than requested
-
Nils Goroll authored
-
Nils Goroll authored
-
Nils Goroll authored
Could have caused #5, related to #10
-
Nils Goroll authored
-
Nils Goroll authored
This is counter-intuitive and could lead to extreme values, for example: default: chunk_exponent = 20, dsk_reserve_chunks 4 adjusted to: 12, 4 << 8 = 1024 now user sets chunk_exponent = 21 adjusted to: 12, 1024 << 9 = 524288 Could have caused #5, related to #10
-
Nils Goroll authored
tiny glitch
-
- 21 Jul, 2023 7 commits
-
-
Nils Goroll authored
The main cause for #11 seems to be that the chunk size in relation to the memory cache was too big. We now clamp it at memsz >> 10 (less than 1/1024 of the memsz). This can still lead to issues when the memory size is reduced and the cache reloaded, but then at least new objects will not compete for the available memory.
-
Nils Goroll authored
-
Nils Goroll authored
-
Nils Goroll authored
Fixes #11 I hope
-
Nils Goroll authored
-
Nils Goroll authored
With the previous commit, we restore the objcore's oa_present field when we first read the object. On top of that, it would be nice if we could restore the field or at least the bit for OA_VARY when we read the log, because a cache lookup checks it and could avoid reading objects if the OA_VARY bit is not set. On the other hand, if vary is used, the object needs to be read anyway, so restoring oa_present during log read only payed off for no vary. The straight forward solution would be to add 2 bytes to struct fellow_dle_obj. This would - require a DLE version bump and compatibility code - bump the dle size from 72 to 80 bytes (because alignment) - consequently, reduce FELLOW_DISK_LOG_BLOCK_ENTRIES from 56 to 4032 / 80 = 50 All this is possible and part of the fellow design, but at this point I conclude that it is not worth the effort. Another option would be to cram two bits into the existing fellow_dle_obj, for example by exploiting the fact that the sign bit of ban and t_origin is always zero. This could break backwards compatibility, so we prepare for the option, but do not implement it. All in all, we just add a TODO to add this when we need to extend the DLE anyway. Closes #17
-
Nils Goroll authored
-
- 20 Jul, 2023 2 commits
-
-
Nils Goroll authored
Implements most of #17, except that we would like to restore OA_VARY with the log read... This should be backwards and forwards compatible, because the default in both cases is 0, and varnish-cache will not use the bitfield if it is 0.
-
Nils Goroll authored
-
- 15 Jul, 2023 2 commits
-
-
Nils Goroll authored
Closes #16
-
Nils Goroll authored
Ref #16
-
- 09 Jul, 2023 8 commits
-
-
Nils Goroll authored
-
Nils Goroll authored
In stvfe_dskoc_fco() we change the oc's stevedore to memstv only _after_ we have taken a refernce via fellow_cache_obj_get() and thus potentially put the fco onto the lru list. This order is important, because other requests racing on stvfe_dskoc_fco() need to also go through fellow_cache_obj_get() until the memstv object has been established for real. Yet this opens a window for a window where an fco is on LRU with oc->stobj->stevedore still set to dskstv, even without dirty reads, and even more so with dirty reads. Because any additional synchronization here could kill performance, we handle this outdated read in stvfe_mutate by not touching the object (yet). It will be hit the next time we iterate the LRU list. Fixes #13, hopefully.
-
Nils Goroll authored
-
Nils Goroll authored
Flexelint does not approve of This reverts commit a1f5225e.
-
Nils Goroll authored
-
Nils Goroll authored
Because of #14 I would like to understand if #13 is really a genuine bug or some odd consequence
-
Nils Goroll authored
-
Nils Goroll authored
This bug was also present in varnish-cache (will document) and found its way into fellow through the api-definition-by-example paradigm of varnish-cache: In varnish-cache vbf_stp_error(), an existing storage object is freed to be replaced with a new one, potentially from a different storage (usually -sTransient). In this case, ObjBocDone is not called, because the boc is to be preserved - from the client's perspective, the backend transaction is still in progress, and the client is waiting for the busy object to finish. At the storage level, however, the process of creation of a new object _is_ finished, and any busy object state which the storage might keep needs to be finalized. So, if oc->boc is not NULL for an ObjFreeObj() call, the stevedore needs to finish any busy object transaction. Fixes #14
-
- 23 Jun, 2023 2 commits
-
-
Nils Goroll authored
-
Nils Goroll authored
-