...
 
Commits (3)
  • Geoff Simmons's avatar
    Re-consider part of the concept, and refactor some code. · ba5ecac5
    Geoff Simmons authored
    POSIX requires that a file description is referenced by mmap(2),
    so the refcount does not reach 0 when the file is deleted while
    it is still mapped. So POSIX-compliant systems can be expected to
    retain the mapping after file deletion.
    
    Unless the VMOD goes to the trouble of copying the file, the safe
    way to update it is to delete it, then write a file with the same
    name with the new contents. A check may happen to run between
    deletion and the new write. But because of the property mentioned
    above, this is not a real problem for memory-mapped files.
    
    So the VMOD will work according to these rules:
    
    - If a check cannot read a file due to ENOENT, it is not an error.
      (It is an error if the file does not exist on initial read.)
      In that case, the file is considered unchanged -- the current
      mapping continues as the cached file contents.
    
    - This means that for the VMOD, a deleted file is *not* considered
      to be in error, provided it could be read initially (that is, it is
      already mapped).
    
    - We set a flag when the file is deleted, so that the condition
      can be detected.
    
    - Users are advised in the docs that the "delete, then write"
      procedure is the *only* method for updating the file that the
      VMOD supports. Other methods may work, some of the time, but
      you do it differently at your own risk.
    
    Also, access(2) is not a reliable means to determine if the file
    can be read and mapped, because if checks permissions against
    the real UID/GID, whereas open(2) depends on effective UID/GID.
    
    So:
    
    - checks begin with open(2), then use fstat(2) for the stat check.
    
    - Path searches are done by attempting open(2) with the filename
      on each directory in the path.
    ba5ecac5
  • Geoff Simmons's avatar
    Add reader.deleted(). · 613201e9
    Geoff Simmons authored
    613201e9
  • Geoff Simmons's avatar
    Add the "File deletion and file updates" section. · e4e31613
    Geoff Simmons authored
    And update related parts of the docs.
    e4e31613
......@@ -88,16 +88,13 @@ subsequent accesses via the reader object in VCL::
set req.http.Myfile = rdr.get();
}
Changed file contents may in fact become visible immediately, before
the TTL elapses; but that is platform-dependent (see the discussion
below).
The content cache takes the form of a memory-mapping of the file, see
``mmap(2)``. This has some consequences that are discussed further
below.
Since the update checks run in the background, the file I/O that the
checks require is not incurred during any request/response
transaction. The I/O effort to read the contents may also happen in
the background, or during the first access after initialization, or
after the file has changed; that too is platform-dependent (see
below).
transaction.
.. _vcl.state: https://varnish-cache.org/docs/trunk/reference/varnish-cli.html#vcl-state-configname-auto-cold-warm
......@@ -117,6 +114,58 @@ the VCL had previously been cold), then the files are immediately
checked for changes, updating the cached contents if necessary, and
the update checks in the background resume at the TTL interval.
File deletion and file updates
------------------------------
POSIX mandates that mmap(2) adds a reference for the file, which is
not removed until the file is unmapped. In particular, it is not
removed when the file is deleted -- the mapping continues to access
the file's contents, even after deletion. (In that case, the file is
not physically removed, but is no longer accessible by name in the
filesystem.)
.. |.deleted()| replace:: ``.deleted()``
For this reason, if an update check finds that the file has been
deleted, *it is not considered an error*, provided that the file has
already been mapped. (It is an error if the file does not exist at
initialization.) The file is considered unchanged, and the cached
contents remain valid, at least until the next check. The
|.deleted()|_ method of the `reader object`_ can be used in VCL to
detect this situation.
POSIX leaves unspecified whether changes in the underlying file
immediately become visible in the memory mapping. On a system like
Linux, changes are immediately visible, and hence will be reflected
immediately be the VMOD. While this may seem ideal for getting fast
updates in VCL, it is in fact problematic:
* File writes are not atomic; so the VMOD may return partial and
inconsistent contents for the file.
* If the changed file is longer than the originally mapped file, the
portion that is longer than the original file is not
mapped. Contents returned by the VMOD will appear truncated.
For these reasons, this is a reliable method to update a file:
* Delete the file
* Write the new contents to a new file of the same name (same path
location)
This is the *only* method for updating files that the VMOD supports.
After the deletion step, the previously cached contents remain valid.
When the next update check detects the change performed by the second
step, the new contents are mapped, and become available in their
correct form via the VMOD.
Other means of updating the file might "happen" to work, some of the
time. But if not, it is not considered a bug of the VMOD. The VMOD
works as designed *only* if the two-step procedure for updating files
is followed.
.. _reader object:
.. _file.reader():
......@@ -170,12 +219,12 @@ fails with a message describing the error. If the read succeeds, then
the file contents are cached, and are available via the reader
object's methods.
The content cache takes the form of a memory-mapping of the file, see
``mmap(2)``.
If initialization succeeds and ``ttl`` > 0s, then update checks begin
at that interval. A file is considered to have changed if any of its
``stat(2)`` fields ``mtime``, ``dev`` or ``ino`` change. If the file
``stat(2)`` fields ``mtime``, ``dev`` or ``ino`` change. As discussed
above, the file is considered unchanged if the update check finds the
the file has been deleted, provided that it has already been mapped;
then the previously cached contents continue to be valid. If the file
has changed when a check is performed, it is re-read and the new
contents are cached, for access via the object's methods.
......@@ -306,6 +355,28 @@ Example::
call do_file_error_handling;
}
.. _.deleted():
.. _xreader.deleted():
BOOL xreader.deleted()
----------------------
Return true if and only if the file was found to have been deleted the
last time the file was checked.
As discussed in `File deletion and file updates`_ above, this is not
an error condition, if the file had been previously mapped. Then the
previously cached contents continue to be valid.
Example::
import std;
if (rdr.deleted()) {
std.log("file deleted, continuing with the current cached contents");
}
.. _xreader.size():
BYTES xreader.size()
......@@ -399,15 +470,18 @@ implement different error handling in VCL.
Errors that may be encountered on the initial read or update checks
include:
* The ``stat(2)`` call to read file meta-data fails. This is what will
happen for typical file errors: when the file has been deleted, the
Varnish process cannot access it, or the process owner does not have
read permissions.
* The file cannot be opened for read. This is what will happen for
typical file errors: the Varnish process cannot access the file, or
the process owner does not have read permissions.
* The file does not exist at initialization time. As discussed above,
this is not an error for an update check, if the file has already
been mapped.
* The file is neither a regular file nor a symbolic link that points
to a regular file.
* Any of the internal calls to open and map the file fail.
* Any of the internal calls to map the file fail.
REQUIREMENTS
============
......
......@@ -31,7 +31,7 @@ client c1 {
expect resp.http.Errmsg == "No error"
} -run
shell {rm -f ${tmpdir}/error}
shell {chmod a-r ${tmpdir}/error}
delay .1
client c2 {
......@@ -39,10 +39,13 @@ client c2 {
rxresp
expect resp.status == 200
expect resp.http.Error == "true"
expect resp.http.Errmsg ~ {^vmod file failure: vcl1\.rdr: cannot read info about}
expect resp.http.Errmsg ~ {^vmod file failure: vcl1\.rdr: cannot open}
} -run
shell {touch ${tmpdir}/error}
shell {
rm -f ${tmpdir}/error
touch ${tmpdir}/error
}
delay .1
client c1 -run
......@@ -92,10 +92,10 @@ logexpect l1 -v v1 -d 1 -g raw -q "Debug" {
expect * * Debug {^vmod file: vcl3\.rdr: check for \S+ finished successfully at}
} -run
shell {rm -f ${tmpdir}/file}
shell {chmod a-r ${tmpdir}/file}
delay .1
client c1 {
client c2 {
txreq
rxresp
expect resp.status == 503
......@@ -104,14 +104,29 @@ client c1 {
logexpect l1 -v v1 -d 1 -g vxid -q "VCL_Error" {
expect 0 * Begin req
expect * = VCL_Error {^rdr\.get\(\): vmod file failure: vcl3\.rdr: cannot read info about}
expect * = VCL_Error {^rdr\.get\(\): vmod file failure: vcl3\.rdr: cannot open}
expect * = End
} -run
logexpect l1 -v v1 -d 1 -g raw -q "Error" {
expect * 0 Error {^vmod file failure: vcl3\.rdr: cannot read info about}
expect * 0 Error {^vmod file failure: vcl3\.rdr: cannot open}
} -run
shell {
rm -f ${tmpdir}/file
echo -n "quux baz bar foo" > ${tmpdir}/file
}
delay .1
client c1 -run
shell {rm -f ${tmpdir}/file}
delay .1
# The file is deleted, but the mapping is retained (until munmap), so
# the client will still see the mapped contents.
client c1 -run
shell {echo -n "The quick brown fox jumps over the lazy dog." > ${tmpdir}/fox}
varnish v1 -vcl {
......@@ -136,26 +151,30 @@ client c1 {
txreq
rxresp
expect resp.status == 200
expect resp.http.Get ~ "fox"
expect resp.http.Get == "The quick brown fox jumps over the lazy dog."
} -run
shell {echo -n "twentieth century fox" > ${tmpdir}/fox}
shell {
rm -f ${tmpdir}/fox
echo -n "twentieth century fox" > ${tmpdir}/fox
}
delay .1
# Whether or not the client sees the change, although no update checks
# are performed, depends on whether changes in MAP_PRIVATE mapped
# files are reflected in the mapped page. According to mmap(2), this
# is unspecified. The change is apparently seen on Linux. Check the
# test log to see if it happened on the present platform.
client c1 -run
client c1 {
txreq
rxresp
expect resp.status == 200
expect resp.http.Get == "The quick brown fox jumps over the lazy dog."
} -run
shell {rm -f ${tmpdir}/fox}
delay .1
varnish v1 -errvcl {cannot open} {
import ${vmod_file};
backend b { .host = "${bad_ip}"; }
# On Linux at least, the mapping is retained after file deletion
# (until munmap), so the client will still see the mapped contents.
# XXX test on other platforms
client c1 -run
sub vcl_init {
new rdr = file.reader("${tmpdir}/nosuchfile");
}
}
varnish v1 -errvcl {not a regular file} {
import ${vmod_file};
......
......@@ -17,6 +17,7 @@ varnish v1 -vcl {
set req.http.Mtime = rdr.mtime();
set req.http.Delta-Mtime = now - rdr.mtime();
set req.http.Next-Check = rdr.next_check();
set req.http.Deleted = rdr.deleted();
return (synth(200));
}
......@@ -25,6 +26,7 @@ varnish v1 -vcl {
set resp.http.Mtime = req.http.Mtime;
set resp.http.Delta-Mtime = req.http.Delta-Mtime;
set resp.http.Next-Check = req.http.Next-Check;
set resp.http.Deleted = req.http.Deleted;
return (deliver);
}
} -start
......@@ -40,6 +42,7 @@ client c1 {
expect resp.http.Delta-Mtime < 1
expect resp.http.Next-Check >= 0
expect resp.http.Next-Check <= 0.1
expect resp.http.Deleted == "false"
} -run
shell {echo -n "foo" > ${tmpdir}/sz}
......@@ -55,9 +58,10 @@ client c1 {
expect resp.http.Delta-Mtime < 1
expect resp.http.Next-Check >= 0
expect resp.http.Next-Check <= 0.1
expect resp.http.Deleted == "false"
} -run
shell {rm -f ${tmpdir}/sz}
shell {chmod a-r ${tmpdir}/sz}
delay .1
client c1 {
......@@ -69,7 +73,7 @@ client c1 {
logexpect l1 -v v1 -d 1 -g vxid -q "VCL_Error" {
expect 0 * Begin req
expect * = VCL_Error {^rdr\.size\(\): vmod file failure: vcl1\.rdr: cannot read info about}
expect * = VCL_Error {^rdr\.size\(\): vmod file failure: vcl1\.rdr: cannot open}
expect * = End
} -run
......@@ -94,12 +98,12 @@ varnish v1 -vcl {
}
}
shell {rm -f ${tmpdir}/mtime}
shell {chmod a-r ${tmpdir}/mtime}
delay .1
logexpect l1 -v v1 -d 0 -g vxid -q "VCL_Error" {
expect 0 * Begin req
expect * = VCL_Error {^rdr\.mtime\(\): vmod file failure: vcl2\.rdr: cannot read info about}
expect * = VCL_Error {^rdr\.mtime\(\): vmod file failure: vcl2\.rdr: cannot open}
expect * = End
} -start
......@@ -129,7 +133,7 @@ varnish v1 -vcl {
}
}
shell {rm -f ${tmpdir}/nxtchk}
shell {chmod a-r ${tmpdir}/nxtchk}
delay .1
# next_check() is not affected by errors.
......@@ -141,3 +145,36 @@ client c1 {
expect resp.http.Next-Check >= 0
expect resp.http.Next-Check <= 0.1
} -run
shell {touch ${tmpdir}/deleteme}
varnish v1 -vcl {
import ${vmod_file};
backend b { .host = "${bad_ip}"; }
sub vcl_init {
new rdr = file.reader("${tmpdir}/deleteme", ttl=0.1s);
}
sub vcl_recv {
return (synth(200));
}
sub vcl_synth {
set resp.http.Error = rdr.error();
set resp.http.Deleted = rdr.deleted();
return (deliver);
}
}
shell {rm -f ${tmpdir}/deleteme}
delay .1
# .error() == false, since deleted files are not considered in error.
client c1 {
txreq
rxresp
expect resp.status == 200
expect resp.http.Error == "false"
expect resp.http.Deleted == "true"
} -run
......@@ -39,7 +39,7 @@ client c1 {
expect resp.body == "quux baz bar foo"
} -run
shell {rm -f ${tmpdir}/synth}
shell {chmod a-r ${tmpdir}/synth}
delay .1
client c1 {
......@@ -49,11 +49,14 @@ client c1 {
logexpect l1 -v v1 -d 1 -g vxid -q "VCL_Error" {
expect 0 * Begin req
expect * = VCL_Error {^rdr\.synth\(\): vmod file failure: vcl1\.rdr: cannot read info about}
expect * = VCL_Error {^rdr\.synth\(\): vmod file failure: vcl1\.rdr: cannot open}
expect * = End
} -run
shell {touch ${tmpdir}/synth}
shell {
rm -f ${tmpdir}/synth
touch ${tmpdir}/synth
}
varnish v1 -vcl {
import ${vmod_file};
......
......@@ -74,6 +74,7 @@ struct file_info {
#define RDR_ERROR (1 << 1)
#define RDR_MAPPED (1 << 2)
#define RDR_TIMER_INIT (1 << 3)
#define RDR_DELETED (1 << 4)
struct VPFX(file_reader) {
unsigned magic;
......@@ -104,7 +105,7 @@ check(union sigval val)
struct VPFX(file_reader) *rdr;
struct file_info *info;
struct stat st;
int fd;
int fd = -1;
void *addr;
char timbuf[VTIM_FORMAT_SIZE];
int err;
......@@ -127,7 +128,24 @@ check(union sigval val)
AZ(pthread_rwlock_wrlock(&rdr->lock));
errno = 0;
if (stat(info->path, &st) != 0) {
if ((fd = open(info->path, O_RDONLY)) < 0) {
if (errno == ENOENT && (rdr->flags & RDR_MAPPED) != 0) {
rdr->flags |= RDR_DELETED;
VSL(SLT_Debug, 0, "vmod file: %s.%s: %s is deleted but "
"already mapped", rdr->vcl_name, rdr->obj_name,
info->path);
goto out;
}
VERRMSG(rdr, "%s.%s: cannot open %s: %s", rdr->vcl_name,
rdr->obj_name, info->path, vstrerror(errno));
VSL(SLT_Error, 0, rdr->errbuf);
rdr->flags |= RDR_ERROR;
goto out;
}
rdr->flags &= ~RDR_DELETED;
errno = 0;
if (fstat(fd, &st) != 0) {
VERRMSG(rdr, "%s.%s: cannot read info about %s: %s",
rdr->vcl_name, rdr->obj_name, info->path,
vstrerror(errno));
......@@ -170,15 +188,6 @@ check(union sigval val)
}
rdr->flags &= ~RDR_MAPPED;
errno = 0;
if ((fd = open(info->path, O_RDONLY)) < 0) {
VERRMSG(rdr, "%s.%s: cannot open %s: %s", rdr->vcl_name,
rdr->obj_name, info->path, vstrerror(errno));
VSL(SLT_Error, 0, rdr->errbuf);
rdr->flags |= RDR_ERROR;
goto out;
}
/*
* By mapping the length st_size + 1, and due to the fact that
* mmap(2) fills the region of the mapped page past the length of
......@@ -194,10 +203,8 @@ check(union sigval val)
rdr->obj_name, info->path, vstrerror(errno));
VSL(SLT_Error, 0, rdr->errbuf);
rdr->flags |= RDR_ERROR;
closefd(&fd);
goto out;
}
closefd(&fd);
AN(addr);
rdr->flags |= RDR_MAPPED;
......@@ -232,6 +239,9 @@ check(union sigval val)
out:
AZ(pthread_rwlock_unlock(&rdr->lock));
if (fd != -1)
closefd(&fd);
if ((rdr->flags & RDR_ERROR) == 0 && info->log_checks) {
VTIM_format(VTIM_real(), timbuf);
VSL(SLT_Debug, 0, "vmod file: %s.%s: check for %s "
......@@ -313,6 +323,7 @@ vmod_reader__init(VRT_CTX, struct VPFX(file_reader) **rdrp,
else {
struct vsb *search;
char *end, delim = ':';
int fd = -1;
AZ(info->path);
if (path == NULL || *path == '\0') {
......@@ -333,12 +344,13 @@ vmod_reader__init(VRT_CTX, struct VPFX(file_reader) **rdrp,
VSB_putc(search, '/');
VSB_cat(search, name);
VSB_finish(search);
if (access(VSB_data(search), R_OK) != 0)
if ((fd = open(VSB_data(search), O_RDONLY)) < 0)
continue;
info->path = malloc(VSB_len(search) + 1);
if (info->path == NULL) {
VSB_destroy(&search);
closefd(&fd);
VFAIL(ctx, "new %s: allocating path", vcl_name);
return;
}
......@@ -346,6 +358,8 @@ vmod_reader__init(VRT_CTX, struct VPFX(file_reader) **rdrp,
break;
}
VSB_destroy(&search);
if (fd != -1)
closefd(&fd);
if (info->path == NULL) {
VFAIL(ctx, "new %s: %s not found or not readable on "
"path %s", vcl_name, name, path);
......@@ -413,7 +427,7 @@ vmod_reader__init(VRT_CTX, struct VPFX(file_reader) **rdrp,
AZ(rdr->addr);
AZ(rdr->info->mtime.tv_sec);
AZ(rdr->info->mtime.tv_nsec);
AZ(rdr->flags & (RDR_INITIALIZED | RDR_ERROR));
AZ(rdr->flags & (RDR_INITIALIZED | RDR_ERROR | RDR_DELETED));
do {
VTIM_sleep(INIT_SLEEP_INTERVAL);
} while ((rdr->flags & (RDR_INITIALIZED | RDR_ERROR)) == 0);
......@@ -425,6 +439,7 @@ vmod_reader__init(VRT_CTX, struct VPFX(file_reader) **rdrp,
}
AN(rdr->flags & RDR_MAPPED);
AZ(rdr->flags & RDR_DELETED);
AN(rdr->addr);
AN(rdr->info->mtime.tv_sec);
AN(rdr->info->mtime.tv_nsec);
......@@ -550,6 +565,15 @@ vmod_reader_errmsg(VRT_CTX, struct VPFX(file_reader) *rdr)
return (rdr->errbuf);
}
VCL_BOOL
vmod_reader_deleted(VRT_CTX, struct VPFX(file_reader) *rdr)
{
CHECK_OBJ_NOTNULL(rdr, FILE_READER_MAGIC);
(void)ctx;
return (rdr->flags & RDR_DELETED);
}
VCL_BYTES
vmod_reader_size(VRT_CTX, struct VPFX(file_reader) *rdr)
{
......
......@@ -84,16 +84,13 @@ subsequent accesses via the reader object in VCL::
set req.http.Myfile = rdr.get();
}
Changed file contents may in fact become visible immediately, before
the TTL elapses; but that is platform-dependent (see the discussion
below).
The content cache takes the form of a memory-mapping of the file, see
``mmap(2)``. This has some consequences that are discussed further
below.
Since the update checks run in the background, the file I/O that the
checks require is not incurred during any request/response
transaction. The I/O effort to read the contents may also happen in
the background, or during the first access after initialization, or
after the file has changed; that too is platform-dependent (see
below).
transaction.
.. _vcl.state: https://varnish-cache.org/docs/trunk/reference/varnish-cli.html#vcl-state-configname-auto-cold-warm
......@@ -113,6 +110,58 @@ the VCL had previously been cold), then the files are immediately
checked for changes, updating the cached contents if necessary, and
the update checks in the background resume at the TTL interval.
File deletion and file updates
------------------------------
POSIX mandates that mmap(2) adds a reference for the file, which is
not removed until the file is unmapped. In particular, it is not
removed when the file is deleted -- the mapping continues to access
the file's contents, even after deletion. (In that case, the file is
not physically removed, but is no longer accessible by name in the
filesystem.)
.. |.deleted()| replace:: ``.deleted()``
For this reason, if an update check finds that the file has been
deleted, *it is not considered an error*, provided that the file has
already been mapped. (It is an error if the file does not exist at
initialization.) The file is considered unchanged, and the cached
contents remain valid, at least until the next check. The
|.deleted()|_ method of the `reader object`_ can be used in VCL to
detect this situation.
POSIX leaves unspecified whether changes in the underlying file
immediately become visible in the memory mapping. On a system like
Linux, changes are immediately visible, and hence will be reflected
immediately be the VMOD. While this may seem ideal for getting fast
updates in VCL, it is in fact problematic:
* File writes are not atomic; so the VMOD may return partial and
inconsistent contents for the file.
* If the changed file is longer than the originally mapped file, the
portion that is longer than the original file is not
mapped. Contents returned by the VMOD will appear truncated.
For these reasons, this is a reliable method to update a file:
* Delete the file
* Write the new contents to a new file of the same name (same path
location)
This is the *only* method for updating files that the VMOD supports.
After the deletion step, the previously cached contents remain valid.
When the next update check detects the change performed by the second
step, the new contents are mapped, and become available in their
correct form via the VMOD.
Other means of updating the file might "happen" to work, some of the
time. But if not, it is not considered a bug of the VMOD. The VMOD
works as designed *only* if the two-step procedure for updating files
is followed.
.. _reader object:
$Object reader(PRIV_VCL, STRING name,
......@@ -156,12 +205,12 @@ fails with a message describing the error. If the read succeeds, then
the file contents are cached, and are available via the reader
object's methods.
The content cache takes the form of a memory-mapping of the file, see
``mmap(2)``.
If initialization succeeds and ``ttl`` > 0s, then update checks begin
at that interval. A file is considered to have changed if any of its
``stat(2)`` fields ``mtime``, ``dev`` or ``ino`` change. If the file
``stat(2)`` fields ``mtime``, ``dev`` or ``ino`` change. As discussed
above, the file is considered unchanged if the update check finds the
the file has been deleted, provided that it has already been mapped;
then the previously cached contents continue to be valid. If the file
has changed when a check is performed, it is re-read and the new
contents are cached, for access via the object's methods.
......@@ -280,6 +329,25 @@ Example::
call do_file_error_handling;
}
.. _.deleted():
$Method BOOL .deleted()
Return true if and only if the file was found to have been deleted the
last time the file was checked.
As discussed in `File deletion and file updates`_ above, this is not
an error condition, if the file had been previously mapped. Then the
previously cached contents continue to be valid.
Example::
import std;
if (rdr.deleted()) {
std.log("file deleted, continuing with the current cached contents");
}
$Method BYTES .size()
Return the size of the file as currently cached. Invokes VCL failure
......@@ -361,15 +429,18 @@ implement different error handling in VCL.
Errors that may be encountered on the initial read or update checks
include:
* The ``stat(2)`` call to read file meta-data fails. This is what will
happen for typical file errors: when the file has been deleted, the
Varnish process cannot access it, or the process owner does not have
read permissions.
* The file cannot be opened for read. This is what will happen for
typical file errors: the Varnish process cannot access the file, or
the process owner does not have read permissions.
* The file does not exist at initialization time. As discussed above,
this is not an error for an update check, if the file has already
been mapped.
* The file is neither a regular file nor a symbolic link that points
to a regular file.
* Any of the internal calls to open and map the file fail.
* Any of the internal calls to map the file fail.
REQUIREMENTS
============
......