trackrdrd: added docs about logging and monitoring

trackrdrd/README.rst

@@ -9,7 +9,7 @@ Tracking Log Reader demon
 -------------------------
 
 :Author: Geoffrey Simmons
-:Date:   2013-03-11
+:Date:   2013-03-12
 :Version: 2.0
 :Manual section: 3
 
@@ -215,3 +215,41 @@ once. All other config parameters have default values, and some of
 them correspond to command-line options, as shown below.
 
 .. include:: config.rst
+
+LOGGING AND MONITORING
+======================
+
+By default, ``trackrdrd`` uses ``syslog(3)`` for logging with facility
+``local0`` (unless otherwise specified by configuration as shown
+above). In addition to informational, error and warning messages about
+the running processes, monitoring information is periodically emitted
+to the log (as configured with the parameter
+``monitor.interval``). The monitoring logs have this form (at the
+``info`` log level, with additional formatting of the log lines,
+depending on how syslog is configured)::
+
+ Hash table: len=16384 seen=23591972 drop_reqstart=0 drop_vcl_log=0 drop_reqend=0 expired=0 evacuated=0 open=0 load=0.00 collisions=58534 insert_probes=58974 find_probes=2625 fail=0 occ_hi=591 occ_hi_this=1 
+ Data table: len=116384 nodata=11590516 submitted=12001456 wait_room=0 data_hi=3187 data_overflows=0 done=0 open=0 load=0.00 sent=12001456 reconnects=17 failed=0 occ_hi=4345 occ_hi_this=1 
+
+If monitoring of worker threads is switched on, then monitoring logs
+such as this are emitted for each thread::
+
+ Worker 1 (waiting): seen=576414 waits=86682 sent=576414 reconnects=0 restarts=0 failed=0
+
+The line prefixed by ``Hash table`` describes the hash table for open
+records -- records for request XIDs for which ``ReqStart`` has been
+read, but not yet ``ReqEnd``. The fields ``open``, ``load`` and
+``occ_hi_this`` are gauges (expressing a current state), and
+``occ_hi`` is monotonic increasing; all other fields are cumulative
+counters:
+
+.. include:: hashlog.rst
+
+The line prefixed by ``Data table`` describes the table of request
+records, including records in the open and done states -- for "done"
+records, ``ReqEnd`` has been read for the XID and the record is
+complete, but it has not yet been sent to a message broker. The fields
+``open``, ``done``, ``load`` and ``occ_hi_this`` are gauges, and
+``occ_hi`` is monotonic increasing; the rest are cumulative counters:
+
+.. include:: datalog.rst

trackrdrd/datalog.rst

+================== ============================================================
+Field              Description
+================== ============================================================
+``len``            Size of the data table
+                   (``maxdone`` + 2\ :sup:``maxopen.scale``\)
+------------------ ------------------------------------------------------------
+``nodata``         Number of request records that contained no data (nothing to
+                   track in a ``VCL_log`` entry). These records are discarded
+                   without sending a message to a message broker.
+------------------ ------------------------------------------------------------
+``submitted``      Number of records passed from the reader thread to worker
+                   threads to be sent to a message broker
+------------------ ------------------------------------------------------------
+``wait_room``      How often the reader thread had to wait for space in the
+                   data table
+------------------ ------------------------------------------------------------
+``data_hi``        Data high watermark -- longest record since startup (in
+                   bytes)
+------------------ ------------------------------------------------------------
+``data_overflows`` How often the accumulated length of a record exceeded
+                   ``maxdata``
+------------------ ------------------------------------------------------------
+``done``           Current number of records in state "done"
+------------------ ------------------------------------------------------------
+``open``           Current number of open records in the table
+------------------ ------------------------------------------------------------
+``load``           Current number records in the table as percent
+                   (100 * (``open`` + ``done``)/``len``)
+------------------ ------------------------------------------------------------
+``sent``           Number of records successfully sent to a message broker
+------------------ ------------------------------------------------------------
+``reconnects``     How often worker threads reconnected to a message broker
+                   after an unsuccessful send
+------------------ ------------------------------------------------------------
+``failed``         Number of failed sends (failure after reconnect)
+------------------ ------------------------------------------------------------
+``occ_hi``         Occupancy high watermark -- highest number of records (open
+                   and done) since startup
+------------------ ------------------------------------------------------------
+``occ_hi_this``    Occupancy high watermark in the current monitoring interval
+================== ============================================================
+

trackrdrd/hashlog.rst

+================= =============================================================
+Field             Description
+================= =============================================================
+``len``           Size of the hash table (2\ :sup:``maxopen.scale``\)
+----------------- -------------------------------------------------------------
+``seen``          Number of request records read (``ReqStart`` seen)
+----------------- -------------------------------------------------------------
+``drop_reqstart`` Number of records that could not be inserted into internal
+                  tables (no data from ``ReqStart`` inserted, nor any other
+                  data for that XID)
+----------------- -------------------------------------------------------------
+``drop_vcl_log``  How often data from ``VCL_log`` could not be inserted
+                  (usually because the XID could not be found)
+----------------- -------------------------------------------------------------
+``drop_reqend``   How often data from ``ReqStart`` could not be inserted
+                  (usually because the XID could not be found)
+----------------- -------------------------------------------------------------
+``expired``       Number of records for which ``hash.ttl`` expired (data sent
+                  to message broker without waiting for ``ReqEnd``)
+----------------- -------------------------------------------------------------
+``evacuated``     Number of records removed to recover space in the hash table
+                  (``hash.mlt`` expired, data possibly incomplete)
+----------------- -------------------------------------------------------------
+``open``          Current number of open records in the table
+----------------- -------------------------------------------------------------
+``load``          Current open records as percent (``open``/``len`` * 100)
+----------------- -------------------------------------------------------------
+``collisions``    Number of hash collisions
+----------------- -------------------------------------------------------------
+``insert_probes`` Number of hash insert probes
+----------------- -------------------------------------------------------------
+``find_probes``   Number of hash find probes
+----------------- -------------------------------------------------------------
+``fail``          Number of failed hash operations (insert or find)
+----------------- -------------------------------------------------------------
+``occ_hi``        Occupancy high watermark -- highest number of open records
+                  since startup
+----------------- -------------------------------------------------------------
+``occ_hi_this``   Occupancy high watermark in the current monitoring interval
+================= =============================================================
+