trackrdrd: expanded documentation in README.rst and added config.rst

trackrdrd/README.rst

@@ -9,8 +9,8 @@ Tracking Log Reader demon
 -------------------------
 
 :Author: Geoffrey Simmons
-:Date:   2012-09-23
-:Version: 0.1
+:Date:   2013-03-11
+:Version: 2.0
 :Manual section: 3
 
 
@@ -23,8 +23,8 @@ DESCRIPTION
 ===========
 
 The ``trackrdrd`` demon reads from the shared memory log of a running
-instance of Varnish and collects data relevant to tracking for the
-Otto project.
+instance of Varnish, collects data relevant to tracking for the Otto
+project, and forwards the data to ActiveMQ message brokers.
 
 OPTIONS
 =======
@@ -40,11 +40,16 @@ The source repository for ``trackrdrd`` is in the subdirectory
 	git@repo.org:trackrdrd
 
 The build requires a source directory for Varnish in which sources
-have been compiled. Varnish sources with features added for Otto are
-in::
+have been compiled. Varnish sources with custom features for Otto
+are in::
 
 	git@repo.org:varnish-cache
 
+``trackrdrd`` must link with the CMS or ActiveMQ-CPP library
+(``libactivemq-cpp``) at runtime. The sources can be obtained from::
+
+        http://activemq.apache.org/cms/
+
 Building Varnish
 ----------------
 
@@ -77,6 +82,15 @@ build::
 	$ CFLAGS=-m64 ./configure
 	$ make
 
+Building and installing ActiveMQ-CPP
+------------------------------------
+
+``trackrdrd`` has been tested with versions 3.4.4 and 3.5.0 of
+ActiveMQ-CPP. If the library ``libactivemq-cpp`` is already installed
+on the platform where ``trackrdrd`` will run, then no further action
+is necessary. To build the library from source, follow the
+instructions in the ``README.txt`` file of its source distribution.
+
 Building and installing trackrdrd
 ---------------------------------
 
@@ -87,14 +101,19 @@ on the same platform as the Varnish build, all requirements are
 fulfilled.)
 
 The steps to build ``trackrdrd`` are very similar to those for
-building Varnish. The only difference is that in the ``configure``
-step, the path to the Varnish source directory must be given in the
-variable ``VARNISHSRC``::
+building Varnish. The only difference is in the ``configure``
+step:
+
+* The path to the Varnish source directory must be given in the variable ``VARNISHSRC``.
+* The flag ``CXXFLAGS``, like ``CFLAGS``, must also be set to ``-m64``, because C++ code is also compiled. It may be necessary to add additional ``CXXFLAGS`` to compile the ActiveMQ API calls, for example as obtained from ``pkg-config --cflags apr-1``.
+
+At minimum, run these steps::
 
 	$ git clone git@repo.org:trackrdrd
 	$ cd trackrdrd/trackrdrd/
 	$ ./autogen.sh
-	$ CXXFLAGS=-m64 CFLAGS=-m64 ./configure VARNISHSRC=/path/to/varnish-cache
+	$ CXXFLAGS=-m64 CFLAGS=-m64 ./configure \\
+          VARNISHSRC=/path/to/varnish-cache
 	$ make
 
 For self-tests after the build, run::
@@ -118,23 +137,27 @@ be shown with::
 For example, to specify a non-standard installation prefix, add the
 ``--prefix`` option::
 
-	$ CFLAGS=-m64 CXXFLAGS=-m64 ./configure VARNISHSRC=/path/to/varnish-cache \\
+	$ CFLAGS=-m64 CXXFLAGS=-m64 ./configure \\
+          VARNISHSRC=/path/to/varnish-cache \\
 	  --prefix=/path/to/varnish_tracking
 
-Developers can add a number of options as an aid to compiling and debugging::
+For Otto, runtime paths for Varnish libraries are at non-standard
+locations, so it is necessary to add the option
+``LDFLAGS=-Wl,-rpath=$LIB_PATHS``::
 
-	$ CFLAGS=-m64 CXXFLAGS=-m64 ./configure VARNISHSRC=/path/to/varnish-cache \\
-          --enable-developer-warnings --enable-debugging-symbols \\
-          --enable-extra-developer-warnings --enable-werror
+        $ export VARNISH_PREFIX=/path/to/varnish
+	$ CFLAGS=-m64 CXXFLAGS=-m64 ./configure \\
+          VARNISHSRC=/path/to/varnish-cache \\
+	  --prefix=/path/to/varnish_tracking \\
+          LDFLAGS=-Wl,-rpath=$VARNISH_PREFIX/lib/varnish:$VARNISH_PREFIX/lib
 
-``--enable-werror`` activates the ``-Werror`` option for compilers,
-which causes compiles to fail on any warning. ``trackrdrd`` should
-*always* build successfully with this option.
+Developers can add a number of options as an aid to compiling and debugging::
 
-``--enable-developer-warnings`` and
-``--enable-extra-developer-warnings`` turn on additional compiler
-switches for warnings -- ``trackrdrd`` builds should succeed with
-these as well.
+	$ CFLAGS=-m64 CXXFLAGS=-m64 ./configure \\
+          VARNISHSRC=/path/to/varnish-cache \\
+          --enable-debugging-symbols --enable-werror \\
+          --enable-developer-warnings --enable-extra-developer-warnings \\
+          --enable-diagnostics
 
 ``--enable-debugging-symbols`` ensures that symbols and source code
 file names are saved in the executable, and thus are available in core
@@ -143,8 +166,52 @@ forth. It is advisable to turn this switch on for production builds
 (not just for developer builds), so that runtime errors can more
 easily be debugged.
 
-If any of the needed requirements (for instance, the ActiveMQ C++ library)
-have been installed into non-default locations, PKG_CONFIG_PATH should
-be set to point to the appropriate pkg-config directories, like in this example::
+``--enable-werror`` activates the ``-Werror`` option for compilers,
+which causes compiles to fail on any warning. ``trackrdrd`` should
+*always* build successfully with this option.
+
+``--enable-developer-warnings``, ``--enable-extra-developer-warnings``
+and ``--enable-diagnostics`` turn on additional compiler switches for
+errors and warnings. ``trackrdrd`` builds should succeed with these as
+well.
+
+It may be necessary to set ``PKG_CONFIG_PATH`` to point to the
+appropriate pkg-config directories, if any of the needed requirements
+(such as the ActiveMQ C++ library) have been installed into
+non-default locations, as in this example::
 
 	$ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./configure #...
+
+CONFIGURATION
+=============
+
+As mentioned above for command-line option ``-c``, configuration values
+are read in this hierarchy:
+
+1. ``/etc/trackrdrd.conf``, if it exists and is readable
+2. a config file specified with the ``-c`` option
+3. config values specified with other command-line options
+
+If the same config parameter is specified in one or more of these
+sources, then the value at the "higher" level is used. For example, if
+``varnish.name`` is specified in both ``/etc/trackrdrd.conf`` and a
+``-c`` file, then the value from the ``-c`` file is used, unless a
+value is specified with the ``-n`` option, in which case that value is
+used.
+
+The syntax of a configuration file is simply::
+
+        # comment
+        <param> = <value>
+
+The ``<value>`` is all of the data from the first non-whitespace
+character after the equals sign up to the last non-whitespace
+character on the line. Comments begin with the hash character and
+extend to the end of the line. There are no continuation lines.
+
+The parameters ``mq.uri`` and ``mq.qname`` are required (have no
+default values), and (only) ``mq.uri`` may be specified more than
+once. All other config parameters have default values, and some of
+them correspond to command-line options, as shown below.
+
+.. include:: config.rst

trackrdrd/config.rst

+==================== ========== ========================================================================================= =======
+Parameter            CLI Option Description                                                                               Default
+==================== ========== ========================================================================================= =======
+``varnish.name``     ``-n``     Like the ``-n`` option for Varnish, this is the path to the file that is mmap'd to the    default for Varnish (the host name)
+                                shared memory segment for the Varnish log. This parameter and ``varnish.bindump`` are
+                                mutually exclusive.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``mq.uri``                      URIs for the message broker. This is the only parameter that can appear more than once in None, this parameter is required.
+                                the configuration; if more than one MQ URIs are specified, than worker threads
+                                distribute their connections to the different message brokers.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``mq.qname``                    Name of the queue (destination) to which messages are sent at the message broker(s).      None, this parameter is required.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``nworkers``                    Number of worker threads used to send messages to the message broker(s).                  1
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``maxopen.scale``               log\ :sub:`2`\(max number of concurrent requests in Varnish). For example, if             10 (= 1024 concurrent requests)
+                                ``maxopen.scale`` = 10, then ``trackrdrd`` can support up to 1024 concurrent requests.
+                                More precisely, this number describes the maximum number of request XIDs for which
+                                ``ReqStart`` has been read, but not yet ``ReqEnd``. It should specify at least the next
+                                power of two larger than (``thread_pools`` * ``thread_pool_max``) in the Varnish
+                                configuration.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``maxdone``                     The maximum number of finished records waiting to be sent to message brokers. That is,    1024
+                                the largest number of request XIDs for which ``ReqEnd`` has been read, but the data have
+                                not yet been sent to a message queue.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``maxdata``                     The maximum length of a data record in characters. Should be at least as large the        1024
+                                Varnish parameter ``shm_reclen``.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``hash.max_probes``             The maximum number of insert or find probes used for the hash table of XIDs. Hash lookups 10
+                                fail if a hit is not found after this many probes.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``hash.ttl``                    Maximum time to live in seconds for an unfinished record. If ``ReqEnd`` is not read for   120
+                                a request XID within this time, then ``trackrdrd`` no longer waits for it, and schedules
+                                the data read thus far to be sent to a message broker. This should be a bit longer than
+                                the sum of all timeouts configured for a Varnish request.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``hash.mlt``                    Minimum lifetime of an open record in seconds. That is, after ``ReqStart`` has been read  5
+                                for a request XID, then ``trackrdrd`` will not evacuate it if space is needed in its hash
+                                table before this interval has elapsed.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``qlen.goal``                   A goal length for the internal queue from the reader thread to the worker thread.         ``maxdone``/2
+                                ``trackrdrd`` uses this value to determine whether a new worker thread should be started
+                                to support increasing load.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``user``             ``-u``     Owner of the child process                                                                ``nobody``, or the user starting ``trackrdrd``
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``pid.file``         ``-P``     Path to the file to which the management process writes its process ID. If the value is   ``/var/run/trackrdrd.pid``
+                                set to be empty (by the line ``pid.file=``, with no value), then no PID file is written.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``restarts``                    Maximum number of restarts of the child process by the management process                 1
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``monitor.interval``            Interval in seconds at which monitoring statistics are emitted to the log. If set to 0,   30
+                                then no statistics are logged.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``monitor.workers``             Whether statistics about worker threads should be logged (boolean)                        false
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``log.file``         ``-l``     Log file for status, warning, debug and error messages, and monitoring statistics. If '-' ``syslog(3)``
+                                is specified, then log messages are written to stdout. This parameter and
+                                ``syslog.facility`` are mutually exclusive.
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``syslog.facility``  ``-y``     See ``syslog(3)``; legal values are ``user`` or ``local0`` through ``local7``. This       ``local0``
+                                parameter and ``log.file`` are mutually exclusive. 
+-------------------- ---------- ----------------------------------------------------------------------------------------- -------
+``varnish.bindump``  ``-f``     A binary dump of the Varnish shared memory log obtained from ``varnishlog -w``. If a
+                                value is specified, ``trackrdrd`` reads from that file instead of a live Varnish log
+                                (useful for testing, debugging and replaying traffic). This parameter and
+                                ``varnish.name`` are mutually exclusive. 
+==================== ========== ========================================================================================= =======
+
+