documentation

README.rst

@@ -2,75 +2,447 @@
 DCS Classifier
 ==============
 
-*TODO* fill in the skeleton
-
 --------------------------------------------
 Varnish Device Classification Service Module
 --------------------------------------------
 
 :Manual section: 3
 :Authors: Nils Goroll
-:Date: 2014-03-03
+:Date: 2014-04-13
 :Version: 0.1
 
 SYNOPSIS
 ========
 
+Command line
+------------
+
 ::
 
-        import dcs;
-        
+	<prefix>/bin/dcs
+
+Varnish VMOD
+------------
+
+VCL:
+
+::
+
+	import dcs [from "path"] ;
+
+	# typical use
+	sub vcl_recv {
+		set req.http.x-nb-classified = dcs.type_name(dcs.classify());
+		# - or-
+		set req.http.x-variant	     = dcs.type_mtd(dcs.classify());
+		# ...
+	}
+
+Varnish 2 inline-C
+------------------
+
+Varnish start:
+
+::
+
+	varnishd "-pcc_command=\"exec gcc -I<prefix>/share ...\""
+
+VCL:
+
+::
+
+	C{
+	#include "dcs_varnish2.c"
+	}C
+
+	sub vcl_recv {
+	    C{ dcs_varnish2_classify_hdrs(sp); }C
+	    ...
+	}
+
 
 DESCRIPTION
 ===========
 
+This Varnish module provides an efficient implementation of device
+detection and classification using the downloadable version of the
+Netbiscuits Device Classifier Service (DCS) database.
 
-FUNCTIONS
-=========
+The DCS database is not part of this module and needs to be obtained
+from Netbiscuits, please refer to
+http://www.netbiscuits.com/device-detection/ as a starting point.
+
+PERFORMANCE
+-----------
 
-Example VCL::
+This module was developed to provide exceptional performance compared
+to previous implementations without requiring any changes to the
+structure of the database or introducing any changes to the
+semantics.
 
-	backend foo { ... };
+All lookups are uncached and lookup complexity does not depend on the
+position of the best match in the dcs database.
 
-	import dcs;
+To achieve high performance, C code for a custom parser for all tokens
+(substrings) from the DCS database is generated. The parser is run to
+detect all tokens from the User-Agent, marking potential matches. As
+the match result, the DCS database entry which comes first in the
+database is returned.
 
-classify()
+Exemplary benchmarks on a single i7-4600M core @2.9 GHz max suggest
+that detection throughput exceeds 200.000 matches per second, which
+corresponds to a latency in the order of 5us (5 microseconds).
+
+INTERFACES
 ----------
 
-Prototype
-        ::
+The following use cases are supported
+
+* Use as a Varnish 3 module
+* Use with Varnish 2 as inline-C
+* Command line tools
+
+PREREQUISITES
+=============
+
+The following tools are required on the system where this module is to
+be build.
+
+When building from the source repository:
+
+* libtool
+* autoconf 2.69 or later
+* automake
+
+For all builds:
+
+* A working C99 compatible compiler. gcc 4 is tested, other gcc
+  versions and clang should work
+
+* A working build environment with all standard headers and tools
+  (e.g. make)
+
+* perl
+
+* The following perl modules:
 
-Returns
+  * `Crypt::RC4`
 
-Description
+  * `Digest::MD5`
 
-Example
-        ::
+  * `MIME::Base64`
+
+To build the documentation
+
+* rst2man
+
+To build the varnish module (VMOD)
+
+* the source tree of the Varnish version the VMOD is to be used
+  with. Varnish must have been configured and compiled in this source
+  tree.
+* python2 or higher
+
+BUILDING
+========
+
+A minimal build will only provide the command line tools. It requires
+the follwing steps:
+
+1. Generate `configure` and automake files (Only when building from the
+   source repository):
+
+::
+
+	sh autogen.sh
+
+2. Run `configure` to use either an existing database file or download
+from the netbiscuits website (standard `configure` arguments are also
+supported). In both cases the licence key from Netbiscuits is
+required:
+
+2a. Existing database file
+
+::
+
+	sh configure DCS_KEY=<key> DCS_DBFILE=<file>
+
+2b. Download with a https://my.netbiscuits.com/ account
+
+::
+
+	sh configure DCS_KEY=<key> DCS_ACCOUNT=<account-name>
+
+3. Run `make`
+
+4. Optionally run `make install` to install at the default location
+   (normally `/usr/local`) or in the prefix specified by the `--prefix`
+   argument to `configure`.
+
+Building the varnish module (vmod)
+----------------------------------
+
+To build the vmod, in addition to the `configure` arguments given
+above, the `VARNISHSRC` argument must be used as in
+
+::
+
+	sh configure DCS_KEY=<key> DCS_ACCOUNT=<account-name> \
+	   VARNISHSRC=/path/to/your/varnish/source/varnish-cache
+
+Optionally, a custom vmod installation directory can be specified
+using `VMODDIR=<dir>`
+
+When building the vmod, an additional
+
+::
+
+	make check
+
+step is recommended to run the bundled `varnishtest` tests.
+
+Building for use with Varnish 2
+-------------------------------
+
+To install all files required to use the DCS module with Varnish 2
+inline-C, use the additional `configure` argument `--enable-varnish2`
+
+With this argument, `make install` will create additional source files
+in the `share` directory of the installation prefix.
+
+Optimizing the DCS database
+---------------------------
+
+The optional `make fixup` before calling `make` will remove entries
+from the DCS database which will never be hit and will reorder entries
+which are likely to be unintentionally masked by previous entries.
+
+Use `make clean` before `make fixup` if the code has already been build.
+
+
+.. _detection_methodology:
+
+DETECTION METHODOLOGY
+=====================
+
+The following applies to the `classify()` function of the Varnish Module and Varnish 2
+inline-C. The `dcs` command line tool only implements the last step.
+
+* If the `x-wap-profile header` is present, the User-Agent will be
+  classified as a mobile phone
+
+* If the `X-OperaMini-Phone-UA` header is present, the string " opera/
+  opera mini/  " gets appended to the `User-Agent` header for
+  classification.
+
+* The contents of the headers `X-OperaMini-Phone-UA`,
+  `X-Device-User-Agent`, `X-Original-User-Agent` and `X-Goog-Source`
+  are appended to the `User-Agent` header for classification.
+
+* The enrichted `User-Agent` string is passed to the DCS classifer and
+  the matching dcs db entry is returned - or a special db entry named
+  "unidentified".
+
+VMOD USAGE - FUNCTIONS
+======================
+
+To import the vmod, use
+
+::
+	import dcs [from "path"] ;
 
-INSTALLATION
-============
+.. _func_classify:
 
-Installation requires the Varnish source tree (only the source matching the
-binary installation).
+INT classify()
+--------------
 
-1. `./autogen.sh`  (for git-installation)
-2. `./configure VARNISHSRC=/path/to/your/varnish/source/varnish-cache`
-3. `make`
-4. `make install` (may require root: sudo make install)
-5. `make check` (Optional for regression tests)
+Runs the :ref:detection_methodology as described.
 
-VARNISHSRCDIR is the directory of the Varnish source tree for which to
-compile your vmod. Both the VARNISHSRCDIR and VARNISHSRCDIR/include
-will be added to the include search paths for your module.
+The return value is the index of the DCS DB entry.
 
-Optionally you can also set the vmod install dir by adding VMODDIR=DIR
-(defaults to the pkg-config discovered directory from your Varnish
-installation).
+This vmod function should be used as an argument to one of the
+functions described below.
 
+As each invocation runs the classifcation again,
+it should only be used once per request.
+
+Example:
+
+::
+
+	set req.http.x-nb-classified = dcs.type_name(dcs.classify());
+
+.. _func_entry_key:
+
+STRING entry_key(INT)
+---------------------
+
+Returns the key of the dcs db entry whose index is given as the integer argument.
+
+Example:
+
+::
+
+	set req.http.xx-entry-key = dcs.entry_key(dcs.classify());
+
+Might set `xx-entry-key` to something like "android*opera mini/"
+
+.. _func_type_id:
+
+INT type_id(INT)
+----------------
+
+Returns the internal type id of the dcs db entry whose index is given as the integer argument.
+
+Example:
+
+::
+
+	set req.http.xx-type-id = dcs.type_id(dcs.classify());
+
+Might set `xx-type-id` to "11"
+
+
+.. _func_type_name:
+
+STRING type_name(INT)
+---------------------
+
+Returns the type name of the dcs db entry whose index is given as the integer argument.
+
+Example:
+
+::
+
+	set req.http.x-nb-classified  = dcs.type_name(dcs.classify());
+
+might set `x-nb-classified` to "Mobile Phone"
+
+.. _func_type_mtd:
+
+STRING type_mtd(INT)
+--------------------
+
+Returns one of three meta type names "mob" (for mobile), "tab" (for
+tablet) or "dsk" (for desktop) for the dcs db entry whose index is given
+as the integer argument.
+
+Example:
+
+::
+
+	set req.http.x-variant        = dcs.type_mtd(dcs.classify());
+
+might set `x-variant` to "mob"
+
+
+COMMAND LINE USAGE
+==================
+
+dcs
+---
+
+This command line tool reads one User-Agent string per input line
+until EOF is reached and outputs the respective classifcation in the
+following format:
+
+::
+
+	--
+	<input-line lowercase>
+	entry id <entry-id> type <type_id> - <type_mdt> - <type_name>
+
+
+VARNISH 2 USAGE
+===============
+
+To use this module with Varnish 2, the classifier code and some vmod
+glue code need to be compiled as inline-C. To do so, the CC command
+executed by VCC needs to be modified such that the dcs source code can
+be found.
+
+In the following section, `$PREFIX` needs to be replaced by the
+installation prefix passed to `configure` using the `--prefix` command
+or the default value `/usr/local`:
+
+* Determine the `cc_command` of the running Varnish instance
+
+::
+
+	varnishadm -T ... param.show cc_command
+
+* Add the following to the `cc_command` after the compiler name
+  (usually "gcc"):
+
+::
+
+	-I$PREFIX/share
+
+  and add the resulting `cc_command` as a paramter to the varnish
+  start script.
+
+  Example: If `cc_command` is
+
+::
+
+	"exec gcc -std=gnu99  -pthread -fpic -shared -Wl,-x -o %o %s"
+
+  then add the following to the Varnish start parameters:
+
+::
+
+	-pcc_command="exec gcc -I$PREFIX/share -std=gnu99  -pthread -fpic -shared -Wl,-x -o %o %s"
+
+To change a running varnish instance without a restart, `varnishadm`
+can be used.
+
+Once the new `cc_command` is active, the following can be used in VCL:
+
+::
+
+	C{
+	#include "dcs_varnish2.c"
+	}C
+
+	sub vcl_recv {
+	    C{
+		dcs_varnish2_classify_hdrs(sp);
+	    }C
+
+	    # ....
+	}
+
+This will make the following headers available in `vcl_recv` after the
+call to `dcs_varnish2_classify_hdrs`:
+
+* `req.http.x-nb-classified`
+  same as :ref:`func_type_name` e.g. "Mobile Phone"
+
+* `req.http.x-variant`
+  same as :ref:`func_type_mtd` e.g. "mob"
+
+
+NOTES
+=====
+
+dynamic linking
+---------------
+
+The varnsh dcs vmod is always compiled to be self-contained, it does
+not link dynamically to `libdcs.so`. This is done for two reasons:
+
+1. To avoid the performance penalty of library calls
+
+2. To avoid version conflicts between `libdcs.so` and the rest of
+   `libvmod_dcs.so` which could happen when a `libdcs.so` generated
+   from one DCS database is used with `libvmod_dcs.so` generated from
+   another
 
 ACKNOWLEDGEMENTS
 ================
 
+Development of this module was sponsored by Deutsche Telekom AG - Products & Innovation
+
 HISTORY
 =======
 
@@ -79,7 +451,7 @@ Version 0.1: Initial version, mostly feature-complete
 BUGS
 ====
 
-No bugs at all!
+None known
 
 SEE ALSO
 ========