==============
DCS Classifier
==============

--------------------------------------------
Varnish Device Classification Service Module
--------------------------------------------

:Manual section: 3
:Authors: Nils Goroll
:Date: 2017-11-20
:Version: 0.6

SYNOPSIS
========

Command line
------------

::

	<prefix>/bin/dcs

Varnish VMOD (Varnish 3 and later)
----------------------------------

VCL:

::

	import dcs [from "path"] ;

	# typical use
	sub vcl_recv {
		set req.http.x-nb-classified = dcs.type_name(dcs.classify());
		# - or-
		set req.http.X-DeviceClass   = dcs.type_class(dcs.classify());
		# ...
	}

Varnish 2 inline-C
------------------

Varnish start:

::

	varnishd "-pcc_command=\"exec gcc -I<prefix>/share ...\""

VCL:

::

	C{
	#include "dcs_varnish2.c"
	}C

	sub vcl_recv {
	    C{ dcs_varnish2_classify_hdrs(sp); }C
	    ...
	}


DESCRIPTION
===========

This Varnish module provides an efficient implementation of device
detection and classification using the downloadable version of the
Netbiscuits Device Classifier Service (DCS) database. or a
self-provided database. An example database is included.

Netbiscuits Device Classifier Service (DCS) database
----------------------------------------------------

The DCS database is not part of this module and needs to be obtained
from Netbiscuits, please refer to
http://www.netbiscuits.com/device-detection/ as a starting point. With
sufficient privileges, a classifier token can be created on
https://my.netbiscuits.com/ under Account -> Token Management. See
http://kb.netbiscuits.com/dcs/dcs_ui_tokenmanagement.html for
instructions.

The classifier token is also referred to as DCS_KEY below.

Demo Database file
------------------

For demonstration purposes, we provice a simple database file with
some minimal and incomplete classification information in
`src/dcs_demo.db`. See :ref:using_the_demo_db for details.

Meta Classes
------------

Classification types from the database file can be associated with
meta-classes in the file `src/classes.conf`. Its format is

::

	[classname]
	Typename from the database

Note that the bundled tests need entries from the bundled
classes.conf.

During the build process, `gen_dcs_classifier.pl` emits warnings if
entries are missing from the classes configuration or if entries
remain unused. It may be advisable to update the configuration when
these warnigs are seen.


PERFORMANCE
-----------

This module was developed to provide exceptional performance compared
to previous implementations without requiring any changes to the
structure of the database or introducing any changes to the
semantics.

All lookups are uncached and lookup complexity does not depend on the
position of the best match in the dcs database.

To achieve high performance, C code for a custom parser for all tokens
(substrings) from the DCS database is generated. The parser is run to
detect all tokens from the User-Agent, marking potential matches. As
the match result, the DCS database entry which comes first in the
database is returned.

Exemplary benchmarks on a single i7-4600M core @2.9 GHz max suggest
that detection throughput exceeds 200.000 matches per second, which
corresponds to a latency in the order of 5us (5 microseconds).

INTERFACES
----------

The following use cases are supported

* Use as a Varnish module (vmod) for varnish 3 and later
* Use with Varnish 2 as inline-C
* Command line tools

PREREQUISITES
=============

The following tools are required on the system where this module is to
be build.

When building from the source repository:

* `libtool`
* `autoconf` 2.69 or later
* `automake`

For all builds:

* A working C99 compatible compiler. gcc 4 is tested, other gcc
  versions and clang should work

* A working build environment with all standard headers and tools
  (e.g. `make`)

* `perl`

* The following perl modules:

  * `Crypt::RC4`

  * `Digest::MD5`

  * `MIME::Base64`

To download the dcs DB (through the `DCS_ACCOUNT` parameter to `configure`)

* `curl`

To build the documentation

* `rst2man`

To build the varnish module (VMOD)

* For Varnish 4 and later

  A full installation of Varnish to be used with the VMOD.

* For Varnish 3:

  The source tree of the Varnish version the VMOD is to be used
  with. Varnish must have been configured and compiled in this source
  tree.

* python2 or higher

BUILDING
========

A minimal build will only provide the command line tools. It requires
the follwing steps:

1. Generate `configure` and automake files (Only when building from the
   source repository):

::

	sh autogen.sh

2. Run `configure` to use either an existing database file or download
from the netbiscuits website (standard `configure` arguments are also
supported). In both cases the classifier token from Netbiscuits is
required:

2a. Existing database file

::

	sh configure DCS_KEY=<key> DCS_DBFILE=<file>

2b. Download with a https://my.netbiscuits.com/ account

::

	sh configure DCS_KEY=<key> DCS_ACCOUNT=<account-name>

.. _using_the_demo_db:

2c. Use the bundled demo Database.

    We provide a very simple database for demonstration purposes.

::

   sh configure DCS_KEY=demo DCS_DBFILE=dcs_demo.db

2d. Optionally add dbfiles to prepend and append

    To override / amend entries from the main ``DCS_DBFILE``,
    additional files can be speficied as ``DCS_DBFILE_PRE`` (to
    prepend / override) and ``DCS_DBFILE_POST``. These files should
    not be encrypted, otherwise the same encryption key must be used.

::

   sh configure DCS_KEY=demo DCS_DBFILE=dcs_demo.db \
		DCS_DBFILE_PRE=/path/to/pre.db \
		DCS_DBFILE_POST=/path/to/post.db


3. Run `make`

4. Optionally run `make install` to install at the default location
   (normally `/usr/local`) or in the prefix specified by the `--prefix`
   argument to `configure`.

Building the varnish module (vmod) for Varnish 4 and higher
-----------------------------------------------------------

If varnish is installed in a standard directory, generic :ref:BUILDING
as explained above should also build the vmod.

If you have installed Varnish to a non-standard directory, call
``autogen.sh`` and ``configure`` with ``PKG_CONFIG_PATH`` pointing to
the appropriate path. For example, when varnishd configure was called
with ``--prefix=$PREFIX``, use

::

   PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig
   export PKG_CONFIG_PATH

When building the vmod, an additional

::

	make check

step is recommended to run the bundled `varnishtest` tests.

Building the varnish module (vmod) for Varnish 3
------------------------------------------------

To build the vmod for varnish 3, in addition to the `configure`
arguments given above, the `VARNISHSRC` argument must be used as in

::

	sh configure DCS_KEY=<key> DCS_ACCOUNT=<account-name> \
	   VARNISHSRC=/path/to/your/varnish/source/varnish-cache

Optionally, a custom vmod installation directory can be specified
using `VMOD_DIR=<dir>`


Building for use with Varnish 2
-------------------------------

To install all files required to use the DCS module with Varnish 2
inline-C, use the additional `configure` argument `--enable-varnish2`

With this argument, `make install` will create additional source files
in the `share` directory of the installation prefix.

Optimizing the DCS database
---------------------------

The optional `make fixup` before calling `make` will remove entries
from the DCS database which will never be hit and will reorder entries
which are likely to be unintentionally masked by previous entries.

Use `make clean` before `make fixup` if the code has already been build.


.. _detection_methodology:

DETECTION METHODOLOGY
=====================

The following applies to the `classify()` function of the Varnish Module and Varnish 2
inline-C. The `dcs` command line tool only implements the last step.

* If the `x-wap-profile header` is present, the User-Agent will be
  classified as a mobile phone

* If the `X-OperaMini-Phone-UA` header is present, the string " opera/
  opera mini/  " gets appended to the `User-Agent` header for
  classification.

* The contents of the headers `X-OperaMini-Phone-UA`,
  `X-Device-User-Agent`, `X-Original-User-Agent` and `X-Goog-Source`
  are appended to the `User-Agent` header for classification.

* The enrichted `User-Agent` string is passed to the DCS classifer and
  the matching dcs db entry is returned - or a special db entry named
  "unidentified".

VMOD USAGE - FUNCTIONS
======================

To import the vmod, use

::

	import dcs [from "path"] ;

.. _func_classify:

INT classify()
--------------

Runs the :ref:detection_methodology as described.

The return value is the index of the DCS DB entry.

This vmod function should be used as an argument to one of the
functions described below.

As each invocation runs the classifcation again,
it should only be used once per request.

Example:

::

	set req.http.x-nb-classified = dcs.type_name(dcs.classify());

.. _func_entry_key:

STRING entry_key(INT)
---------------------

Returns the key of the dcs db entry whose index is given as the integer argument.

Example:

::

	set req.http.xx-entry-key = dcs.entry_key(dcs.classify());

Might set `xx-entry-key` to something like "android*opera mini/"

.. _func_type_id:

INT type_id(INT)
----------------

Returns the internal type id of the dcs db entry whose index is given as the integer argument.

Example:

::

	set req.http.xx-type-id = dcs.type_id(dcs.classify());

Might set `xx-type-id` to "11"


.. _func_type_name:

STRING type_name(INT)
---------------------

Returns the type name of the dcs db entry whose index is given as the integer argument.

Example:

::

	set req.http.x-nb-classified  = dcs.type_name(dcs.classify());

might set `x-nb-classified` to "Mobile Phone"

.. _func_type_class:

STRING type_class(INT)
----------------------

Returns one of the meta types defined in `src/classes.conf`

Example:

::

	set req.http.X-DeviceClass    = dcs.type_class(dcs.classify());

might set `X-DeviceClass` to "smartphone"


COMMAND LINE USAGE
==================

dcs
---

This command line tool reads one User-Agent string per input line
until EOF is reached and outputs the respective classifcation in the
following format:

::

	--
	<input-line lowercase>
	entry id <entry-id> type <type_id> - <type_class> - <type_name>


VARNISH 2 USAGE
===============

To use this module with Varnish 2, the classifier code and some vmod
glue code need to be compiled as inline-C. To do so, the CC command
executed by VCC needs to be modified such that the dcs source code can
be found.

In the following section, `$PREFIX` needs to be replaced by the
installation prefix passed to `configure` using the `--prefix` command
or the default value `/usr/local`:

* Determine the `cc_command` of the running Varnish instance

::

	varnishadm -T ... param.show cc_command

* Add the following to the `cc_command` after the compiler name
  (usually "gcc"):

::

	-I$PREFIX/share

  and add the resulting `cc_command` as a paramter to the varnish
  start script.

  Example: If `cc_command` is

::

	"exec gcc -std=gnu99  -pthread -fpic -shared -Wl,-x -o %o %s"

  then add the following to the Varnish start parameters:

::

	-pcc_command="exec gcc -I$PREFIX/share -std=gnu99  -pthread -fpic -shared -Wl,-x -o %o %s"

To change a running varnish instance without a restart, `varnishadm`
can be used.

Once the new `cc_command` is active, the following can be used in VCL:

::

	C{
	#include "dcs_varnish2.c"
	}C

	sub vcl_recv {
	    C{
		dcs_varnish2_classify_hdrs(sp);
	    }C

	    # ....
	}

This will make the following headers available in `vcl_recv` after the
call to `dcs_varnish2_classify_hdrs`:

* `req.http.x-nb-classified`
  same as :ref:func_type_name e.g. "Mobile Phone"

* `req.http.X-DeviceClass`
  same as :ref:func_type_class e.g. "desktop"


NOTES
=====

dynamic linking
---------------

The varnsh dcs vmod is always compiled to be self-contained, it does
not link dynamically to `libdcs.so`. This is done for two reasons:

1. To avoid the performance penalty of library calls

2. To avoid version conflicts between `libdcs.so` and the rest of
   `libvmod_dcs.so` which could happen when a `libdcs.so` generated
   from one DCS database is used with `libvmod_dcs.so` generated from
   another

cURL
----

`curl` uses environment variables like `http_proxy`. If they do not pass through
the Makefiles (as with some versions of `make`),
`curl` can be configured using `~/.curlrc`.


ACKNOWLEDGEMENTS
================

Development of this module was sponsored by Deutsche Telekom AG - Products & Innovation

HISTORY
=======

* Version 0.1: Initial version, mostly feature-complete

* Version 0.2: Rename: `x-variant` -> `X-DeviceClass`, `type_mtd` ->
  `type_class`, change return values. `dcs.type_class` will now return one of:

::

        new value       old value
        ---------       ---------
        desktop         dsk
        smartphone      mob
        tablet          tab

* Version 0.3: Class assignments can now be defined in `src/classes.conf`

* Version 0.4: Also support Varnish 4. Take memory from Varnish workspace
  instead of stack for Varnish 4.

* Version 0.5: Also uspport Varnish 6 (post 5.2)

BUGS
====

None known

SEE ALSO
========

* varnishd(1)
* vcl(7)

COPYRIGHT
=========

* Copyright 2014-2017 UPLEX - Nils Goroll Systemoptimierung

LICENSE
=======

.. include:: LICENSE