D

dcs_classifier

Varnish Device Classification Service Module

DCS Classifier

Varnish Device Classification Service Module

Manual section: 3
Authors: Nils Goroll
Date: 2015-06-16
Version: 0.5

SYNOPSIS

Command line

<prefix>/bin/dcs

Varnish VMOD (Varnish 3 and Varnish 4)

VCL:

import dcs [from "path"] ;

# typical use
sub vcl_recv {
        set req.http.x-nb-classified = dcs.type_name(dcs.classify());
        # - or-
        set req.http.X-DeviceClass   = dcs.type_class(dcs.classify());
        # ...
}

Varnish 2 inline-C

Varnish start:

varnishd "-pcc_command=\"exec gcc -I<prefix>/share ...\""

VCL:

C{
#include "dcs_varnish2.c"
}C

sub vcl_recv {
    C{ dcs_varnish2_classify_hdrs(sp); }C
    ...
}

DESCRIPTION

This Varnish module provides an efficient implementation of device detection and classification using the downloadable version of the Netbiscuits Device Classifier Service (DCS) database. or a self-provided database. An example database is included.

Netbiscuits Device Classifier Service (DCS) database

The DCS database is not part of this module and needs to be obtained from Netbiscuits, please refer to http://www.netbiscuits.com/device-detection/ as a starting point. With sufficient privileges, a classifier token can be created on https://my.netbiscuits.com/ under Account -> Token Management. See http://kb.netbiscuits.com/dcs/dcs_ui_tokenmanagement.html for instructions.

The classifier token is also referred to as DCS_KEY below.

Demo Database file

For demonstration purposes, we provice a simple database file with some minimal and incomplete classification information in src/dcs_demo.db. See :ref:using_the_demo_db for details.

Meta Classes

Classification types from the database file can be associated with meta-classes in the file src/classes.conf. Its format is

[classname]
Typename from the database

Note that the bundled tests need entries from the bundled classes.conf.

During the build process, gen_dcs_classifier.pl emits warnings if entries are missing from the classes configuration or if entries remain unused. It may be advisable to update the configuration when these warnigs are seen.

PERFORMANCE

This module was developed to provide exceptional performance compared to previous implementations without requiring any changes to the structure of the database or introducing any changes to the semantics.

All lookups are uncached and lookup complexity does not depend on the position of the best match in the dcs database.

To achieve high performance, C code for a custom parser for all tokens (substrings) from the DCS database is generated. The parser is run to detect all tokens from the User-Agent, marking potential matches. As the match result, the DCS database entry which comes first in the database is returned.

Exemplary benchmarks on a single i7-4600M core @2.9 GHz max suggest that detection throughput exceeds 200.000 matches per second, which corresponds to a latency in the order of 5us (5 microseconds).

INTERFACES

The following use cases are supported

  • Use as a Varnish 4 module
  • Use as a Varnish 3 module
  • Use with Varnish 2 as inline-C
  • Command line tools

PREREQUISITES

The following tools are required on the system where this module is to be build.

When building from the source repository:

  • libtool
  • autoconf 2.69 or later
  • automake

For all builds:

  • A working C99 compatible compiler. gcc 4 is tested, other gcc versions and clang should work
  • A working build environment with all standard headers and tools (e.g. make)
  • perl
  • The following perl modules:
    • Crypt::RC4
    • Digest::MD5
    • MIME::Base64

To download the dcs DB (through the DCS_ACCOUNT parameter to configure)

  • curl

To build the documentation

  • rst2man

To build the varnish module (VMOD)

  • the source tree of the Varnish version the VMOD is to be used with. Varnish must have been configured and compiled in this source tree.
  • python2 or higher

BUILDING

A minimal build will only provide the command line tools. It requires the follwing steps:

  1. Generate configure and automake files (Only when building from the source repository):
sh autogen.sh

2. Run configure to use either an existing database file or download from the netbiscuits website (standard configure arguments are also supported). In both cases the classifier token from Netbiscuits is required:

2a. Existing database file

sh configure DCS_KEY=<key> DCS_DBFILE=<file>

2b. Download with a https://my.netbiscuits.com/ account

sh configure DCS_KEY=<key> DCS_ACCOUNT=<account-name>

2c. Use the bundled demo Database.

We provide a very simple database for demonstration purposes.
sh configure DCS_KEY=demo DCS_DBFILE=dcs_demo.db

2d. Optionally add dbfiles to prepend and append

To override / amend entries from the main DCS_DBFILE, additional files can be speficied as DCS_DBFILE_PRE (to prepend / override) and DCS_DBFILE_POST. These files should not be encrypted, otherwise the same encryption key must be used.
sh configure DCS_KEY=demo DCS_DBFILE=dcs_demo.db \
             DCS_DBFILE_PRE=/path/to/pre.db \
             DCS_DBFILE_POST=/path/to/post.db
  1. Run make
  2. Optionally run make install to install at the default location (normally /usr/local) or in the prefix specified by the --prefix argument to configure.

Building the varnish module (vmod) for Varnish 4 and higher

If varnish is installed in a standard directory, generic :ref:BUILDING as explained above should also build the vmod.

If you have installed Varnish to a non-standard directory, call autogen.sh and configure with PKG_CONFIG_PATH pointing to the appropriate path. For example, when varnishd configure was called with --prefix=$PREFIX, use

PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig
export PKG_CONFIG_PATH

When building the vmod, an additional

make check

step is recommended to run the bundled varnishtest tests.

Building the varnish module (vmod) for Varnish 3

To build the vmod for varnish 3, in addition to the configure arguments given above, the VARNISHSRC argument must be used as in

sh configure DCS_KEY=<key> DCS_ACCOUNT=<account-name> \
   VARNISHSRC=/path/to/your/varnish/source/varnish-cache

Optionally, a custom vmod installation directory can be specified using VMOD_DIR=<dir>

Building for use with Varnish 2

To install all files required to use the DCS module with Varnish 2 inline-C, use the additional configure argument --enable-varnish2

With this argument, make install will create additional source files in the share directory of the installation prefix.

Optimizing the DCS database

The optional make fixup before calling make will remove entries from the DCS database which will never be hit and will reorder entries which are likely to be unintentionally masked by previous entries.

Use make clean before make fixup if the code has already been build.

DETECTION METHODOLOGY

The following applies to the classify() function of the Varnish Module and Varnish 2 inline-C. The dcs command line tool only implements the last step.

  • If the x-wap-profile header is present, the User-Agent will be classified as a mobile phone
  • If the X-OperaMini-Phone-UA header is present, the string " opera/ opera mini/ " gets appended to the User-Agent header for classification.
  • The contents of the headers X-OperaMini-Phone-UA, X-Device-User-Agent, X-Original-User-Agent and X-Goog-Source are appended to the User-Agent header for classification.
  • The enrichted User-Agent string is passed to the DCS classifer and the matching dcs db entry is returned - or a special db entry named "unidentified".

VMOD USAGE - FUNCTIONS

To import the vmod, use

import dcs [from "path"] ;

INT classify()

Runs the :ref:detection_methodology as described.

The return value is the index of the DCS DB entry.

This vmod function should be used as an argument to one of the functions described below.

As each invocation runs the classifcation again, it should only be used once per request.

Example:

set req.http.x-nb-classified = dcs.type_name(dcs.classify());

STRING entry_key(INT)

Returns the key of the dcs db entry whose index is given as the integer argument.

Example:

set req.http.xx-entry-key = dcs.entry_key(dcs.classify());

Might set xx-entry-key to something like "android*opera mini/"

INT type_id(INT)

Returns the internal type id of the dcs db entry whose index is given as the integer argument.

Example:

set req.http.xx-type-id = dcs.type_id(dcs.classify());

Might set xx-type-id to "11"

STRING type_name(INT)

Returns the type name of the dcs db entry whose index is given as the integer argument.

Example:

set req.http.x-nb-classified  = dcs.type_name(dcs.classify());

might set x-nb-classified to "Mobile Phone"

STRING type_class(INT)

Returns one of the meta types defined in src/classes.conf

Example:

set req.http.X-DeviceClass    = dcs.type_class(dcs.classify());

might set X-DeviceClass to "smartphone"

COMMAND LINE USAGE

dcs

This command line tool reads one User-Agent string per input line until EOF is reached and outputs the respective classifcation in the following format:

--
<input-line lowercase>
entry id <entry-id> type <type_id> - <type_class> - <type_name>

VARNISH 2 USAGE

To use this module with Varnish 2, the classifier code and some vmod glue code need to be compiled as inline-C. To do so, the CC command executed by VCC needs to be modified such that the dcs source code can be found.

In the following section, $PREFIX needs to be replaced by the installation prefix passed to configure using the --prefix command or the default value /usr/local:

  • Determine the cc_command of the running Varnish instance
varnishadm -T ... param.show cc_command
  • Add the following to the cc_command after the compiler name (usually "gcc"):
      -I$PREFIX/share

and add the resulting `cc_command` as a paramter to the varnish
start script.

Example: If `cc_command` is
      "exec gcc -std=gnu99  -pthread -fpic -shared -Wl,-x -o %o %s"

then add the following to the Varnish start parameters:
-pcc_command="exec gcc -I$PREFIX/share -std=gnu99  -pthread -fpic -shared -Wl,-x -o %o %s"

To change a running varnish instance without a restart, varnishadm can be used.

Once the new cc_command is active, the following can be used in VCL:

C{
#include "dcs_varnish2.c"
}C

sub vcl_recv {
    C{
        dcs_varnish2_classify_hdrs(sp);
    }C

    # ....
}

This will make the following headers available in vcl_recv after the call to dcs_varnish2_classify_hdrs:

  • req.http.x-nb-classified same as :ref:func_type_name e.g. "Mobile Phone"
  • req.http.X-DeviceClass same as :ref:func_type_class e.g. "desktop"

NOTES

dynamic linking

The varnsh dcs vmod is always compiled to be self-contained, it does not link dynamically to libdcs.so. This is done for two reasons:

  1. To avoid the performance penalty of library calls
  2. To avoid version conflicts between libdcs.so and the rest of libvmod_dcs.so which could happen when a libdcs.so generated from one DCS database is used with libvmod_dcs.so generated from another

cURL

curl uses environment variables like http_proxy. If they do not pass through the Makefiles (as with some versions of make), curl can be configured using ~/.curlrc.

ACKNOWLEDGEMENTS

Development of this module was sponsored by Deutsche Telekom AG - Products & Innovation

HISTORY

  • Version 0.1: Initial version, mostly feature-complete
  • Version 0.2: Rename: x-variant -> X-DeviceClass, type_mtd -> type_class, change return values. dcs.type_class will now return one of:
new value       old value
---------       ---------
desktop         dsk
smartphone      mob
tablet          tab
  • Version 0.3: Class assignments can now be defined in src/classes.conf
  • Version 0.4: Also support Varnish 4. Take memory from Varnish workspace instead of stack for Varnish 4.

BUGS

None known

SEE ALSO

  • varnishd(1)
  • vcl(7)

COPYRIGHT

  • Copyright 2014 UPLEX - Nils Goroll Systemoptimierung

LICENSE