Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
D
dcs_classifier
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
uplex-varnish
dcs_classifier
Commits
06fcbcf3
Commit
06fcbcf3
authored
Nov 28, 2016
by
Nils Goroll
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
support for varnish 5 / master, malloc fallback when there is not enough workspace
parent
fb588789
Pipeline
#86
skipped
Changes
9
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
9 changed files
with
358 additions
and
29 deletions
+358
-29
LICENSE
LICENSE
+1
-1
Makefile.am
check/Makefile.am
+1
-0
dcs01.vtc
check/tests4/dcs01.vtc
+33
-1
configure.ac
configure.ac
+26
-6
dcs_varnish.c
src/dcs_varnish.c
+29
-7
vmod_dcs.c
src/vmod_dcs.c
+62
-14
vmod_dcs40.vcc
src/vmod_dcs40.vcc
+0
-0
vmod_dcs41.vcc
src/vmod_dcs41.vcc
+7
-0
vmod_dcs5.vcc
src/vmod_dcs5.vcc
+199
-0
No files found.
LICENSE
View file @
06fcbcf3
Copyright 2014 UPLEX - Nils Goroll Systemoptimierung
Copyright 2014
-2016
UPLEX - Nils Goroll Systemoptimierung
All rights reserved.
All rights reserved.
Redistribution and use in source and binary forms, with or without
Redistribution and use in source and binary forms, with or without
...
...
check/Makefile.am
View file @
06fcbcf3
...
@@ -9,6 +9,7 @@ VMOD_TESTS = $(srcdir)/@TESTDIR@/*.vtc
...
@@ -9,6 +9,7 @@ VMOD_TESTS = $(srcdir)/@TESTDIR@/*.vtc
.PHONY
:
$(VMOD_TESTS)
.PHONY
:
$(VMOD_TESTS)
$(srcdir)/@TESTDIR@/*.vtc
:
$(srcdir)/@TESTDIR@/*.vtc
:
PATH
=
@LIBVARNISHAPI_SBINDIR@:
$$
PATH
\
@VARNISHTEST@
-Dvarnishd
=
@VARNISHD@
-Dvmod_topbuild
=
$(abs_top_builddir)
$@
@VARNISHTEST@
-Dvarnishd
=
@VARNISHD@
-Dvmod_topbuild
=
$(abs_top_builddir)
$@
check
:
$(VMOD_TESTS)
check
:
$(VMOD_TESTS)
...
...
check/tests4/dcs01.vtc
View file @
06fcbcf3
...
@@ -7,7 +7,7 @@ server s1 {
...
@@ -7,7 +7,7 @@ server s1 {
txresp
txresp
} -start
} -start
varnish v1 -vcl+backend {
varnish v1 -
arg "-p workspace_client=9k" -
vcl+backend {
import dcs from "${vmod_topbuild}/src/.libs/libvmod_dcs.so";
import dcs from "${vmod_topbuild}/src/.libs/libvmod_dcs.so";
sub vcl_recv {
sub vcl_recv {
...
@@ -41,6 +41,36 @@ varnish v1 -vcl+backend {
...
@@ -41,6 +41,36 @@ varnish v1 -vcl+backend {
} -start
} -start
# set the workspace too small after the vmod has been initialized
# so we run into the malloc case for the first request
varnish v1 -cliok "param.show workspace_client"
varnish v1 -cliok "param.set workspace_client 9k"
varnish v1 -cliok "param.show workspace_client"
logexpect l1 -v v1 -g raw -d 1 {
expect * 1001 VCL_call {^RECV}
expect 0 = Error {^notice: workspace_client is set too low}
expect 0 = Error {^notice: malloc'ing}
expect 0 = ReqHeader {^xx-entry-key: unidentified}
expect 0 = Error {^notice: malloc'ing}
expect 0 = ReqHeader {^xx-type-id: 0}
expect 0 = Error {^notice: malloc'ing}
expect 0 = ReqHeader {^x-nb-classified: unidentified}
expect 0 = Error {^notice: malloc'ing}
expect 0 = ReqHeader {^X-DeviceClass: desktop}
# for the next request, we must not see the notices because
# workspace should be adjusted now
expect * 1005 VCL_call {^RECV}
expect 0 = ReqHeader {^xx-entry-key: android}
expect 0 = ReqHeader {^xx-type-id:}
expect 0 = ReqHeader {^x-nb-classified: Tablet}
expect 0 = ReqHeader {^X-DeviceClass: tablet}
}
logexpect l1 -start
client c1 {
client c1 {
# no User-Agent
# no User-Agent
txreq -url "/"
txreq -url "/"
...
@@ -90,3 +120,5 @@ client c1 {
...
@@ -90,3 +120,5 @@ client c1 {
expect resp.http.x-nb-classified == "Mobile Phone"
expect resp.http.x-nb-classified == "Mobile Phone"
expect resp.http.X-DeviceClass == "smartphone"
expect resp.http.X-DeviceClass == "smartphone"
} -run
} -run
logexpect l1 -wait
configure.ac
View file @
06fcbcf3
AC_PREREQ(2.59)
AC_PREREQ(2.59)
AC_COPYRIGHT([Copyright 201
5
UPLEX - Nils Goroll Systemoptimierung])
AC_COPYRIGHT([Copyright 201
6
UPLEX - Nils Goroll Systemoptimierung])
AC_INIT([dcs_classifier], [trunk])
AC_INIT([dcs_classifier], [trunk])
AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_SRCDIR(src/gen/gen_dcs_classifier.pl)
AC_CONFIG_SRCDIR(src/gen/gen_dcs_classifier.pl)
...
@@ -156,10 +156,30 @@ if test "x$VARNISHSRC" = x; then
...
@@ -156,10 +156,30 @@ if test "x$VARNISHSRC" = x; then
[$LIBVARNISHAPI_SBINDIR:$LIBVARNISHAPI_BINDIR:$PATH])
[$LIBVARNISHAPI_SBINDIR:$LIBVARNISHAPI_BINDIR:$PATH])
AM_CONDITIONAL([BUILD_VMOD], [true])
AM_CONDITIONAL([BUILD_VMOD], [true])
AC_SUBST([VCCFILE], [vmod_dcs4.vcc])
AC_SUBST([VARNISH_VERSION], [$($PKG_CONFIG --modversion varnishapi)])
AC_DEFINE([VARNISH_MAJOR], [4],
case "x$VARNISH_VERSION" in
[Define the Varnish major version we compile against])
xtrunk|x5*)
AC_SUBST([TESTDIR], [tests4])
AC_SUBST([VCCFILE], [vmod_dcs5.vcc])
AC_DEFINE([VARNISH_MAJMIN], [50],
[Define the Varnish major version we compile against])
AC_SUBST([TESTDIR], [tests4])
;;
x4.1*)
AC_SUBST([VCCFILE], [vmod_dcs41.vcc])
AC_DEFINE([VARNISH_MAJMIN], [41],
[Define the Varnish major version we compile against])
AC_SUBST([TESTDIR], [tests4])
;;
x4.0*)
AC_SUBST([VCCFILE], [vmod_dcs40.vcc])
AC_DEFINE([VARNISH_MAJMIN], [40],
[Define the Varnish major version we compile against])
AC_SUBST([TESTDIR], [tests4])
;;
*)
AC_MSG_ERROR([unsupported varnish varsion])
;;
esac
else
else
AM_CONDITIONAL([HAVE_VARNISH_M4], [false])
AM_CONDITIONAL([HAVE_VARNISH_M4], [false])
AC_MSG_NOTICE([VARNISHSRC overrides pkg-config])
AC_MSG_NOTICE([VARNISHSRC overrides pkg-config])
...
@@ -229,7 +249,7 @@ else
...
@@ -229,7 +249,7 @@ else
])
])
AC_SUBST([VCCFILE], [vmod_dcs3.vcc])
AC_SUBST([VCCFILE], [vmod_dcs3.vcc])
AC_DEFINE([VARNISH_MAJ
OR], [3
], [Define the Varnish major version we compile against])
AC_DEFINE([VARNISH_MAJ
MIN], [30
], [Define the Varnish major version we compile against])
AC_SUBST([TESTDIR], [tests])
AC_SUBST([TESTDIR], [tests])
# Check that varnishtest is built in the varnish source directory
# Check that varnishtest is built in the varnish source directory
...
...
src/dcs_varnish.c
View file @
06fcbcf3
/*
/*
* Copyright
(c) 2014-2015
UPLEX - Nils Goroll Systemoptimierung
* Copyright
2014-2016
UPLEX - Nils Goroll Systemoptimierung
* All rights reserved.
* All rights reserved.
*
*
* Redistribution and use in source and binary forms, with or without
* Redistribution and use in source and binary forms, with or without
...
@@ -42,7 +42,7 @@
...
@@ -42,7 +42,7 @@
#define UA_LIMIT 1024
#define UA_LIMIT 1024
#define DCS_VARNISH2_NHDRS 4
#define DCS_VARNISH2_NHDRS 4
#if VARNISH_MAJ
OR == 4
#if VARNISH_MAJ
MIN >= 40
typedef
const
struct
vrt_ctx
dcs_ctx
;
typedef
const
struct
vrt_ctx
dcs_ctx
;
...
@@ -56,8 +56,10 @@ const char *DCS_GetHdr(dcs_ctx *ctx, const struct gethdr_s *hdr) {
...
@@ -56,8 +56,10 @@ const char *DCS_GetHdr(dcs_ctx *ctx, const struct gethdr_s *hdr) {
* reduced to 48k, so we take our allocation from the workspace
* reduced to 48k, so we take our allocation from the workspace
*/
*/
#define DCS_USE_WS
#define DCS_USE_WS
// malloc
#include <stdlib.h>
#else
/* VARNISH_MAJ
OR == 3
*/
#else
/* VARNISH_MAJ
MIN == 30
*/
struct
gethdr_s
{
struct
gethdr_s
{
enum
gethdr_e
where
;
enum
gethdr_e
where
;
...
@@ -125,16 +127,34 @@ dcs_varnish_classify(dcs_ctx *ctx) {
...
@@ -125,16 +127,34 @@ dcs_varnish_classify(dcs_ctx *ctx) {
#ifdef DCS_USE_WS
#ifdef DCS_USE_WS
#define WS_SIZE (DCS_MATCH_MEM_SZ + UA_LIMIT)
#define WS_SIZE (DCS_MATCH_MEM_SZ + UA_LIMIT)
unsigned
space
=
WS_Reserve
(
ctx
->
ws
,
WS_SIZE
);
unsigned
space
;
void
*
mem
=
ctx
->
ws
->
f
;
void
*
mem
;
char
*
uabuf
=
ctx
->
ws
->
f
+
DCS_MATCH_MEM_SZ
;
char
*
malloced
=
NULL
;
char
*
uabuf
;
space
=
WS_Reserve
(
ctx
->
ws
,
0
);
if
(
space
>=
WS_SIZE
)
{
mem
=
ctx
->
ws
->
f
;
uabuf
=
ctx
->
ws
->
f
+
DCS_MATCH_MEM_SZ
;
}
else
{
TWEAK_NOTICE
(
ctx
,
"malloc'ing ctx->ws: ws %u avail, need %u"
,
space
,
WS_SIZE
);
space
=
WS_SIZE
;
malloced
=
malloc
(
space
);
AN
(
malloced
);
mem
=
malloced
;
uabuf
=
malloced
+
DCS_MATCH_MEM_SZ
;
}
assert
(
space
!=
0
);
assert
(
space
>=
WS_SIZE
);
assert
(
space
>=
WS_SIZE
);
space
-=
DCS_MATCH_MEM_SZ
;
space
-=
DCS_MATCH_MEM_SZ
;
if
(
space
>
UA_LIMIT
)
if
(
space
>
UA_LIMIT
)
space
=
UA_LIMIT
;
space
=
UA_LIMIT
;
else
assert
(
space
==
UA_LIMIT
);
#else
#else
size_t
space
=
UA_LIMIT
;
size_t
space
=
UA_LIMIT
;
char
mem
[
DCS_MATCH_MEM_SZ
];
char
mem
[
DCS_MATCH_MEM_SZ
];
...
@@ -197,6 +217,8 @@ dcs_varnish_classify(dcs_ctx *ctx) {
...
@@ -197,6 +217,8 @@ dcs_varnish_classify(dcs_ctx *ctx) {
#ifdef DCS_USE_WS
#ifdef DCS_USE_WS
/* used workspace only as scratch - releasing all */
/* used workspace only as scratch - releasing all */
WS_Release
(
ctx
->
ws
,
0
);
WS_Release
(
ctx
->
ws
,
0
);
if
(
malloced
)
free
(
malloced
);
#endif
#endif
return
(
ret
);
return
(
ret
);
}
}
src/vmod_dcs.c
View file @
06fcbcf3
/*
/*
* Copyright
(c) 2014
UPLEX - Nils Goroll Systemoptimierung
* Copyright
2014-2016
UPLEX - Nils Goroll Systemoptimierung
* All rights reserved.
* All rights reserved.
*
*
* Redistribution and use in source and binary forms, with or without
* Redistribution and use in source and binary forms, with or without
...
@@ -30,32 +30,80 @@
...
@@ -30,32 +30,80 @@
#include "dcs_config.h"
#include "dcs_config.h"
#if VARNISH_MAJMIN >= 41
#include "vcl.h"
#endif
#include "vrt.h"
#include "vrt.h"
#include "vcc_if.h"
#include "vcc_if.h"
#define TWEAK_NOTICE(ctx, ...) do { \
if ((ctx) && (ctx)->vsl) { \
VSLb((ctx)->vsl, SLT_Error, \
"notice: " __VA_ARGS__); \
} else { \
fprintf(stderr, "notice: " __VA_ARGS__); \
} \
} while (0)
#include "dcs_varnish.c"
#include "dcs_varnish.c"
static
inline
void
tweak_ws
(
dcs_ctx
*
ctx
)
{
(
void
)
ctx
;
#ifdef DCS_USE_WS
#if VARNISH_MAJMIN < 40
ctx
=
NULL
;
#endif
if
(
cache_param
->
workspace_client
>=
DCS_USE_WS_MINIMUM
)
return
;
TWEAK_NOTICE
(
ctx
,
"workspace_client is set too low for vmod_dcs, "
"adjusting from %u to %lu bytes
\n
"
,
cache_param
->
workspace_client
,
DCS_USE_WS_MINIMUM
);
cache_param
->
workspace_client
=
DCS_USE_WS_MINIMUM
;
#endif
return
;
}
#if VARNISH_MAJMIN >= 41
int
__match_proto__
(
vmod_event_f
)
vmod_event
(
VRT_CTX
,
struct
vmod_priv
*
priv
,
enum
vcl_event_e
e
)
{
(
void
)
priv
;
switch
(
e
)
{
case
VCL_EVENT_LOAD
:
case
VCL_EVENT_WARM
:
tweak_ws
(
ctx
);
return
(
dcs_match_init
());
;;
case
VCL_EVENT_COLD
:
return
(
0
);
default:
return
(
0
);
}
return
(
0
);
}
#else
/* VARNISH_MAJMIN < 41 */
int
int
vmod_init
(
struct
vmod_priv
*
priv
,
const
struct
VCL_conf
*
cfg
)
vmod_init
(
struct
vmod_priv
*
priv
,
const
struct
VCL_conf
*
cfg
)
{
{
(
void
)
priv
;
(
void
)
priv
;
(
void
)
cfg
;
(
void
)
cfg
;
#ifdef DCS_USE_WS
if
(
cache_param
->
workspace_client
<
DCS_USE_WS_MINIMUM
)
{
tweak_ws
(
NULL
);
/* XXX better function for emiting this warning */
fprintf
(
stderr
,
"notice: workspace_client is set too low for vmod_dcs, adjusting from %u to %lu bytes
\n
"
,
cache_param
->
workspace_client
,
DCS_USE_WS_MINIMUM
);
cache_param
->
workspace_client
=
DCS_USE_WS_MINIMUM
;
}
#endif
return
dcs_match_init
(
);
return
(
dcs_match_init
()
);
}
}
#endif
VCL_INT
VCL_INT
vmod_classify
(
dcs_ctx
*
ctx
)
{
vmod_classify
(
dcs_ctx
*
ctx
)
{
return
dcs_varnish_classify
(
ctx
);
tweak_ws
(
ctx
);
return
(
dcs_varnish_classify
(
ctx
));
}
}
VCL_STRING
VCL_STRING
...
@@ -69,7 +117,7 @@ vmod_type_id(dcs_ctx *ctx, VCL_INT e) {
...
@@ -69,7 +117,7 @@ vmod_type_id(dcs_ctx *ctx, VCL_INT e) {
(
void
)
ctx
;
(
void
)
ctx
;
return
dcs_match_type_id
(
e
);
return
dcs_match_type_id
(
e
);
}
}
VCL_STRING
VCL_STRING
vmod_type_name
(
dcs_ctx
*
ctx
,
VCL_INT
e
)
{
vmod_type_name
(
dcs_ctx
*
ctx
,
VCL_INT
e
)
{
const
VCL_INT
t
=
dcs_match_type_id
(
e
);
const
VCL_INT
t
=
dcs_match_type_id
(
e
);
...
...
src/vmod_dcs4.vcc
→
src/vmod_dcs4
0
.vcc
View file @
06fcbcf3
File moved
src/vmod_dcs41.vcc
0 → 100644
View file @
06fcbcf3
$Module dcs 3 Varnish Device Classification Service Module
$Event vmod_event
$Function INT classify()
$Function STRING entry_key(INT)
$Function INT type_id(INT)
$Function STRING type_name(INT)
$Function STRING type_class(INT)
src/vmod_dcs5.vcc
0 → 100644
View file @
06fcbcf3
# Copyright 2014-2016 UPLEX - Nils Goroll Systemoptimierung
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the
# distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS *AS IS* AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
# OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
# IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
$Module dcs 3 Varnish Device Classification Service Module
::
import dcs [from "path"] ;
# typical use
sub vcl_recv {
set req.http.x-nb-classified = dcs.type_name(dcs.classify());
# - or-
set req.http.X-DeviceClass = dcs.type_class(dcs.classify());
# ...
}
DESCRIPTION
===========
This Varnish module provides an efficient implementation of device
detection and classification using the downloadable version of the
Netbiscuits Device Classifier Service (DCS) database. or a
self-provided database. An example database is included.
Netbiscuits Device Classifier Service (DCS) database
----------------------------------------------------
The DCS database is not part of this module and needs to be obtained
from Netbiscuits, please refer to
http://www.netbiscuits.com/device-detection/ as a starting point. With
sufficient privileges, a classifier token can be created on
https://my.netbiscuits.com/ under Account -> Token Management. See
http://kb.netbiscuits.com/dcs/dcs_ui_tokenmanagement.html for
instructions.
The classifier token is also referred to as DCS_KEY below.
Demo Database file
------------------
For demonstration purposes, we provice a simple database file with
some minimal and incomplete classification information in
`src/dcs_demo.db`. See :ref:using_the_demo_db for details.
Meta Classes
------------
Classification types from the database file can be associated with
meta-classes in the file `src/classes.conf`. Its format is
::
[classname]
Typename from the database
Note that the bundled tests need entries from the bundled
classes.conf.
During the build process, `gen_dcs_classifier.pl` emits warnings if
entries are missing from the classes configuration or if entries
remain unused. It may be advisable to update the configuration when
these warnigs are seen.
PERFORMANCE
-----------
This module was developed to provide exceptional performance compared
to previous implementations without requiring any changes to the
structure of the database or introducing any changes to the
semantics.
All lookups are uncached and lookup complexity does not depend on the
position of the best match in the dcs database.
To achieve high performance, C code for a custom parser for all tokens
(substrings) from the DCS database is generated. The parser is run to
detect all tokens from the User-Agent, marking potential matches. As
the match result, the DCS database entry which comes first in the
database is returned.
Exemplary benchmarks on a single i7-4600M core @2.9 GHz max suggest
that detection throughput exceeds 200.000 matches per second, which
corresponds to a latency in the order of 5us (5 microseconds).
.. _detection_methodology:
DETECTION METHODOLOGY
=====================
The following applies to the `classify()` function of the Varnish Module and Varnish 2
inline-C. The `dcs` command line tool only implements the last step.
* If the `x-wap-profile header` is present, the User-Agent will be
classified as a mobile phone
* If the `X-OperaMini-Phone-UA` header is present, the string " opera/
opera mini/ " gets appended to the `User-Agent` header for
classification.
* The contents of the headers `X-OperaMini-Phone-UA`,
`X-Device-User-Agent`, `X-Original-User-Agent` and `X-Goog-Source`
are appended to the `User-Agent` header for classification.
* The enrichted `User-Agent` string is passed to the DCS classifer and
the matching dcs db entry is returned - or a special db entry named
"unidentified".
$Event vmod_event
$Function INT classify()
Runs the :ref:detection_methodology as described.
The return value is the index of the DCS DB entry.
This vmod function should be used as an argument to one of the
functions described below.
As each invocation runs the classifcation again,
it should only be used once per request.
Example:
::
set req.http.x-nb-classified = dcs.type_name(dcs.classify());
$Function STRING entry_key(INT)
Returns the key of the dcs db entry whose index is given as the integer argument.
Example:
::
set req.http.xx-entry-key = dcs.entry_key(dcs.classify());
Might set `xx-entry-key` to something like "android*opera mini/"
$Function INT type_id(INT)
Returns the internal type id of the dcs db entry whose index is given as the integer argument.
Example:
::
set req.http.xx-type-id = dcs.type_id(dcs.classify());
Might set `xx-type-id` to "11"
$Function STRING type_name(INT)
Returns the type name of the dcs db entry whose index is given as the integer argument.
Example:
::
set req.http.x-nb-classified = dcs.type_name(dcs.classify());
might set `x-nb-classified` to "Mobile Phone"
$Function STRING type_class(INT)
Returns one of the meta types defined in `src/classes.conf`
Example:
::
set req.http.X-DeviceClass = dcs.type_class(dcs.classify());
might set `X-DeviceClass` to "smartphone"
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment