Skip to content

k8s-ingress
k8s-ingress
Commit 8e64d461 authored by Geoff Simmons's avatar Geoff Simmons
Browse files
Options

Document the VarnishConfig Custom Resource. 

And update docs for self-sharding as configured by VarnishConfig.
parent 635df9e7
Hide whitespace changes
Inline Side-by-side
Showing with 330 additions and 142 deletions
deploy/README.md
View file @ 8e64d461
......@@ -13,7 +13,22 @@ namespace, see the [instructions for single-namespace
deployments](/examples/namespace) in the [``/examples``
folder](/examples).
## ServiceAccount and RBAC
## The first time
The first steps must be executed for a new cluster, and will probably
be repeated only rarely afterward (for example for a software update).
### Containers
These containers must be availabe for pull in the Kubernetes cluster:
* Controller: ``varnish-ingress/controller``
* Varnish to implement Ingress: ``varnish-ingress/varnish``
See the [``container/`` folder](/container) for instructions for
building the containers.
### ServiceAccount and RBAC
Define a ServiceAccount named ``varnish-ingress-controller`` and apply
[Role-based access
......@@ -21,9 +36,55 @@ control](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)
(RBAC) to permit the necessary API access for the Ingress controller:
```
$ kubectl apply -f serviceaccount.yaml
$ kubectl apply -f rbac.yaml
```
### VarnishConfig Custom Resource definition
The project defines a
[Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
``VarnishConfig`` to specify special configurations and features of
Varnish running as an Ingress (beyond the standard Ingress
specification):
```
$ kubectl apply -f varnishcfg-crd.yaml
```
### Deploy the controller
This example uses a Deployment to run the controller container in the
``kube-system`` namespace:
```
$ kubectl apply -f controller.yaml
```
The ``image`` in the manifest MUST be ``varnish-ingress/controller``.
It does *not* make sense to deploy more than one replica of the
controller. If there are more controllers, all of them will connect to
the Varnish instances and send them the same administrative
commands. That is not an error (or there is a bug in the controller if
it does cause errors), but the extra work is superflous.
#### Controller options
Command-line options for the controller invocation can be set using the
``args`` section of the ``container`` specification:
```
containers:
- image: varnish-ingress/controller
name: varnish-ingress-controller
# [...]
args:
- -log-level=info
```
## Admin Secret
## Deploying Varnish as an Ingress
These steps are executed for each namespace in which Varnish is to be
deployed as an Ingress implementation.
### Admin Secret
The controller uses Varnish's admin interface to manage Varnish
instances, which requires authorization using a shared secret. This is
......@@ -59,7 +120,7 @@ everyone uses the same secret from an example.
**TO DO**: The key for the Secret (in the ``data`` field) is
hard-wired to ``admin``.
## Deploy Varnish containers
### Deploy Varnish containers
The present example uses a Deployment to deploy Varnish instances
(other possibilities are a DaemonSet or a StatefulSet):
......@@ -151,7 +212,7 @@ properly:
The port name must match the name given for port 8080 above.
### varnishd options
#### varnishd options
Varnish command-line options can be specified using the ``args`` section
of the ``container`` specfication:
......@@ -206,7 +267,7 @@ Among these restrictions are:
unload one, otherwise it interferes with the implementation of
Ingress.
## Expose the Varnish HTTP and admin ports
### Expose the Varnish HTTP and admin ports
With a Deployment, you may choose a resource such as a LoadBalancer or
Nodeport to create external access to Varnish's HTTP port. The present
......@@ -269,35 +330,7 @@ spec:
app: varnish-ingress
```
## Deploy the controller
This example uses a Deployment to run the controller container:
```
$ kubectl apply -f controller.yaml
```
The ``image`` must be ``varnish-ingress/controller``.
It does *not* make sense to deploy more than one replica of the
controller. If there are more controllers, all of them will connect to
the Varnish instances and send them the same administrative
commands. That is not an error (or there is a bug in the controller if
it does cause errors), but the extra work is superflous.
### Controller options
Command-line options for the controller invocation can be set using the
``args`` section of the ``container`` specification:
```
containers:
- image: varnish-ingress/controller
name: varnish-ingress-controller
# [...]
args:
- -log-level=info
```
# Done
## Done
When these commands succeed:
......
docs/README.md
View file @ 8e64d461
......@@ -2,7 +2,11 @@
The docs in this folder cover these topics:
* Technical references: authoritative documentation for these subjects:
* [``VarnishConfig`` Custom Resource](ref-varnish-cfg.md)
* [Varnish as a Kubernetes Ingress](varnish-as-ingress.md)
* [Self-sharding Varnish cluster](self-sharding.md): Discussion of the
design, and reference for the configuration
* [Self-sharding Varnish cluster](self-sharding.md): High-level
discussion of the design
docs/ref-varnish-cfg.md 0 → 100644
View file @ 8e64d461
# VarnishConfig Custom Resource reference
This is the authoritative reference for the ``VarnishConfig``
[Custom Resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/),
which is defined in this project to specify special configurations and
features for Varnish clusters running as Ingress, in addition to the
routing rules given in an Ingress specification.
If you only intend to use Varnish-as-Ingress Services to implement the
routing rules for an Ingress, then no ``VarnishConfig`` resource needs to
be defined. Use the ``VarnishConfig`` resource to apply additional
configurations such as [self-sharding](/docs/self-sharding.md).
Constraints on individual properties in ``VarnishConfig`` are checked
against validation rules when the manifest is applied, so you may get
immediate feedback about invalid values from a ``create`` or ``apply``
command for ``kubectl``. Other constraints, such as legal relations
between values or valid VCL syntax, cannot currently be checked until
the controller attempts to load the definition, and hence will not be
reported at apply time. Check the log of the controller and Events
created by the controller for error conditions -- these may include
error messages from the VCL compiler.
Examples for the use of ``VarnishConfig`` resources can be found in
the [``examples/``](/examples) folder.
## Custom Resouce definition
The Custom Resource is created with the ``CustomResourceDefinition`` defined
in [``varnishcfg-crd.yaml``](/deploy/varnishcfg-crd.yaml) in the
[``deploy/``](/deploy) folder:
```
$ kubectl apply -f deploy/varnishcfg-crd.yaml
```
## API Group, version and resource names
The API group in use for this project is
``ingress.varnish-cache.org``, currently at version ``v1alpha1``. So a
manifest specifying a ``VarnishConfig`` resource MUST begin with:
```
apiVersion: "ingress.varnish-cache.org/v1alpha1"
kind: VarnishConfig
```
You can choose any ``name`` and ``namespace`` in the ``metadata``
section. ``VarnishConfig`` has Namespaced scope, so its name must be
unique in a namespace, and its content is applied to Varnish Services
in the same namespace.
Existing ``VarnishConfig`` resources can be referred to in ``kubectl``
commands as ``varnishconfig``, ``varnishconfigs`` or with the short
name ``vcfg``:
```
$ kubectl get varnishconfigs -n my-namespace
$ kubectl describe vcfg my-vcfg
```
## ``spec``
The ``spec`` section of a ``VarnishConfig`` is required.
### ``spec.services``
The ``spec.services`` array is required, and MUST have at least one
element:
```
spec:
# The services array is required and must have at least one element.
# Lists the Service names of Varnish services in the same namespace
# to which this config is to be applied.
services:
- my-ingress
```
The strings in the ``services`` array MUST match the Service names of
Varnish Services to implement Ingress in the same namespace as the
``VarnishConfig`` Resource. The configuration in the Resource is
applied to those resources -- this makes it possible to have more than
one Varnish-as-Ingress Service in a namespace with different
configurations.
### ``spec.self-sharding``
The ``self-sharding`` object is optional. If it is present in the
manifest, then the [self-sharding](/docs/self-sharding.md) feature is
implemented for Services listed in the ``services`` array.
All of the properties of a ``self-sharding`` object are optional, and
default values hold for any properties that are not specified. To
specify self-sharding with all default values, just use an empty object:
```
spec:
# Implement self-sharding with defaults for all properties:
self-sharding: {}
```
Properties that may be specifed for ``self-sharding`` are:
* ``max-secondary-ttl``: string
* ``probe``: object
If specified, ``max-secondary-ttl`` MUST have the form of the VCL
[DURATION type](https://varnish-cache.org/docs/6.1/reference/vcl.html#durations)
(examples are ``90s`` for ninety seconds, or ``2m`` for two
minutes). This value is the TTL for "secondary" caching -- the upper
bound for a cached response forwarded from the "primary" Varnish instance
for a cacheable response (see the
[self-sharding document](/docs/self-sharding.md) for details).
``max-secondary-ttl`` defaults to ``5m`` (5 minutes).
The ``probe`` object specifies the health probes that Varnish instances
in a cluster use for one another (since they are defined as backends
for one another). Its properties are:
* ``timeout``: string
* ``interval``: string
* ``initial``: integer
* ``window``: integer
* ``threshold``: integer
These properties configure the corresponding values for
[health probes](https://varnish-cache.org/docs/6.1/reference/vcl.html#probes),
and they default to the default values for Varnish probes. If the
``probe`` object is left out altogether, then defaults hold for all of
its properties.
``timeout`` and ``interval`` MUST have the form of VCL DURATIONs, and
each of ``initial``, ``window`` and ``threshold`` MUST be >= 0.
``window`` and ``threshold`` MUST also be <= 64, and ``threshold``
MAY NOT be larger than ``window``.
Validation for ``VarnishConfig`` will report errors in the individual
fields at apply time, for example if the VCL DURATION properties do
not have the proper form. The ``threshold`` <= ``window`` constraint
is checked at VCL load time; if violated, it is reported in the
controller log and in Events generated by the controller for the
``VarnishConfig`` resource (with the error message from the VCL
compiler).
Example:
```
spec:
self-sharding:
# Any of these properties may be left out, in which case default
# values hold.
max-secondary-ttl: 2m
probe:
timeout: 6s
interval: 6s
initial: 2
window: 4
threshold: 3
```
docs/self-sharding.md
View file @ 8e64d461
......@@ -10,12 +10,13 @@ forwards requests to the cluster (such as a load balancer) is required
to shard the requests. Requests may be distributed to the cluster in
any way, and the Varnish instances take care of the sharding.
A sample manifest for an Ingress with annotations for self-sharding is
in the [``examples/``](/examples/self-sharding) folder.
A sample manifest for the ``VarnishConfig`` Custom Resource to
configure self-sharding is in the
[``examples/``](/examples/self-sharding) folder.
The technical details for
[configuring self-sharding](#annotation-syntax-for-self-sharding) are given
further below. The next two sections describe in more detail how the
The [technical reference](/docs/ref-varnish-cfg.md) for
``VarnishConfig`` specifies the configuration syntax for
self-sharding. The next two sections describe in more detail how the
sharding feature affects a Varnish cluster.
## Clustering without sharding
......@@ -121,95 +122,63 @@ Some features that result from self-sharding are:
probe are configurable with the ``probe-*`` group of annotations
described below.
## Annotation syntax for self-sharding
## Configuration
Self-sharding is configured by using the annotations
``ingress.varnish-cache.org/self-sharding*`` in an Ingress, for
example:
Self-sharding is configured by specifying a ``self-sharding`` object
in a ``VarnishConfig`` Custom Resource, and naming the Service names
for Varnish Services running as Ingress implementations in the same
namespace as the Custom Resource. For example:
```
apiVersion: extensions/v1beta1
kind: Ingress
apiVersion: "ingress.varnish-cache.org/v1alpha1"
kind: VarnishConfig
metadata:
name: cafe-ingress-varnish
annotations:
kubernetes.io/ingress.class: "varnish"
ingress.varnish-cache.org/self-sharding: "on"
ingress.varnish-cache.org/self-sharding-probe-timeout: "6s"
ingress.varnish-cache.org/self-sharding-probe-interval: "6s"
ingress.varnish-cache.org/self-sharding-probe-initial: "2"
ingress.varnish-cache.org/self-sharding-probe-window: "4"
ingress.varnish-cache.org/self-sharding-probe-threshold: "3"
ingress.varnish-cache.org/self-sharding-max-secondary-ttl: "1m"
namespace: varnish-ingress
# ...
name: self-sharding-cfg
spec:
services:
- my-ingress
self-sharding:
max-secondary-ttl: 3m
probe:
timeout: 6s
interval: 6s
initial: 2
window: 4
threshold: 3
```
VCL to implement self-sharding is generated by the controller for any
Varnish cluster that implements the Ingress.
Self-sharding is implemented when the
``ingress.varnish-cache.org/self-sharding`` annotation has the value
``on`` or ``true`` (case-insensitive). If the annotation is not
defined for an Ingress, or if it has any other value, then sharding is
not configured, and all other ``self-sharding-*`` annotations are
ignored.
The ``self-sharding-*`` parameters are all optional; default values
hold if they are left out of the configuration.
These parameters configure the corresponding values for the
[health probes](https://varnish-cache.org/docs/6.1/reference/vcl.html#probes)
that instances in the cluster use to check one another's health:
* ``self-sharding-probe-timeout``
* ``self-sharding-probe-interval``
* ``self-sharding-probe-initial``
* ``self-sharding-probe-window``
* ``self-sharding-probe-threshold``
The default values are the Varnish defaults for probes (in Varnish
6.1.1).
The values for probes MUST be valid as values in the VCL source for
the probe; for example, the ``timeout`` and ``interval`` parameters
must be valid for the VCL
[DURATION type](https://varnish-cache.org/docs/6.1/reference/vcl.html#durations)
(examples are ``1s`` for one second, or ``1m`` for a
minute). Constraints on the values of ``initial``, ``window`` and
``threshold`` MUST also be satisfied; for example, they must all be >=
0, and ``threshold`` may not be larger than ``window``. Check the
documentation linked above for details.
The ``self-sharding-max-secondary-ttl`` parameter defaults to ``5m``
(5 minutes), and sets the upper bound for "secondary" caching as
discussed above. If an instance forwards a request to another instance
and the response is cacheable, then the forwarding instance may also
cache the response (as does the "primary" instance). But the secondary
instance will not retain the response for longer than the value of
VCL to implement self-sharding is generated by the controller for
Varnish-as-Ingress clusters running as a Service named in
``services``.
Self-sharding is implemented when the ``self-sharding`` object is
present. Properties of the ``self-sharding`` object are all optional,
and default values hold if they are left out of the configuration. To
configure self-sharding with all default values, just specify an empty
object:
```
spec:
# Self-sharding with all properties set to defaults.
self-sharding: {}
```
The ``max-secondary-ttl`` parameter defaults to ``5m`` (5 minutes),
and sets the upper bound for "secondary" caching as discussed
above. If an instance forwards a request to another instance and the
response is cacheable, then the forwarding instance may also cache the
response (as does the "primary" instance). But the secondary instance
will not retain the response for longer than the value of
``max-secondary-ttl`` (it is likely to hit the primary cache after
that TTL expires). Keeping this value low relative to the "primary"
TTLs serves to reduce the total memory load for caching in the
cluster.
As with the values for ``probe-timeout`` and ``probe-interval``, the
value of ``max-secondary-ttl`` MUST be a legal DURATION in VCL.
## Errors
As discussed above, the configuration values in the annotations for
self-sharding must be legal for the generated VCL (for example with
correct use of DURATION values). Otherwise, VCL will fail to load, and
the desired state for the Ingress containing the annotations will not
be achieved.
Because of the asynchronous nature of Kubernetes, the validity of the
annotations unfortunately cannot be checked when the manifest is
applied. Errors only become known when the controller attempts to
load the generated VCL corresponding to the Ingress and the
self-sharding annotations, and Varnish responds with an error.
The ``probe`` object specfies properties of the
[health probes](https://varnish-cache.org/docs/6.1/reference/vcl.html#probes),
that Varnish instances in the cluster send to one another (since they
are backends for one another when self-sharding is implemented).
Check the log of the Ingress controller to verify successful load of a
configuration corresponding to the Ingress. If the load failed, the
log will contain error entries that include the VCL compiler error
message from Varnish.
See the [technical reference](/docs/ref-varnish-cfg.md#spec-self-sharding)
for ``VarnishConfig`` for details of the configuration, and the
[``examples/``](/examples/self-sharding) folder for a working example.
examples/self-sharding/README.md
View file @ 8e64d461
# Self-sharding Varnish cluster
The manifest in this folder is an example configuration for an Ingress
with annotations for sharding the cache in a Varnish cluster. See
[the documentation](/docs/self-sharding.md) for details.
The manifest in this folder is an example specification for a
``VarnishConfig`` Custom Resource that defines sharding the cache in a
Varnish cluster. See [the documentation](/docs/self-sharding.md) for a
high-level discussion of the concept.
The Ingress may be deployed with the Services from
[the "cafe" example](/examples/hello).
```
$ kubectl apply -f self-sharding-cfg.yaml
```
The YAML specifies the API group and version for the Custom Resource,
and ``VarnishConfig`` as the ``kind``:
```
apiVersion: "ingress.varnish-cache.org/v1alpha1"
kind: VarnishConfig
```
The ``spec`` section of the ``VarnishConfig`` manifest MUST include the
``services`` array, which MUST have at least one element. Strings in
this array name Services in the same namespace in which the Custom
Resource is defined, identifying the Varnish-as-Ingress Services to
which the configuration is to be applied:
```
# Apply the configuration to the Service 'varnish-ingress' in the
# same namespace.
services:
- varnish-ingress
```
Self-sharding is applied if the ``self-sharding`` object is present in
the ``VarnishConfig`` resource. All of its config elements are optional,
and default values hold if they are left out. To just specify self-sharding
with all defaults, include ``self-sharding`` in the manifest as an
empty object:
```
$ kubectl apply -f cafe-ingress-selfshard.yaml
# Implement self-sharding in the Varnish Services with all default
# options.
self-sharding: {}
```
Its rules specification is the same as those in the Ingress from the cafe
example, but it contains annotations to configure self-sharding:
The sample YAML sets values for all of the possible options:
```
kind: Ingress
metadata:
name: cafe-ingress-varnish
annotations:
kubernetes.io/ingress.class: "varnish"
ingress.varnish-cache.org/self-sharding: "on"
ingress.varnish-cache.org/self-sharding-probe-timeout: "6s"
ingress.varnish-cache.org/self-sharding-probe-interval: "6s"
ingress.varnish-cache.org/self-sharding-probe-initial: "2"
ingress.varnish-cache.org/self-sharding-probe-window: "4"
ingress.varnish-cache.org/self-sharding-probe-threshold: "3"
ingress.varnish-cache.org/self-sharding-max-secondary-ttl: "1m"
namespace: varnish-ingress
self-sharding:
max-secondary-ttl: 2m
probe:
timeout: 6s
interval: 6s
initial: 2
window: 4
threshold: 3
```
Only the first annotation (``self-sharding``) is required to activate
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment