Partial rewrite of the docs on writing a director

Better ordering of the topics, and hopefully better wording overall.

doc/sphinx/reference/directors.rst

@@ -54,59 +54,49 @@ A director can be summed up as:
 - a set of operations
 - the associated state
 
-What's the difference between a *cluster* director and a *backend* director?
+The difference between a *cluster* director and a *backend* director is mainly
 The functions they will implement.
 
 
-Creating a Director
-===================
-
-Custom Backends
----------------
-
-If you want to implement a custom backend, have a look at how Varnish
-implements native backends. It is the canonical implementation, and though it
-provides other services like connection pooling or statistics, it is
-essentially a director which state is a ``struct backend``. Varnish native
-backends currently speak HTTP/1 over TCP, and as such, you need to make your
-own custom backend if you want Varnish to do otherwise such as connect over
-UDP or UNIX-domain sockets or speak a different protocol.
+Cluster Directors
+=================
 
-You may also consider making your custom backend compliant with regards to the
-VCL state (see :ref:`ref-vmod-event-functions`).
+As in :ref:`vmod_directors(3)`, you can write directors that will group
+backends sharing the same role, and pick them according to a strategy. If you
+need more than the built-in strategies (round-robin, hash, ...), even though
+they can be stacked, it is always possible to write your own.
 
-A "backend" director must not implement the ``resolve`` function. More on that
-below (:ref:`ref-writing-a-director-cluster`).
+In this case you simply need to implement the ``resolve`` function for the
+director. Directors are walked until a leaf director is found. A leaf director
+doesn't have a ``resolve`` function and is used to actually make the backend
+request, just like the backends you declare in VCL.
 
 
 Dynamic Backends
-----------------
-
-.. TODO document the VRT_BACKEND_FIELDS dance
+================
 
 If you want to speak HTTP/1 over TCP, but for some reason VCL does not fit the
 bill, you can instead reuse the whole backend facility. It allows you for
 instance to add and remove backends on-demand without the need to reload your
 VCL. You can then leverage your provisioning system.
 
-You don't really deal with ``struct backend``, all you need is available
-through the runtime API. Consider the following snippet::
+Consider the following snippet::
 
     backend default {
         .host = "localhost";
     }
 
-The VCL compiler will turn this declaration into a ``struct vrt_backend``. When
-the VCL is loaded, Varnish will call ``VRT_new_backend`` in order to create the
-director. Dynamic backends are built just like static backends, one *struct* at
-a time. You can get rid of the ``struct vrt_backend`` as soon as you have the
-``struct director``.
+The VCL compiler turns this declaration into a ``struct vrt_backend``. When the
+VCL is loaded, Varnish calls ``VRT_new_backend`` in order to create the
+director. Varnish doesn't expose its data structure for actual backends, only
+the director abstraction and dynamic backends are built just like static
+backends, one *struct* at a time. You can get rid of the ``struct vrt_backend``
+as soon as you have the ``struct director``.
 
-Unlike a custom backend, a dynamic backend can't exceed its VCL's lifespan,
-because native backends are *owned* by VCLs. Though a dynamic backend can't
-outlive its VCL, it can be deleted any time with ``VRT_delete_backend``. The
-VCL will delete the remaining backends once discarded, you don't need to take
-care of it.
+A (dynamic) backend can't exceed its VCL's lifespan, because native backends
+are *owned* by VCLs. Though a dynamic backend can't outlive its VCL, it can be
+deleted any time with ``VRT_delete_backend``. The VCL will delete the remaining
+backends once discarded, you don't need to take care of it.
 
 Consider using an object (see :ref:`ref-vmod-objects`) to manipulate dynamic
 backends. They are tied to the VCL life cycle and make a handy data structure
@@ -124,20 +114,6 @@ subscribe to VCL events and watch for VCL state (see
 
 .. _ref-writing-a-director-cluster:
 
-Cluster Directors
------------------
-
-As in :ref:`vmod_directors(3)`, you can write directors that will group
-backends sharing the same role, and pick them according to a strategy. If you
-need more than the built-in strategies (round-robin, hash, ...), even though
-they can be stacked, it is always possible to write your own.
-
-In this case you simply need to implement the ``resolve`` function for the
-director. Directors are walked until a leaf director is found. A leaf director
-doesn't have a ``resolve`` function and is used to actually make the backend
-request.
-
-
 Health Probes
 =============
 
@@ -157,3 +133,39 @@ probe and disable the feature on a cold VCL (see
 
 Instead of initializing your own probe definition, you can get a ``VCL_PROBE``
 directly built from VCL (see :ref:`ref-vmod-vcl-c-types`).
+
+What's the difference ?
+
+
+Custom Backends
+===============
+
+If you want to implement a custom backend, have a look at how Varnish
+implements native backends. It is the canonical implementation, and though it
+provides other services like connection pooling or statistics, it is
+essentially a director which state is a ``struct backend``. Varnish native
+backends currently speak HTTP/1 over TCP, and as such, you need to make your
+own custom backend if you want Varnish to do otherwise such as connect over
+UDP or UNIX-domain sockets or speak a different protocol.
+
+If you want to leverage probes declarations in VCL, which have the advantage of
+being reusable since they are only specifications, you can. However, you need
+to implement the whole probing infrastructure from scratch.
+
+You may also consider making your custom backend compliant with regards to the
+VCL state (see :ref:`ref-vmod-event-functions`).
+
+
+Data structure considerations
+-----------------------------
+
+When you are creating a custom backend, you may want to provide the semantics
+of the native backends. In this case, instead of repeating the redundant fields
+between data structures, you can use the macros ``VRT_BACKEND_FIELDS`` and
+``VRT_BACKEND_PROBE_FIELDS`` to declare them all at once. This is the little
+dance Varnish uses to copy data between the ``struct vrt_backend`` and its
+internal data structure for example.
+
+The copy can be automated with the macros ``VRT_BACKEND_HANDLE`` and
+``VRT_BACKEND_PROBE_HANDLE``. You can look at how they can be used in the
+Varnish code base.