Move storage backend prosa into users guide. Rework cmd line and config

doc/sphinx/include/storage_backends.rst

-
-malloc
-~~~~~~
-
-syntax: malloc[,size]
-
-Malloc is a memory based backend. Each object will be allocated from
-memory. If your system runs low on memory swap will be used. Be aware
-that the size limitation only limits the actual storage and that
-approximately 1k of memory per object will be used for various
-internal structures.
-
-The size parameter specifies the maximum amount of memory varnishd
-will allocate.  The size is assumed to be in bytes, unless followed by
-one of the following suffixes:
-
-      K, k    The size is expressed in kibibytes.
-
-      M, m    The size is expressed in mebibytes.
-
-      G, g    The size is expressed in gibibytes.
-
-      T, t    The size is expressed in tebibytes.
-
-The default size is unlimited. 
-
-Mallocs performance is bound by memory speed so it is very fast. 
-
-file
-~~~~
-
-syntax: file[,path[,size[,granularity]]]
-
-The file backend stores objects in memory backed by a file on disk
-with mmap. This is the default storage backend and unless you specify
-another storage this one will used along with Transient storage.
-
-The path parameter specifies either the path to the backing file or
-the path to a directory in which varnishd will create the backing
-file.  The default is /tmp.
-
-The size parameter specifies the size of the backing file.  The size
-is assumed to be in bytes, unless fol‐ lowed by one of the following
-suffixes:
-
-      K, k    The size is expressed in kibibytes.
-
-      M, m    The size is expressed in mebibytes.
-
-      G, g    The size is expressed in gibibytes.
-
-      T, t    The size is expressed in tebibytes.
-
-      %       The size is expressed as a percentage of the free space on the
-              file system where it resides.
-
-The default size is 50%.
-
-If the backing file already exists, it will be truncated or expanded
-to the specified size.
-
-Note that if varnishd has to create or expand the file, it will not
-pre-allocate the added space, leading to fragmentation, which may
-adversely impact performance.  Pre-creating the storage file using
-dd(1) will reduce fragmentation to a minimum.
-
-The granularity parameter specifies the granularity of
-allocation.  All allocations are rounded up to this size.  The
-is assumed to be in bytes, unless followed by one of the
-suffixes described for size except for %.
-
-The default size is the VM page size.  The size should be reduced if
-you have many small objects.
-
-File performance is typically limited by the write speed of the
-device, and depending on use, the seek time.
-
-persistent (experimental)
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-syntax: persistent,path,size {experimental}
-
-Persistent storage. Varnish will store objects in a file in a manner
-that will secure the survival of *most* of the objects in the event of
-a planned or unplanned shutdown of Varnish.
-
-The path parameter specifies the path to the backing file. If
-the file doesn't exist Varnish will create it.
-
-The size parameter specifies the size of the backing file.  The
-size is assumed to be in bytes, unless followed by one of the
-following suffixes:
-
-      K, k    The size is expressed in kibibytes.
-
-      M, m    The size is expressed in mebibytes.
-
-      G, g    The size is expressed in gibibytes.
-
-      T, t    The size is expressed in tebibytes.
-
-Varnish will split the file into logical *silos* and write to the
-silos in the manner of a circular buffer. Only one silo will be kept
-open at any given point in time. Full silos are *sealed*. When Varnish
-starts after a shutdown it will discard the content of any silo that
-isn't sealed.
-
-Note that taking persistent silos offline and at the same time using
-bans can cause problems. This because bans added while the silo was
-offline will not be applied to the silo when it reenters the cache,
-and can make previously banned objects reappear.
-
-Transient Storage
------------------
-      
-If you name any of your storage backend "Transient" it will be
-used for transient (short lived) objects. By default Varnish
-would use an unlimited malloc backend for this.

doc/sphinx/reference/varnishd.rst

@@ -157,7 +157,124 @@ Storage Types
 
 The following storage types are available:
 
-.. include:: ../include/storage_backends.rst
+malloc
+~~~~~~
+
+syntax: malloc[,size]
+
+Malloc is a memory based backend. Each object will be allocated from
+memory. If your system runs low on memory swap will be used. Be aware
+that the size limitation only limits the actual storage and that
+approximately 1k of memory per object will be used for various
+internal structures.
+
+The size parameter specifies the maximum amount of memory varnishd
+will allocate.  The size is assumed to be in bytes, unless followed by
+one of the following suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+The default size is unlimited. 
+
+Mallocs performance is bound by memory speed so it is very fast. 
+
+file
+~~~~
+
+syntax: file[,path[,size[,granularity]]]
+
+The file backend stores objects in memory backed by a file on disk
+with mmap. This is the default storage backend and unless you specify
+another storage this one will used along with Transient storage.
+
+The path parameter specifies either the path to the backing file or
+the path to a directory in which varnishd will create the backing
+file.  The default is /tmp.
+
+The size parameter specifies the size of the backing file.  The size
+is assumed to be in bytes, unless fol‐ lowed by one of the following
+suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+      %       The size is expressed as a percentage of the free space on the
+              file system where it resides.
+
+The default size is 50%.
+
+If the backing file already exists, it will be truncated or expanded
+to the specified size.
+
+Note that if varnishd has to create or expand the file, it will not
+pre-allocate the added space, leading to fragmentation, which may
+adversely impact performance.  Pre-creating the storage file using
+dd(1) will reduce fragmentation to a minimum.
+
+The granularity parameter specifies the granularity of
+allocation.  All allocations are rounded up to this size.  The
+is assumed to be in bytes, unless followed by one of the
+suffixes described for size except for %.
+
+The default size is the VM page size.  The size should be reduced if
+you have many small objects.
+
+File performance is typically limited by the write speed of the
+device, and depending on use, the seek time.
+
+persistent (experimental)
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+syntax: persistent,path,size {experimental}
+
+Persistent storage. Varnish will store objects in a file in a manner
+that will secure the survival of *most* of the objects in the event of
+a planned or unplanned shutdown of Varnish.
+
+The path parameter specifies the path to the backing file. If
+the file doesn't exist Varnish will create it.
+
+The size parameter specifies the size of the backing file.  The
+size is assumed to be in bytes, unless followed by one of the
+following suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+Varnish will split the file into logical *silos* and write to the
+silos in the manner of a circular buffer. Only one silo will be kept
+open at any given point in time. Full silos are *sealed*. When Varnish
+starts after a shutdown it will discard the content of any silo that
+isn't sealed.
+
+Note that taking persistent silos offline and at the same time using
+bans can cause problems. This because bans added while the silo was
+offline will not be applied to the silo when it reenters the cache,
+and can make previously banned objects reappear.
+
+Transient Storage
+-----------------
+      
+If you name any of your storage backend "Transient" it will be
+used for transient (short lived) objects. By default Varnish
+would use an unlimited malloc backend for this.
+
 
 Management Interface
 --------------------

doc/sphinx/tutorial/index.rst

@@ -22,5 +22,6 @@ Then we hand things over to the user guide.
    starting_varnish
    putting_varnish_on_port_80
    backend_servers
+   peculiarities.rst
    now_what
 

doc/sphinx/users-guide/command-line.rst

 .. _users-guide-command-line:
 
-XXX: Total rewrite of this
-
-Command Line options
---------------------
-
-I assume varnishd is in your path. You might want to run ``pkill
-varnishd`` to make sure varnishd isn't running. 
-
-Become root and type:
-
-``# varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1:2000 -a 0.0.0.0:8080``
-
-I added a few options, lets go through them:
-
-``-f /usr/local/etc/varnish/default.vcl``
- The -f options specifies what configuration varnishd should use.
-
-``-s malloc,1G``
- The -s options chooses the storage type Varnish should use for
- storing its content. I used the type *malloc*, which just uses memory
- for storage. There are other backends as well, described in 
- :ref:users-guide-storage. 1G specifies how much memory should be allocated 
- - one gigabyte. 
-
-``-T 127.0.0.1:2000``
- Varnish has a built-in text-based administration
- interface. Activating the interface makes Varnish manageble without
- stopping it. You can specify what interface the management interface
- should listen to. Make sure you don't expose the management interface
- to the world as you can easily gain root access to a system via the
- Varnish management interface. I recommend tieing it to localhost. If
- you have users on your system that you don't fully trust, use firewall
- rules to restrict access to the interface to root only.
-
-``-a 0.0.0.0:8080``
- I specify that I want Varnish to listen on port 8080 for incomming
- HTTP requests. For a production environment you would probably make
- Varnish listen on port 80, which is the default.
+Typical command line options
+----------------------------
+
+On a modern Linux distro the various options that are used when
+starting up Varnish are stored in /etc/default/varnish (Debian, Ubuntu) or
+/etc/sysconfig/varnish (Red Hat, Centos).
+
+There are quite a few options you can tweak but most of you will only
+need to change a few them.
+
+The typical command line options you want to change are:
+
+-a *listen_address*
+    What address should Varnish listen to. The default is to listen to
+    all IP adresses and stick to port 80. ":80" will ask Varnish to
+    listen to all adresses, both IPv4 and IPv6 and is probably a
+    sensible thing.
+ 
+-f *config file*
+     The -f options specifies what VCL file Varnish should use as the default.
+
+-s *storage options*
+
+     This is probably the most important one. The default is to use
+     the memory storage backend and to allocate a small amount of
+     memory. On a small site this might suffice. If you have dedicated
+     Varnish Cache server you most definitivly want to increase
+     the memory allocated or consider another backend. 
+     Please note that in addition to the memory allocated by the
+     storage engine itself Varnish also has internal data structures
+     that consume memory. More or less 1kb per object.  
+     See also :ref:`guide-storage`.
+
+-T *listen address*  
+     Varnish has a built-in text-based administration
+     interface. Activating the interface makes Varnish manageble
+     without stopping it. You can specify what interface the
+     management interface should listen to. Make sure you don't expose
+     the management interface to the world as you can easily gain root
+     access to a system via the Varnish management interface. I
+     recommend tieing it to localhost. If you have users on your
+     system that you don't fully trust, use firewall rules to restrict
+     access to the interface to root only.
 
 For a complete list of the command line parameters please see
 :ref:`ref-varnishd-options`.

doc/sphinx/users-guide/configuration.rst

 Configuration
 =============
 
-This should deal with 
+This chapter deals with configuration of Varnish Cache. It will not
+cover the actual configuration policy that's written in VCL. 
+
+This guide assumes you're running Debian, Ubuntu, Red Hat or
+Centos. If you're on any other platform you're probably used to doing
+some mental translations.
 
 .. toctree::
    :maxdepth: 2

doc/sphinx/users-guide/storage-backends.rst

 
 
+.. _guide-storage:
+
 Storage backends
 ----------------
 
@@ -9,5 +11,120 @@ Intro
 Varnish has pluggable storage backends. It can store data in various
 backends which have different performance characteristics.
 
-.. include:: ../include/storage_backends.rst
+malloc
+~~~~~~
+
+syntax: malloc[,size]
+
+Malloc is a memory based backend. Each object will be allocated from
+memory. If your system runs low on memory swap will be used. Be aware
+that the size limitation only limits the actual storage and that
+approximately 1k of memory per object will be used for various
+internal structures.
+
+The size parameter specifies the maximum amount of memory varnishd
+will allocate.  The size is assumed to be in bytes, unless followed by
+one of the following suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+The default size is unlimited. 
+
+Mallocs performance is bound by memory speed so it is very fast. 
+
+file
+~~~~
+
+syntax: file[,path[,size[,granularity]]]
+
+The file backend stores objects in memory backed by a file on disk
+with mmap. This is the default storage backend and unless you specify
+another storage this one will used along with Transient storage.
+
+The path parameter specifies either the path to the backing file or
+the path to a directory in which varnishd will create the backing
+file.  The default is /tmp.
+
+The size parameter specifies the size of the backing file.  The size
+is assumed to be in bytes, unless fol‐ lowed by one of the following
+suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+      %       The size is expressed as a percentage of the free space on the
+              file system where it resides.
+
+The default size is 50%.
+
+If the backing file already exists, it will be truncated or expanded
+to the specified size.
+
+Note that if varnishd has to create or expand the file, it will not
+pre-allocate the added space, leading to fragmentation, which may
+adversely impact performance.  Pre-creating the storage file using
+dd(1) will reduce fragmentation to a minimum.
+
+The granularity parameter specifies the granularity of
+allocation.  All allocations are rounded up to this size.  The
+is assumed to be in bytes, unless followed by one of the
+suffixes described for size except for %.
+
+The default size is the VM page size.  The size should be reduced if
+you have many small objects.
+
+File performance is typically limited by the write speed of the
+device, and depending on use, the seek time.
+
+persistent (experimental)
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+syntax: persistent,path,size {experimental}
+
+Persistent storage. Varnish will store objects in a file in a manner
+that will secure the survival of *most* of the objects in the event of
+a planned or unplanned shutdown of Varnish.
+
+The path parameter specifies the path to the backing file. If
+the file doesn't exist Varnish will create it.
+
+The size parameter specifies the size of the backing file.  The
+size is assumed to be in bytes, unless followed by one of the
+following suffixes:
+
+      K, k    The size is expressed in kibibytes.
+
+      M, m    The size is expressed in mebibytes.
+
+      G, g    The size is expressed in gibibytes.
+
+      T, t    The size is expressed in tebibytes.
+
+Varnish will split the file into logical *silos* and write to the
+silos in the manner of a circular buffer. Only one silo will be kept
+open at any given point in time. Full silos are *sealed*. When Varnish
+starts after a shutdown it will discard the content of any silo that
+isn't sealed.
+
+Note that taking persistent silos offline and at the same time using
+bans can cause problems. This because bans added while the silo was
+offline will not be applied to the silo when it reenters the cache,
+and can make previously banned objects reappear.
 
+Transient Storage
+-----------------
+      
+If you name any of your storage backend "Transient" it will be
+used for transient (short lived) objects. By default Varnish
+would use an unlimited malloc backend for this.