New docs structure according to the scoof ToC

doc/sphinx/index.rst

@@ -16,7 +16,7 @@ our tutorial - :ref:`tutorial-index`.
 Contents:
 
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 2
 
    installation/index.rst
    tutorial/index.rst

doc/sphinx/reference/varnishd.rst

@@ -31,6 +31,8 @@ DESCRIPTION
 The varnishd daemon accepts HTTP requests from clients, passes them on to a backend server and caches the
 returned documents to better satisfy future requests for the same document.
 
+.. _ref-varnishd-options:
+
 OPTIONS
 =======
 

doc/sphinx/tutorial/introduction.rst

 .. _tutorial-intro:
 
 What is Varnish?
--------------
+----------------
 
 Varnish Cache is a web application accelerator. It can also be called
 a HTTP reverse proxy. The next chapter :ref:`tutorial-web-accelerator`
@@ -27,14 +27,14 @@ where you want to get the content and how the request or response
 should be altered. 
 
 Supported plattforms
------------------
+--------------------
 
 Varnish is written to run on modern versions of Linux and FreeBSD and
 the best experience is had on those plattforms. Thanks to our
 contributors it also runs on NetBSD, OpenBSD and OS X.
 
 About the Varnish development process
--------------------------------
+-------------------------------------
 
 Varnish is a community driven project. The development is overseen by
 the Varnish Governing Board which currently consist of Poul-Henning
@@ -42,7 +42,7 @@ Kamp (Architect), Rogier Mulhuijzen (Fastly) and Kristian Lyngstøl
 (Varnish Software).
 
 Getting in touch
--------------
+----------------
 
 You can get in touch with us trough many channels. For real time chat
 you can reach us on IRC trough the server irc.linpro.net on the

doc/sphinx/tutorial/web_accelerator.rst

@@ -3,6 +3,8 @@
 What is a web accelerator
 -------------------------
 
+Really.XXX.
+
 
 The problem
 -----------

doc/sphinx/users-guide/command_line.rst → doc/sphinx/users-guide/command-line.rst

@@ -2,8 +2,8 @@
 
 XXX: Total rewrite of this
 
-Starting Varnish
-----------------
+Command Line options
+--------------------
 
 I assume varnishd is in your path. You might want to run ``pkill
 varnishd`` to make sure varnishd isn't running. 
@@ -39,17 +39,6 @@ I added a few options, lets go through them:
  HTTP requests. For a production environment you would probably make
  Varnish listen on port 80, which is the default.
 
-Now you have Varnish running. Let us make sure that it works
-properly. Use your browser to go to http://192.168.2.2:8080/
-(obviously, you should replace the IP address with one on your own
-system) - you should now see your web application running there.
+For a complete list of the command line parameters please see
+:ref:`ref-varnishd-options`.
 
-Whether or not the application actually goes faster when run through
-Varnish depends on a few factors. If you application uses cookies for
-every session (a lot of PHP and Java applications seem to send a
-session cookie if it is needed or not) or if it uses authentication
-chances are Varnish won't do much caching. Ignore that for the moment,
-we come back to that in :ref:`users-guide-increasing_your_hitrate`.
-
-Lets make sure that Varnish really does do something to your web
-site. To do that we'll take a look at the logs.

doc/sphinx/users-guide/configuration.rst

+Configuration
+=============
+
+This should deal with 
+
+.. toctree::
+   :maxdepth: 2
+
+   command-line
+   storage-backends
+   params
+

doc/sphinx/users-guide/index.rst

 .. _users-guide-index:
 
-%%%%%%%%%%%%%
-Using Varnish
-%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+Using Varnish - A Users Guide
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
 This guide is intended for system administrators managing Varnish
 Cache. 
@@ -10,13 +10,24 @@ Cache.
 The guide is split into short chapters, each chapter explaining a
 separate topic.
 
-.. toctree:: :maxdepth: 1
+.. toctree::
+   :maxdepth: 3
 
+   configuration
+   vcl
+   operation
+   troubleshooting
+
+.. customizing (which is a non ideal title)
+
+.. No longer used:
+
+        configuration
         command_line
+        VCL
 	backend_servers
 	logging
         sizing_your_cache
-	vcl
         statistics
         increasing_your_hitrate
 	cookies

doc/sphinx/users-guide/operation-cli.rst

+
+
+Varnishadm
+----------
+
+You connect to it and everything becomes awesome.

doc/sphinx/users-guide/operation.rst

+Operation
+=========
+
+.. toctree::
+   :maxdepth: 2
+
+   operation-logging
+   operation-statistics
+   operation-cli
+   purging
+   sizing-your-cache
+   increasing-your-hitrate
+   compression
+   esi
+   vary
+   cookies
+   virtualized
\ No newline at end of file

doc/sphinx/users-guide/params.rst

+
+
+Parameters
+----------
\ No newline at end of file

doc/sphinx/users-guide/purging.rst

 .. _users-guide-purging:
 
-=====================
- Purging and banning
-=====================
+
+Purging and banning
+-------------------
 
 One of the most effective ways of increasing your hit ratio is to
 increase the time-to-live (ttl) of your objects. But, as you're aware
@@ -15,7 +15,7 @@ banning and forced cache misses. First, let me explain the HTTP purges.
 
 
 HTTP Purges
-===========
+-----------
 
 A *purge* is what happens when you pick out an object from the cache
 and discard it along with its variants. Usually a purge is invoked
@@ -75,7 +75,7 @@ And Varnish would then discard the front page. This will remove all
 variants as defined by Vary.
 
 Bans
-====
+----
 
 There is another way to invalidate content: Bans. You can think of
 bans as a sort of a filter on objects already in the cache. You *ban*
@@ -164,7 +164,7 @@ be marked as Gone if it is a duplicate ban, but is still kept in the list
 for optimization purposes.
 
 Forcing a cache miss
-====================
+--------------------
 
 The final way to invalidate an object is a method that allows you to
 refresh an object by forcing a hash miss for a single request. If you set

doc/sphinx/users-guide/storage-backends.rst

+
+
+Storage backends
+----------------
+
+Intro
+~~~~~
+
+Malloc
+~~~~~~
+
+File
+~~~~
+
+Persistent
+~~~~~~~~~~
+
+Transient
+~~~~~~~~~
+
+

doc/sphinx/users-guide/troubleshooting.rst

 Troubleshooting Varnish
------------------------
+=======================
 
 Sometimes Varnish misbehaves. In order for you to understand whats
 going on there are a couple of places you can check. varnishlog,
 /var/log/syslog, /var/log/messages are all places where varnish might
-leave clues of whats going on.
+leave clues of whats going on. This chapter will guide you through
+basic troubleshooting in Varnish.
 
 
 When Varnish won't start
-~~~~~~~~~~~~~~~~~~~~~~~~
+------------------------
 
 Sometimes Varnish wont start. There is a plethora of reasons why
 Varnish wont start on your machine. We've seen everything from wrong
@@ -18,7 +19,7 @@ Starting Varnish in debug mode to see what is going on.
 
 Try to start varnish by::
 
-    # varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1:2000  -a 0.0.0.0:8080 -d
+    # varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1: 2000  -a 0.0.0.0:8080 -d
 
 Notice the -d option. It will give you some more information on what
 is going on. Let us see how Varnish will react to something else
@@ -51,7 +52,7 @@ on IRC.
 
 
 Varnish is crashing
-~~~~~~~~~~~~~~~~~~~
+-------------------
 
 When varnish goes bust the child processes crashes. Usually the mother
 process will manage this by restarting the child process again. Any
@@ -70,7 +71,7 @@ XXX: Describe crashing child process and crashing mother process here too.
 XXX: panic.show
 
 Varnish gives me Guru meditation
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+--------------------------------
 
 First find the relevant log entries in varnishlog. That will probably
 give you a clue. Since varnishlog logs so much data it might be hard
@@ -90,7 +91,7 @@ filtering capabilities and explanation of the various options.
 
 
 Varnish doesn't cache
-~~~~~~~~~~~~~~~~~~~~~
+---------------------
 
 See :ref:`users-guide-increasing_your_hitrate`.
 

doc/sphinx/users-guide/vcl-actions.rst

+actions
+~~~~~~~
+
+The most common actions to return are these:
+
+*pass*
+ When you return pass the request and subsequent response will be passed to
+ and from the backend server. It won't be cached. pass can be returned from
+ vcl_recv
+
+*hit_for_pass*
+  Similar to pass, but accessible from vcl_fetch. Unlike pass, hit_for_pass
+  will create a hitforpass object in the cache. This has the side-effect of
+  caching the decision not to cache. This is to allow would-be uncachable
+  requests to be passed to the backend at the same time. The same logic is
+  not necessary in vcl_recv because this happens before any potential
+  queueing for an object takes place.
+
+*lookup*
+  When you return lookup from vcl_recv you tell Varnish to deliver content 
+  from cache even if the request othervise indicates that the request 
+  should be passed. You can't return lookup from vcl_fetch.
+
+*pipe*
+  Pipe can be returned from vcl_recv as well. Pipe short circuits the
+  client and the backend connections and Varnish will just sit there
+  and shuffle bytes back and forth. Varnish will not look at the data being 
+  send back and forth - so your logs will be incomplete. 
+  Beware that with HTTP 1.1 a client can send several requests on the same 
+  connection and so you should instruct Varnish to add a "Connection: close"
+  header before actually returning pipe. 
+
+*deliver*
+ Deliver the cached object to the client.  Usually returned from vcl_fetch. 

doc/sphinx/users-guide/vcl-examples.rst

+Example 1 - manipulating headers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Lets say we want to remove the cookie for all objects in the /images
+directory of our web server::
+
+  sub vcl_recv {
+    if (req.url ~ "^/images") {
+      unset req.http.cookie;
+    }
+  }
+
+Now, when the request is handled to the backend server there will be
+no cookie header. The interesting line is the one with the
+if-statement. It matches the URL, taken from the request object, and
+matches it against the regular expression. Note the match operator. If
+it matches the Cookie: header of the request is unset (deleted). 
+
+Example 2 - manipulating beresp
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Here we override the TTL of a object comming from the backend if it
+matches certain criteria::
+
+  sub vcl_fetch {
+     if (req.url ~ "\.(png|gif|jpg)$") {
+       unset beresp.http.set-cookie;
+       set beresp.ttl = 1h;
+    }
+  }
+
+Example 3 - ACLs
+~~~~~~~~~~~~~~~~
+
+You create a named access control list with the *acl* keyword. You can match
+the IP address of the client against an ACL with the match operator.::
+
+  # Who is allowed to purge....
+  acl local {
+      "localhost";
+      "192.168.1.0"/24; /* and everyone on the local network */
+      ! "192.168.1.23"; /* except for the dialin router */
+  }
+  
+  sub vcl_recv {
+    if (req.request == "PURGE") {
+      if (client.ip ~ local) {
+         return(lookup);
+      }
+    } 
+  }
+  
+  sub vcl_hit {
+     if (req.request == "PURGE") {
+       set obj.ttl = 0s;
+       error 200 "Purged.";
+      }
+  }
+
+  sub vcl_miss {
+    if (req.request == "PURGE") {
+      error 404 "Not in cache.";
+    }
+  }
+

doc/sphinx/users-guide/hashing.rst → doc/sphinx/users-guide/vcl-hashing.rst

@@ -38,11 +38,11 @@ In vcl_recv::
 
   set req.http.X-Country-Code = geoip.lookup(client.ip);
 
-And then add a vcl_hash:
+And then add a vcl_hash::
 
-sub vcl_hash {
-  hash_data(req.http.X-Country-Code);
-}
+ sub vcl_hash {
+   hash_data(req.http.X-Country-Code);
+ }
 
 As the default VCL will take care of adding the host and URL to the
 hash we don't have to do anything else. Be careful calling

doc/sphinx/users-guide/advanced_topics.rst → doc/sphinx/users-guide/vcl-inline-c.rst

-.. _users-guide-advanced_topics:
 
-Advanced topics
----------------
 
-This guide has covered the basics in Varnish. If you read through
-it all you should now have the skills to run Varnish.
-
-Here is a short overview of topics that we haven't covered in the guide. 
-
-More VCL
-~~~~~~~~
-
-VCL is a bit more complex then what we've covered so far. There are a
-few more subroutines available and there a few actions that we haven't
-discussed. For a complete(ish) guide to VCL have a look at the VCL man
-page - ref:`reference-vcl`.
 
 Using In-line C to extend Varnish
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+---------------------------------
 
-(Here there be dragons)
+(Here there be dragons. Big and mean ones.)
 
 You can use *in-line C* to extend Varnish. Please note that you can
 seriously mess up Varnish this way. The C code runs within the Varnish
@@ -27,15 +12,13 @@ Cache process so if your code generates a segfault the cache will crash.
 
 One of the first uses I saw of In-line C was logging to syslog.::
 
-	# The include statements must be outside the subroutines.
-	C{
-		#include <syslog.h>
+        # The include statements must be outside the subroutines.
+        C{
+                #include <syslog.h>
         }C
-	
+        
         sub vcl_something {
                 C{
-		        syslog(LOG_INFO, "Something happened at VCL line XX.");
-	        }C
+                        syslog(LOG_INFO, "Something happened at VCL line XX.");
+                }C
         }
-
-

doc/sphinx/users-guide/vcl-intro.rst

+Varnish Configuration Language - VCL
+-------------------------------------
+
+Varnish has a great configuration system. Most other systems use
+configuration directives, where you basically turn on and off lots of
+switches. Varnish uses a domain specific language called Varnish
+Configuration Language, or VCL for short. Varnish translates this
+configuration into binary code which is then executed when requests
+arrive.
+
+The VCL files are divided into subroutines. The different subroutines
+are executed at different times. One is executed when we get the
+request, another when files are fetched from the backend server.
+
+Varnish will execute these subroutines of code at different stages of
+its work. Because it is code it is execute line by line precedence
+isn't a problem. At some point you call an action in this subroutine
+and then the execution of the subroutine stops.
+
+If you don't call an action in your subroutine and it reaches the end
+Varnish will execute some built in VCL code. You will see this VCL
+code commented out in default.vcl.
+
+99% of all the changes you'll need to do will be done in two of these
+subroutines. *vcl_recv* and *vcl_fetch*.
+
+
+.. _users-guide-vcl_fetch_actions:
+
+

doc/sphinx/users-guide/vcl-subs.rst

+
+vcl_recv
+~~~~~~~~
+
+vcl_recv (yes, we're skimpy with characters, it's Unix) is called at
+the beginning of a request, after the complete request has been
+received and parsed.  Its purpose is to decide whether or not to serve
+the request, how to do it, and, if applicable, which backend to use.
+
+In vcl_recv you can also alter the request. Typically you can alter
+the cookies and add and remove request headers.
+
+Note that in vcl_recv only the request object, req is available.
+
+vcl_fetch
+~~~~~~~~~
+
+vcl_fetch is called *after* a document has been successfully retrieved
+from the backend. Normal tasks her are to alter the response headers,
+trigger ESI processing, try alternate backend servers in case the
+request failed.
+
+In vcl_fetch you still have the request object, req, available. There
+is also a *backend response*, beresp. beresp will contain the HTTP
+headers from the backend.

doc/sphinx/users-guide/vcl-syntax.rst

+VCL Syntax
+----------
+
+VCL has inherited a lot from C and it reads much like simple C or Perl.
+
+Blocks are delimited by curly braces, statements end with semicolons,
+and comments may be written as in C, C++ or Perl according to your own
+preferences.
+
+Note that VCL doesn't contain any loops or jump statements.
+
+
+Strings
+~~~~~~~
+
+
+
+Access control lists (ACLs)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+Operators
+~~~~~~~~~
+
+The following operators are available in VCL. See the examples further
+down for, uhm, examples.
+
+= 
+ Assignment operator.
+
+== 
+ Comparison.
+
+~
+ Match. Can either be used with regular expressions or ACLs.
+
+!
+ Negation.
+
+&&
+ Logical *and*
+
+||
+ Logical *or*
+

doc/sphinx/users-guide/vcl-variables.rst

+
+Requests, responses and objects
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In VCL, there are three important data structures. The request, coming
+from the client, the response coming from the backend server and the
+object, stored in cache.
+
+In VCL you should know the following structures.
+
+*req*
+ The request object. When Varnish has received the request the req object is 
+ created and populated. Most of the work you do in vcl_recv you 
+ do on or with the req object.
+
+*beresp*
+ The backend respons object. It contains the headers of the object 
+ comming from the backend. Most of the work you do in vcl_fetch you 
+ do on the beresp object.
+
+*obj*
+ The cached object. Mostly a read only object that resides in memory. 
+ obj.ttl is writable, the rest is read only.

doc/sphinx/users-guide/vcl.rst

-Varnish Configuration Language - VCL
--------------------------------------
+VCL
+---
 
-Varnish has a great configuration system. Most other systems use
-configuration directives, where you basically turn on and off lots of
-switches. Varnish uses a domain specific language called Varnish
-Configuration Language, or VCL for short. Varnish translates this
-configuration into binary code which is then executed when requests
-arrive.
 
-The VCL files are divided into subroutines. The different subroutines
-are executed at different times. One is executed when we get the
-request, another when files are fetched from the backend server.
+Yes. Is great. Ja.
 
-Varnish will execute these subroutines of code at different stages of
-its work. Because it is code it is execute line by line precedence
-isn't a problem. At some point you call an action in this subroutine
-and then the execution of the subroutine stops.
 
-If you don't call an action in your subroutine and it reaches the end
-Varnish will execute some built in VCL code. You will see this VCL
-code commented out in default.vcl.
-
-99% of all the changes you'll need to do will be done in two of these
-subroutines. *vcl_recv* and *vcl_fetch*.
-
-vcl_recv
-~~~~~~~~
-
-vcl_recv (yes, we're skimpy with characters, it's Unix) is called at
-the beginning of a request, after the complete request has been
-received and parsed.  Its purpose is to decide whether or not to serve
-the request, how to do it, and, if applicable, which backend to use.
-
-In vcl_recv you can also alter the request. Typically you can alter
-the cookies and add and remove request headers.
-
-Note that in vcl_recv only the request object, req is available.
-
-vcl_fetch
-~~~~~~~~~
-
-vcl_fetch is called *after* a document has been successfully retrieved
-from the backend. Normal tasks her are to alter the response headers,
-trigger ESI processing, try alternate backend servers in case the
-request failed.
-
-In vcl_fetch you still have the request object, req, available. There
-is also a *backend response*, beresp. beresp will contain the HTTP
-headers from the backend.
-
-.. _users-guide-vcl_fetch_actions:
-
-actions
-~~~~~~~
-
-The most common actions to return are these:
-
-*pass*
- When you return pass the request and subsequent response will be passed to
- and from the backend server. It won't be cached. pass can be returned from
- vcl_recv
-
-*hit_for_pass*
-  Similar to pass, but accessible from vcl_fetch. Unlike pass, hit_for_pass
-  will create a hitforpass object in the cache. This has the side-effect of
-  caching the decision not to cache. This is to allow would-be uncachable
-  requests to be passed to the backend at the same time. The same logic is
-  not necessary in vcl_recv because this happens before any potential
-  queueing for an object takes place.
-
-*lookup*
-  When you return lookup from vcl_recv you tell Varnish to deliver content 
-  from cache even if the request othervise indicates that the request 
-  should be passed. You can't return lookup from vcl_fetch.
-
-*pipe*
-  Pipe can be returned from vcl_recv as well. Pipe short circuits the
-  client and the backend connections and Varnish will just sit there
-  and shuffle bytes back and forth. Varnish will not look at the data being 
-  send back and forth - so your logs will be incomplete. 
-  Beware that with HTTP 1.1 a client can send several requests on the same 
-  connection and so you should instruct Varnish to add a "Connection: close"
-  header before actually returning pipe. 
-
-*deliver*
- Deliver the cached object to the client.  Usually returned from vcl_fetch. 
-
-Requests, responses and objects
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-In VCL, there are three important data structures. The request, coming
-from the client, the response coming from the backend server and the
-object, stored in cache.
-
-In VCL you should know the following structures.
-
-*req*
- The request object. When Varnish has received the request the req object is 
- created and populated. Most of the work you do in vcl_recv you 
- do on or with the req object.
-
-*beresp*
- The backend respons object. It contains the headers of the object 
- comming from the backend. Most of the work you do in vcl_fetch you 
- do on the beresp object.
-
-*obj*
- The cached object. Mostly a read only object that resides in memory. 
- obj.ttl is writable, the rest is read only.
-
-Operators
-~~~~~~~~~
-
-The following operators are available in VCL. See the examples further
-down for, uhm, examples.
-
-= 
- Assignment operator.
-
-== 
- Comparison.
-
-~
- Match. Can either be used with regular expressions or ACLs.
-
-!
- Negation.
-
-&&
- Logical *and*
-
-||
- Logical *or*
-
-Example 1 - manipulating headers
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Lets say we want to remove the cookie for all objects in the /images
-directory of our web server::
-
-  sub vcl_recv {
-    if (req.url ~ "^/images") {
-      unset req.http.cookie;
-    }
-  }
-
-Now, when the request is handled to the backend server there will be
-no cookie header. The interesting line is the one with the
-if-statement. It matches the URL, taken from the request object, and
-matches it against the regular expression. Note the match operator. If
-it matches the Cookie: header of the request is unset (deleted). 
-
-Example 2 - manipulating beresp
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Here we override the TTL of a object comming from the backend if it
-matches certain criteria::
-
-  sub vcl_fetch {
-     if (req.url ~ "\.(png|gif|jpg)$") {
-       unset beresp.http.set-cookie;
-       set beresp.ttl = 1h;
-    }
-  }
-
-Example 3 - ACLs
-~~~~~~~~~~~~~~~~
-
-You create a named access control list with the *acl* keyword. You can match
-the IP address of the client against an ACL with the match operator.::
-
-  # Who is allowed to purge....
-  acl local {
-      "localhost";
-      "192.168.1.0"/24; /* and everyone on the local network */
-      ! "192.168.1.23"; /* except for the dialin router */
-  }
-  
-  sub vcl_recv {
-    if (req.request == "PURGE") {
-      if (client.ip ~ local) {
-         return(lookup);
-      }
-    } 
-  }
-  
-  sub vcl_hit {
-     if (req.request == "PURGE") {
-       set obj.ttl = 0s;
-       error 200 "Purged.";
-      }
-  }
-
-  sub vcl_miss {
-    if (req.request == "PURGE") {
-      error 404 "Not in cache.";
-    }
-  }
+.. toctree::
+   :maxdepth: 2
 
+   vcl-intro
+   vcl-syntax
+   vcl-variables
+   vcl-actions
+   vcl-subs
+   vcl-backends
+   vcl-hashing
+   vcl-saint-and-grace
+   vcl-inline-c
+   vcl-examples
+   websockets
+   devicedetection
+   
\ No newline at end of file