Merge branch 'list_safety'

[exim.git] / doc / doc-docbook / HowItWorks.txt

diff --git a/doc/doc-docbook/HowItWorks.txt b/doc/doc-docbook/HowItWorks.txt
index 4c51ae34db4d548ec591890964287504f1c1eda6..c5e328a09408245f464bd531e29a211ac22969f4 100644 (file)
--- a/doc/doc-docbook/HowItWorks.txt
+++ b/doc/doc-docbook/HowItWorks.txt
@@ -1,5 +1,3 @@
-$Cambridge: exim/doc/doc-docbook/HowItWorks.txt,v 1.6 2007/04/11 15:26:09 ph10 Exp $
-

 CREATING THE EXIM DOCUMENTATION
 
 "You are lost in a maze of twisty little scripts."
 CREATING THE EXIM DOCUMENTATION
 
 "You are lost in a maze of twisty little scripts."


@@ -74,10 +72,10 @@ adopted without affecting the source maintenance.
 A number of issues arose while setting this all up, which are best summed up by
 the statement that a lot of the technology was (in 2006) still very immature.
 Trying to do this conversion any earlier would probably not have been anywhere
 A number of issues arose while setting this all up, which are best summed up by
 the statement that a lot of the technology was (in 2006) still very immature.
 Trying to do this conversion any earlier would probably not have been anywhere

-near as successful. The main problems that bother me in the XML-generated
+near as successful. The main issues that bother me in the XML-generated

 documentation are described in the penultimate section of this document.
 
 documentation are described in the penultimate section of this document.
 

-The major problems were originally in producing PostScript and PDF outputs. The
+Initially, the major problems were in producing PostScript and PDF outputs. The

 available free software for doing this was and still is (we are now in 2007)
 cumbersome and slow, and does not support certain output features that I would
 like. My response to this was, over a period of two years, to write an XML
 available free software for doing this was and still is (we are now in 2007)
 cumbersome and slow, and does not support certain output features that I would
 like. My response to this was, over a period of two years, to write an XML


@@ -85,11 +83,11 @@ processor called SDoP (Simple DocBook Processor). This program reads DocBook
 XML and writes PostScript, without using any of the heavyweight apparatus that
 is required for xmlto and fop (the previously used software).
 
 XML and writes PostScript, without using any of the heavyweight apparatus that
 is required for xmlto and fop (the previously used software).
 

-An experimental first version of SDoP will be used for the Exim 4.67
-documentation. A full release of SDoP requires further work. SDoP's output
+An experimental first version of SDoP was used for the Exim 4.67
+documentation. Subsequently SDoP was released for general use. SDoP's output

 includes features that are missing when xmlto/fop is used, and it also runs
 includes features that are missing when xmlto/fop is used, and it also runs

-about 60 times faster. The main manual can be formatted in 2 seconds instead of
-2 minutes, which makes checking and fixing mistakes much easier.
+about 60 times faster. The main manual can be formatted in 2.5 seconds instead
+of 2.5 minutes, which makes checking and fixing mistakes much easier.

 
 The Makefile that is used to build the various forms of output will, for the
 moment, support both ways of producing PostScript and PDF output, though the
 
 The Makefile that is used to build the various forms of output will, for the
 moment, support both ways of producing PostScript and PDF output, though the


@@ -107,13 +105,13 @@ run Gentoo Linux, and a lot of things have been installed as dependencies that
 I am not fully aware of. This is what I know about (version numbers are current
 at the time of writing):
 
 I am not fully aware of. This is what I know about (version numbers are current
 at the time of writing):
 

-. xfpt 0.01
+. xfpt 0.03

 
   This converts the master source file into a DocBook XML file.
 
 
   This converts the master source file into a DocBook XML file.
 

-. sdop 0.00
+. sdop 0.03

 
 

-  This is my new, still-very-alpha, DocBook-to-PostScript processor.
+  This is my new DocBook-to-PostScript processor.

 
 . ps2pdf
 
 
 . ps2pdf
 


@@ -131,25 +129,28 @@ at the time of writing):
   things that I have not figured out, to apply the DocBook XSLT stylesheets.
 
 . libxml 1.8.17
   things that I have not figured out, to apply the DocBook XSLT stylesheets.
 
 . libxml 1.8.17

-  libxml2 2.6.22
-  libxslt 1.1.15
+  libxml2 2.6.28
+  libxslt 1.1.20

 
   These are all installed on my box; I do not know which of libxml or libxml2
   the various scripts are actually using.
 
 
   These are all installed on my box; I do not know which of libxml or libxml2
   the various scripts are actually using.
 

-. xsl-stylesheets-1.70.1
+. xsl-stylesheets-<version>

 
   These are the standard DocBook XSL stylesheets.
 
 
   These are the standard DocBook XSL stylesheets.
 

-. fop 0.20.5
+  The documents use http://docbook.sourceforge.net/release/xsl/current/ which
+  should be mapped to an appropriate local path via the system catalogs.
+
+. fop 0.93

 
   FOP is a processor for "formatted objects". It is written in Java. The fop
   command is a shell script that drives it. It required only if you do not
   want to use SDoP and ps2pdf to generate PostScript and PDF output.
 
 
   FOP is a processor for "formatted objects". It is written in Java. The fop
   command is a shell script that drives it. It required only if you do not
   want to use SDoP and ps2pdf to generate PostScript and PDF output.
 

-. w3m 0.5.1
+. w3m 0.5.2

 
 

-  This is a text-oriented web brower. It is used to produce the Ascii form of
+  This is a text-oriented web brower. It is used to produce the ASCII form of

   the Exim documentation (spec.txt) from a specially-created HTML format. It
   seems to do a better job than lynx.
 
   the Exim documentation (spec.txt) from a specially-created HTML format. It
   seems to do a better job than lynx.
 


@@ -166,7 +167,7 @@ at the time of writing):
 
 . makeinfo 4.8
 
 
 . makeinfo 4.8
 

-  This is used to make a set of "info" files from a Texinfo file.
+  This is used to make an "info" file from a Texinfo file.

 
 In addition, there are a number of locally written Perl scripts. These are
 described below.
 
 In addition, there are a number of locally written Perl scripts. These are
 described below.


@@ -218,8 +219,8 @@ DOCBOOK PROCESSING
 Processing a .xml file into the five different output formats is not entirely
 straightforward. For a start, the same XML is not suitable for all the
 different output styles. When the final output is in a text format (.txt,
 Processing a .xml file into the five different output formats is not entirely
 straightforward. For a start, the same XML is not suitable for all the
 different output styles. When the final output is in a text format (.txt,

-.texinfo) for instance, all non-Ascii characters in the input must be converted
-to Ascii transliterations because the current processing tools do not do this
+.texinfo) for instance, all non-ASCII characters in the input must be converted
+to ASCII transliterations because the current processing tools do not do this

 correctly automatically.
 
 In order to cope with these issues in a flexible way, a Perl script called
 correctly automatically.
 
 In order to cope with these issues in a flexible way, a Perl script called


@@ -234,6 +235,39 @@ comments in these style files to explain what changes I have made. Some of the
 changes are quite significant.
 
 
 changes are quite significant.
 
 

+XSL INCLUDES
+
+References to XSL paths should use the public URLs, such as:
+  http://docbook.sourceforge.net/release/xsl/current/xhtml/docbook.xsl
+If this fails to work for you, then there is a problem with your system
+catalogs.  As a work-around, you can adjust the OS-Fixups script and then:
+$ make os-fixup
+
+As an example of how this should normally work, on a FreeBSD system the
+resolution goes to /usr/local/share/xml/catalog which contains a directive:
+  <nextCatalog catalog="/usr/local/share/xml/catalog.ports" />
+to pull in the file automatically maintained by the Ports system.  That file
+will contain:
+  <delegateSystem
+   systemIdStartString="http://docbook.sourceforge.net/release/xsl/"
+   catalog="file:///usr/local/share/xsl/docbook/catalog" />
+  <delegateURI
+   uriStartString="http://docbook.sourceforge.net/release/xsl/"
+   catalog="file:///usr/local/share/xsl/docbook/catalog" />
+and that catalog file contains:
+  <rewriteSystem
+   systemIdStartString="http://docbook.sourceforge.net/release/xsl/current"
+   rewritePrefix="file:///usr/local/share/xsl/docbook" />
+  <rewriteURI
+   uriStartString="http://docbook.sourceforge.net/release/xsl/current"
+    rewritePrefix="file:///usr/local/share/xsl/docbook" />
+and the full path is thus eventually arrived at.
+
+See also the tools:
+  xmlcatalog(1) from libxml2
+  xmlcatmgr(1) for a lightweight tool written for the NetBSD Packages system.
+
+

 THE PRE-XML SCRIPT
 
 The Pre-xml script copies a .xml file, making certain changes according to the
 THE PRE-XML SCRIPT
 
 The Pre-xml script copies a .xml file, making certain changes according to the


@@ -241,7 +275,7 @@ options it is given. The currently available options are as follows:
 
 -ascii
 
 
 -ascii
 

-  This option is used for Ascii output formats. It makes the following
+  This option is used for ASCII output formats. It makes the following

   character replacements:
 
     &#x2019;  =>  '         apostrophe
   character replacements:
 
     &#x2019;  =>  '         apostrophe


@@ -252,14 +286,14 @@ options it is given. The currently available options are as follows:
     &ndash;   =>  -         en dash
 
   The apostrophe is specified numerically because that is what xfpt generates
     &ndash;   =>  -         en dash
 
   The apostrophe is specified numerically because that is what xfpt generates

-  from an Ascii single quote character. Non-Ascii characters that are not in
+  from an ASCII single quote character. Non-ASCII characters that are not in

   this list should not be used without thinking about how they might be
   this list should not be used without thinking about how they might be

-  converted for the Ascii formats.
+  converted for the ASCII formats.

 
   In addition to the character replacements, this option causes quotes to be
   put round <literal> text items, and <quote> and </quote> to be replaced by
 
   In addition to the character replacements, this option causes quotes to be
   put round <literal> text items, and <quote> and </quote> to be replaced by

-  Ascii quote marks. You would think the stylesheet would cope with the latter,
-  but it seems to generate non-Ascii characters that w3m then turns into
+  ASCII quote marks. You would think the stylesheet would cope with the latter,
+  but it seems to generate non-ASCII characters that w3m then turns into

   question marks.
 
 -bookinfo
   question marks.
 
 -bookinfo


@@ -286,17 +320,18 @@ options it is given. The currently available options are as follows:
 -noindex
 
   Remove the XML to generate a Concept Index and an Options index. The source
 -noindex
 
   Remove the XML to generate a Concept Index and an Options index. The source

-  document has two types of index entry, for a concept and an options index.
-  However, no index is required for the .txt and .texinfo outputs.
+  document has three types of index entry, for variables, options, and concept
+  indexes. However, no index is required for the .txt and .texinfo outputs.

 
 -oneindex
 
 
 -oneindex
 

-  Remove the XML to generate a Concept and an Options Index, and add XML to
-  generate a single index. The only output processors that support multiple
-  indexes are SDoP and the processor that produces "formatted objects" for
-  PostScript and PDF output for fop. The HTML processor ignores the XML
-  settings for multiple indexes and just makes one unified index. Specifying
-  two indexes gets you two copies of the same index, so this has to be changed.
+  Remove the XML to generate separate variables, options, and concept indexes,
+  and add XML to generate a single index. The only output processors that
+  support multiple indexes are SDoP and the processor that produces "formatted
+  objects" for PostScript and PDF output for fop. The HTML processor ignores
+  the XML settings for multiple indexes and just makes one unified index.
+  Specifying three indexes gets you three copies of the same index, so this has
+  to be changed.

 
 -optbreak
 
 
 -optbreak
 


@@ -479,7 +514,7 @@ so the logic is somewhat different.
 CREATING TEXT FILES
 
 This happens in four stages. The Pre-xml script is called with the -ascii,
 CREATING TEXT FILES
 
 This happens in four stages. The Pre-xml script is called with the -ascii,

--optbreak, and -noindex options to convert the input to Ascii characters,
+-optbreak, and -noindex options to convert the input to ASCII characters,

 insert line break points, and disable the production of an index. Then the
 xmlto command converts the XML to a single HTML document, using these
 stylesheets:
 insert line break points, and disable the production of an index. Then the
 xmlto command converts the XML to a single HTML document, using these
 stylesheets:


@@ -494,7 +529,7 @@ symbol is output as "(c)" rather than the Unicode character. This is necessary
 because the stylesheet itself generates a copyright symbol as part of the
 document title; the character is not in the original input.
 
 because the stylesheet itself generates a copyright symbol as part of the
 document title; the character is not in the original input.
 

-The w3m command is used with the -dump option to turn the HTML file into Ascii
+The w3m command is used with the -dump option to turn the HTML file into ASCII

 text, but this contains multiple sequences of blank lines that make it look
 awkward. Furthermore, chapter and section titles do not stand out very well. A
 local Perl script called Tidytxt is used to post-process the output. First, it
 text, but this contains multiple sequences of blank lines that make it look
 awkward. Furthermore, chapter and section titles do not stand out very well. A
 local Perl script called Tidytxt is used to post-process the output. First, it


@@ -504,6 +539,14 @@ preceded by an extra two blank lines and a line of equals characters. An extra
 newline is inserted before each section heading, and they are underlined with
 hyphens.
 
 newline is inserted before each section heading, and they are underlined with
 hyphens.
 

+The output of xmlto also contains non-ASCII Unicode characters that w3m passes
+through. Fortunately, they are few, and Tidytxt cleans them up as well. Some
+headings use "box drawing" characters in the range U+2500 to U+253F which are
+translated into -+| as appropriate, and U+00A0 (hard space) and U+25CF (bullet)
+are translated into plain spaces and asterisks. (It might be possible to do all
+this in the same way as I dealt with copyright - see above - but adding a few
+lines of Perl to an existing script was a lot easier.)
+

 
 CREATING INFO FILES
 
 
 CREATING INFO FILES
 


@@ -516,7 +559,7 @@ makeinfo command. These have to be changed to "cindex" and "findex"
 respectively in the final .texinfo file. Furthermore, the main menu lacks a
 pointer to the index, and indeed the index node itself is missing. These
 problems are fixed by running the file through a script called TidyInfo.
 respectively in the final .texinfo file. Furthermore, the main menu lacks a
 pointer to the index, and indeed the index node itself is missing. These
 problems are fixed by running the file through a script called TidyInfo.

-Finally, a call of makeinfo creates a set of .info files.
+Finally, a call of makeinfo creates a .info file.

 
 There is one apparently unconfigurable feature of docbook2texi: it does not
 seem possible to give it a file name for its output. It chooses a name based on
 
 There is one apparently unconfigurable feature of docbook2texi: it does not
 seem possible to give it a file name for its output. It chooses a name based on


@@ -566,16 +609,16 @@ using SDoP instead of xmlto/fop to produce PostScript and PDF output.
      that is referenced, instead of to the point in the section where the index
      marker was set.
 
      that is referenced, instead of to the point in the section where the index
      marker was set.
 

-(4)  The HTML output supports only a single index, so the concept and options
-     index entries have to be merged.
+(4)  The HTML output supports only a single index, so the variable, options,
+     and concept index entries have to be merged.

 
 (5)  The index for the PostScript/PDF output created by xmlto/fop does not
      merge identical page numbers, which makes some entries look ugly. This is
      not a problem when SDoP is used.
 
 
 (5)  The index for the PostScript/PDF output created by xmlto/fop does not
      merge identical page numbers, which makes some entries look ugly. This is
      not a problem when SDoP is used.
 

-(6)  None of the indexes (PostScript/PDF and HTML) make use of textual
-     markup; the text is all roman, without any italic or boldface. For
-     PostScript/PDF, this is not a problem when SDoP is used.
+(6)  The HTML index and the PostScript/PDF indexes, when made with xmlto/fop,
+     make no use of textual markup; the text is all roman, without any italic
+     or boldface. For PostScript/PDF, this is not a problem when SDoP is used.

 
 (7)  I turned off hyphenation in the PostScript/PDF output produced by
      xmlto/fop, because it was being done so badly. Needless to say, I made
 
 (7)  I turned off hyphenation in the PostScript/PDF output produced by
      xmlto/fop, because it was being done so badly. Needless to say, I made


@@ -588,7 +631,8 @@ using SDoP instead of xmlto/fop to produce PostScript and PDF output.
          hyphenations, often for several lines in succession.
 
      (b) It uses an algorithmic form of hyphenation that doesn't always produce
          hyphenations, often for several lines in succession.
 
      (b) It uses an algorithmic form of hyphenation that doesn't always produce

-         acceptable word breaks. (I prefer to use a hyphenation dictionary.)
+         acceptable word breaks. (I prefer to use a hyphenation dictionary,
+         which is what SDoP does.)

 
 (8)  The PostScript/PDF output produced by xmlto/fop is badly paginated:
 
 
 (8)  The PostScript/PDF output produced by xmlto/fop is badly paginated:
 


@@ -662,5 +706,6 @@ spec.xfpt                      xfpt source of the specification document
 x2man                          Script to make the Exim man page from the XML
 
 
 x2man                          Script to make the Exim man page from the XML
 
 

-Philip Hazel
-Last updated: 27 March 2007
+(Originally, and for the most part: Philip Hazel)
+The Exim Maintainers
+Last updated: 5 July 2010