Rename dns_use_dnssec to dns_dnssec_ok.

[exim.git] / doc / doc-docbook / HowItWorks.txt

diff --git a/doc/doc-docbook/HowItWorks.txt b/doc/doc-docbook/HowItWorks.txt
index 228709273318d5b360cba9b0414fe6cbdde34037..c5e328a09408245f464bd531e29a211ac22969f4 100644 (file)
--- a/doc/doc-docbook/HowItWorks.txt
+++ b/doc/doc-docbook/HowItWorks.txt
@@ -1,5 +1,3 @@
-$Cambridge: exim/doc/doc-docbook/HowItWorks.txt,v 1.3 2006/02/01 11:01:01 ph10 Exp $
-

 CREATING THE EXIM DOCUMENTATION
 
 "You are lost in a maze of twisty little scripts."
 CREATING THE EXIM DOCUMENTATION
 
 "You are lost in a maze of twisty little scripts."


@@ -21,7 +19,7 @@ bought the Adobe distiller software.)
 
 A demand for a version in "info" format led me to write a Perl script that
 converted the SGCAL input into a Texinfo file. Because of the somewhat
 
 A demand for a version in "info" format led me to write a Perl script that
 converted the SGCAL input into a Texinfo file. Because of the somewhat

-restrictive requirements of Texinfo, this script has always needed a lot of
+restrictive requirements of Texinfo, this script always needed a lot of

 maintenance, and was never totally satisfactory.
 
 The HTML version of the documentation was originally produced from the Texinfo
 maintenance, and was never totally satisfactory.
 
 The HTML version of the documentation was originally produced from the Texinfo


@@ -72,10 +70,28 @@ Equally, if better ways of processing the XML become available, they can be
 adopted without affecting the source maintenance.
 
 A number of issues arose while setting this all up, which are best summed up by
 adopted without affecting the source maintenance.
 
 A number of issues arose while setting this all up, which are best summed up by

-the statement that a lot of the technology is (in 2006) still very immature. It
-is probable that trying to do this conversion any earlier would not have been
-anywhere near as successful. The main problems that still bother me are
-described in the penultimate section of this document.
+the statement that a lot of the technology was (in 2006) still very immature.
+Trying to do this conversion any earlier would probably not have been anywhere
+near as successful. The main issues that bother me in the XML-generated
+documentation are described in the penultimate section of this document.
+
+Initially, the major problems were in producing PostScript and PDF outputs. The
+available free software for doing this was and still is (we are now in 2007)
+cumbersome and slow, and does not support certain output features that I would
+like. My response to this was, over a period of two years, to write an XML
+processor called SDoP (Simple DocBook Processor). This program reads DocBook
+XML and writes PostScript, without using any of the heavyweight apparatus that
+is required for xmlto and fop (the previously used software).
+
+An experimental first version of SDoP was used for the Exim 4.67
+documentation. Subsequently SDoP was released for general use. SDoP's output
+includes features that are missing when xmlto/fop is used, and it also runs
+about 60 times faster. The main manual can be formatted in 2.5 seconds instead
+of 2.5 minutes, which makes checking and fixing mistakes much easier.
+
+The Makefile that is used to build the various forms of output will, for the
+moment, support both ways of producing PostScript and PDF output, though the
+default is now to use SDoP.

 
 The following sections describe the processes by which the xfpt files are
 transformed into the final output documents. In practice, the details are coded
 
 The following sections describe the processes by which the xfpt files are
 transformed into the final output documents. In practice, the details are coded


@@ -89,37 +105,52 @@ run Gentoo Linux, and a lot of things have been installed as dependencies that
 I am not fully aware of. This is what I know about (version numbers are current
 at the time of writing):
 
 I am not fully aware of. This is what I know about (version numbers are current
 at the time of writing):
 

-. xfpt 0.00
+. xfpt 0.03

 
   This converts the master source file into a DocBook XML file.
 
 
   This converts the master source file into a DocBook XML file.
 

+. sdop 0.03
+
+  This is my new DocBook-to-PostScript processor.
+
+. ps2pdf
+
+  This is a wrapper script that is part of the GhostScript distribution. It
+  converts a PostScript file into a PDF file. It is used to process the output
+  from SDoP. It is not required when xmlto/fop is being used to generate PDF
+  output.
+

 . xmlto 0.0.18
 
   This is a shell script that drives various XML processors. It is used to
 . xmlto 0.0.18
 
   This is a shell script that drives various XML processors. It is used to

-  produce "formatted objects" for PostScript and PDF output, and to produce
-  HTML output. It uses xsltproc, libxml, libxslt, libexslt, and possibly other
+  produce "formatted objects" when PostScript and PDF output is being generated
+  using fop (the old way) rather than SDoP. It is always used to produce HTML
+  output. It uses xsltproc, libxml, libxslt, libexslt, and possibly other

   things that I have not figured out, to apply the DocBook XSLT stylesheets.
 
 . libxml 1.8.17
   things that I have not figured out, to apply the DocBook XSLT stylesheets.
 
 . libxml 1.8.17

-  libxml2 2.6.22
-  libxslt 1.1.15
+  libxml2 2.6.28
+  libxslt 1.1.20

 
   These are all installed on my box; I do not know which of libxml or libxml2
   the various scripts are actually using.
 
 
   These are all installed on my box; I do not know which of libxml or libxml2
   the various scripts are actually using.
 

-. xsl-stylesheets-1.68.1
+. xsl-stylesheets-<version>

 
   These are the standard DocBook XSL stylesheets.
 
 
   These are the standard DocBook XSL stylesheets.
 

-. fop 0.20.5
+  The documents use http://docbook.sourceforge.net/release/xsl/current/ which
+  should be mapped to an appropriate local path via the system catalogs.
+
+. fop 0.93

 
   FOP is a processor for "formatted objects". It is written in Java. The fop
 
   FOP is a processor for "formatted objects". It is written in Java. The fop

-  command is a shell script that drives it. It is used to generate PostScript
-  and PDF output.
+  command is a shell script that drives it. It required only if you do not
+  want to use SDoP and ps2pdf to generate PostScript and PDF output.

 
 

-. w3m 0.5.1
+. w3m 0.5.2

 
 

-  This is a text-oriented web brower. It is used to produce the Ascii form of
+  This is a text-oriented web brower. It is used to produce the ASCII form of

   the Exim documentation (spec.txt) from a specially-created HTML format. It
   seems to do a better job than lynx.
 
   the Exim documentation (spec.txt) from a specially-created HTML format. It
   seems to do a better job than lynx.
 


@@ -136,7 +167,7 @@ at the time of writing):
 
 . makeinfo 4.8
 
 
 . makeinfo 4.8
 

-  This is used to make a set of "info" files from a Texinfo file.
+  This is used to make an "info" file from a Texinfo file.

 
 In addition, there are a number of locally written Perl scripts. These are
 described below.
 
 In addition, there are a number of locally written Perl scripts. These are
 described below.


@@ -157,9 +188,15 @@ this process, including the master DocBook source, called spec.xml. Of course,
 the usual features of "make" ensure that if this already exists and is
 up-to-date, it is not needlessly rebuilt.
 
 the usual features of "make" ensure that if this already exists and is
 up-to-date, it is not needlessly rebuilt.
 

+Because there are now two ways of creating the PostScript and PDF outputs,
+there are two targets for each one. For example fop-spec.ps makes PostScript
+using fop, and sdop-spec.ps makes it using SDoP. The generic targets spec.ps
+and spec.pdf now point to the SDoP versions.
+

 The "test" series of targets were created so that small tests could easily be
 run fairly quickly, because processing even the shortish XML document takes
 The "test" series of targets were created so that small tests could easily be
 run fairly quickly, because processing even the shortish XML document takes

-a bit of time, and processing the main specification takes ages.
+a bit of time, and processing the main specification takes ages -- except when
+using SDoP for PostScript and PDF.

 
 Another target is "exim.8". This runs a locally written Perl script called
 x2man, which extracts the list of command line options from the spec.xml file,
 
 Another target is "exim.8". This runs a locally written Perl script called
 x2man, which extracts the list of command line options from the spec.xml file,


@@ -182,8 +219,8 @@ DOCBOOK PROCESSING
 Processing a .xml file into the five different output formats is not entirely
 straightforward. For a start, the same XML is not suitable for all the
 different output styles. When the final output is in a text format (.txt,
 Processing a .xml file into the five different output formats is not entirely
 straightforward. For a start, the same XML is not suitable for all the
 different output styles. When the final output is in a text format (.txt,

-.texinfo) for instance, all non-Ascii characters in the input must be converted
-to Ascii transliterations because the current processing tools do not do this
+.texinfo) for instance, all non-ASCII characters in the input must be converted
+to ASCII transliterations because the current processing tools do not do this

 correctly automatically.
 
 In order to cope with these issues in a flexible way, a Perl script called
 correctly automatically.
 
 In order to cope with these issues in a flexible way, a Perl script called


@@ -191,11 +228,44 @@ Pre-xml was written. This is used to preprocess the .xml files before they are
 handed to the main processors. Adding one more tool onto the front of the
 processing chain does at least seem to be in the spirit of XML processing.
 
 handed to the main processors. Adding one more tool onto the front of the
 processing chain does at least seem to be in the spirit of XML processing.
 

-The XML processors themselves make use of style files, which can be overridden
-by local versions. There is one that applies to all styles, called MyStyle.xsl,
-and others for the different output formats. I have included comments in these
-style files to explain what changes I have made. Some of the changes are quite
-significant.
+The XML processors other than SDoP make use of style files, which can be
+overridden by local versions. There is one that applies to all styles, called
+MyStyle.xsl, and others for the different output formats. I have included
+comments in these style files to explain what changes I have made. Some of the
+changes are quite significant.
+
+
+XSL INCLUDES
+
+References to XSL paths should use the public URLs, such as:
+  http://docbook.sourceforge.net/release/xsl/current/xhtml/docbook.xsl
+If this fails to work for you, then there is a problem with your system
+catalogs.  As a work-around, you can adjust the OS-Fixups script and then:
+$ make os-fixup
+
+As an example of how this should normally work, on a FreeBSD system the
+resolution goes to /usr/local/share/xml/catalog which contains a directive:
+  <nextCatalog catalog="/usr/local/share/xml/catalog.ports" />
+to pull in the file automatically maintained by the Ports system.  That file
+will contain:
+  <delegateSystem
+   systemIdStartString="http://docbook.sourceforge.net/release/xsl/"
+   catalog="file:///usr/local/share/xsl/docbook/catalog" />
+  <delegateURI
+   uriStartString="http://docbook.sourceforge.net/release/xsl/"
+   catalog="file:///usr/local/share/xsl/docbook/catalog" />
+and that catalog file contains:
+  <rewriteSystem
+   systemIdStartString="http://docbook.sourceforge.net/release/xsl/current"
+   rewritePrefix="file:///usr/local/share/xsl/docbook" />
+  <rewriteURI
+   uriStartString="http://docbook.sourceforge.net/release/xsl/current"
+    rewritePrefix="file:///usr/local/share/xsl/docbook" />
+and the full path is thus eventually arrived at.
+
+See also the tools:
+  xmlcatalog(1) from libxml2
+  xmlcatmgr(1) for a lightweight tool written for the NetBSD Packages system.

 
 
 THE PRE-XML SCRIPT
 
 
 THE PRE-XML SCRIPT


@@ -205,7 +275,7 @@ options it is given. The currently available options are as follows:
 
 -ascii
 
 
 -ascii
 

-  This option is used for Ascii output formats. It makes the following
+  This option is used for ASCII output formats. It makes the following

   character replacements:
 
     &#x2019;  =>  '         apostrophe
   character replacements:
 
     &#x2019;  =>  '         apostrophe


@@ -216,14 +286,14 @@ options it is given. The currently available options are as follows:
     &ndash;   =>  -         en dash
 
   The apostrophe is specified numerically because that is what xfpt generates
     &ndash;   =>  -         en dash
 
   The apostrophe is specified numerically because that is what xfpt generates

-  from an Ascii single quote character. Non-Ascii characters that are not in
+  from an ASCII single quote character. Non-ASCII characters that are not in

   this list should not be used without thinking about how they might be
   this list should not be used without thinking about how they might be

-  converted for the Ascii formats.
+  converted for the ASCII formats.

 
   In addition to the character replacements, this option causes quotes to be
   put round <literal> text items, and <quote> and </quote> to be replaced by
 
   In addition to the character replacements, this option causes quotes to be
   put round <literal> text items, and <quote> and </quote> to be replaced by

-  Ascii quote marks. You would think the stylesheet would cope with the latter,
-  but it seems to generate non-Ascii characters that w3m then turns into
+  ASCII quote marks. You would think the stylesheet would cope with the latter,
+  but it seems to generate non-ASCII characters that w3m then turns into

   question marks.
 
 -bookinfo
   question marks.
 
 -bookinfo


@@ -239,24 +309,29 @@ options it is given. The currently available options are as follows:
 
   The use of ligatures would be nice for the PostScript and PDF formats. Sadly,
   it turns out that fop cannot at present handle the FB01 character correctly.
 
   The use of ligatures would be nice for the PostScript and PDF formats. Sadly,
   it turns out that fop cannot at present handle the FB01 character correctly.

-  The only format that does so is the HTML format, but when I used this in the
-  test version, people complained that it made searching for words difficult.
-  So at the moment, this option is not used. :-(
+  Happily this problem is now avoided when SDoP is used to generate PostScript
+  (and thence PDF) because SDoP automatically uses an "fi" ligature for
+  non-fixed-width fonts.
+
+  The only xmlto format that handles FB01 is the HTML format, but when I used
+  this in the test version, people complained that it made searching for words
+  difficult. So this option is in practice not used at all.

 
 -noindex
 
   Remove the XML to generate a Concept Index and an Options index. The source
 
 -noindex
 
   Remove the XML to generate a Concept Index and an Options index. The source

-  document has two types of index entry, for a concept and an options index.
-  However, no index is required for the .txt and .texinfo outputs.
+  document has three types of index entry, for variables, options, and concept
+  indexes. However, no index is required for the .txt and .texinfo outputs.

 
 -oneindex
 
 
 -oneindex
 

-  Remove the XML to generate a Concept and an Options Index, and add XML to
-  generate a single index. The only output processor that supports multiple
-  indexes is the processor that produces "formatted objects" for PostScript and
-  PDF output. The HTML processor ignores the XML settings for multiple indexes
-  and just makes one unified index. Specifying two indexes gets you two copies
-  of the same index, so this has to be changed.
+  Remove the XML to generate separate variables, options, and concept indexes,
+  and add XML to generate a single index. The only output processors that
+  support multiple indexes are SDoP and the processor that produces "formatted
+  objects" for PostScript and PDF output for fop. The HTML processor ignores
+  the XML settings for multiple indexes and just makes one unified index.
+  Specifying three indexes gets you three copies of the same index, so this has
+  to be changed.

 
 -optbreak
 
 
 -optbreak
 


@@ -270,11 +345,31 @@ options it is given. The currently available options are as follows:
 
 CREATING POSTSCRIPT AND PDF
 
 
 CREATING POSTSCRIPT AND PDF
 

-These two output formats are created in three stages, with an additional fourth
-stage for PDF. First, the XML is pre-processed by the Pre-xml script. For the
-filter document, the <bookinfo> element is removed so that no title page is
-generated. For the main specification, the only change is to insert line
-breakpoints via -optbreak.
+These two output formats are created either by using my new SDoP program to
+produce PostScript which can then be run through ps2pdf to make a PDF, or by
+using xmlto and fop in the old way.
+
+
+USING SDOP TO CREATE POSTSCRIPT AND PDF
+
+PostScript output is created in two stages. First, the XML is pre-processed by
+the Pre-xml script. For the filter document, the <bookinfo> element is removed
+so that no title page is generated. For the main specification, the only change
+is to insert line breakpoints via -optbreak.
+
+The SDoP program is then used to create PostScript output directly from the XML
+input. Then the ps2pdf command is used to generated a PDF from the PostScript.
+There are no external stylesheets that are used by SDoP. Any variations to the
+default format are specified inline using "processing instructions".
+
+
+USING XMLTO AND FOP TO CREATE POSTSCRIPT AND PDF
+
+This is the original way of creating PostScript and PDF output. The processing
+happens in three stages, with an additional fourth stage for PDF. First, the
+XML is pre-processed by the Pre-xml script. For the filter document, the
+<bookinfo> element is removed so that no title page is generated. For the main
+specification, the only change is to insert line breakpoints via -optbreak.

 
 Second, the xmlto command is used to produce a "formatted objects" (.fo) file.
 This process uses the following stylesheets:
 
 Second, the xmlto command is used to produce a "formatted objects" (.fo) file.
 This process uses the following stylesheets:


@@ -328,9 +423,9 @@ The last one is particularly meaningless gobbledegook. Some of the errors and
 warnings are repeated many times. Nevertheless, it does eventually produce
 usable output, though I have a number of issues with it (see a later section of
 this document). Maybe one day there will be a new release of fop that does
 warnings are repeated many times. Nevertheless, it does eventually produce
 usable output, though I have a number of issues with it (see a later section of
 this document). Maybe one day there will be a new release of fop that does

-better (there are now signs - February 2006 - that this may be happening).
-Maybe there will be some other means of producing PostScript and PDF from
-DocBook XML. Maybe porcine aeronautics will really happen.
+better. In the meantime, I have written my own program for making PostScript
+output -- see the previous section -- because the problems with xmlto/fop were
+sufficiently annoying.

 
 The PDF file that is produced by this process has one problem: the pages, as
 shown by acroread in its thumbnail display, are numbered sequentially from one
 
 The PDF file that is produced by this process has one problem: the pages, as
 shown by acroread in its thumbnail display, are numbered sequentially from one


@@ -339,18 +434,22 @@ of the document, which makes finding a page from the index awkward. There is a
 facility in the PDF format to give pages appropriate "labels", but I cannot
 find a way of persuading fop to generate these. Fortunately, it is possibly to
 fix up the PDF to add page labels. I wrote a script called PageLabelPDF which
 facility in the PDF format to give pages appropriate "labels", but I cannot
 find a way of persuading fop to generate these. Fortunately, it is possibly to
 fix up the PDF to add page labels. I wrote a script called PageLabelPDF which

-does this. They are shown correctly by acroread, but not by GhostScript (gv).
+does this. They are shown correctly by acroread and xpdf, but not by
+GhostScript (gv).

 
 
 THE PAGELABELPDF SCRIPT
 
 
 
 THE PAGELABELPDF SCRIPT
 

-This script reads the standard input and writes the standard output. It
-searches for the PDF object that sets data in its "Catalog", and adds
-appropriate information about page labels. The number of front-matter pages
-(those before chapter 1) is hard-wired into this script as 12 because I could
-not find a way of determining it automatically. As the current table of
-contents finishes near the top of the 11th page, there is plenty of room for
-expansion, so it is unlikely to be a problem.
+This script reads the standard input and writes the standard output. It is used
+to "tidy up" the PDF output that is produced by fop. It is not needed when
+PDF output is generated from SDoP's output using ps2pdf.
+
+The PageLabelPDF script searches for the PDF object that sets data in its
+"Catalog", and adds appropriate information about page labels. The number of
+front-matter pages (those before chapter 1) is hard-wired into this script as
+12 because I could not find a way of determining it automatically. As the
+current table of contents finishes near the top of the 11th page, there is
+plenty of room for expansion, so it is unlikely to be a problem.

 
 Having added data to the PDF file, the script then finds the xref table at the
 end of the file, and adjusts its entries to allow for the added text. This
 
 Having added data to the PDF file, the script then finds the xref table at the
 end of the file, and adjusts its entries to allow for the added text. This


@@ -401,7 +500,7 @@ of </td> it inserts &nbsp; so that the table's cell is a little bit wider than
 the text itself.
 
 The TidyHTML-spec script also takes the opportunity to postprocess the
 the text itself.
 
 The TidyHTML-spec script also takes the opportunity to postprocess the

-spec.html/ix01.html file, which contains the document index. Again, the index
+spec_html/ix01.html file, which contains the document index. Again, the index

 is generated as one single line, so it splits it up. Then it creates a list of
 letters at the top of the index and hyperlinks them both ways from the
 different letter portions of the index.
 is generated as one single line, so it splits it up. Then it creates a list of
 letters at the top of the index and hyperlinks them both ways from the
 different letter portions of the index.


@@ -415,7 +514,7 @@ so the logic is somewhat different.
 CREATING TEXT FILES
 
 This happens in four stages. The Pre-xml script is called with the -ascii,
 CREATING TEXT FILES
 
 This happens in four stages. The Pre-xml script is called with the -ascii,

--optbreak, and -noindex options to convert the input to Ascii characters,
+-optbreak, and -noindex options to convert the input to ASCII characters,

 insert line break points, and disable the production of an index. Then the
 xmlto command converts the XML to a single HTML document, using these
 stylesheets:
 insert line break points, and disable the production of an index. Then the
 xmlto command converts the XML to a single HTML document, using these
 stylesheets:


@@ -430,7 +529,7 @@ symbol is output as "(c)" rather than the Unicode character. This is necessary
 because the stylesheet itself generates a copyright symbol as part of the
 document title; the character is not in the original input.
 
 because the stylesheet itself generates a copyright symbol as part of the
 document title; the character is not in the original input.
 

-The w3m command is used with the -dump option to turn the HTML file into Ascii
+The w3m command is used with the -dump option to turn the HTML file into ASCII

 text, but this contains multiple sequences of blank lines that make it look
 awkward. Furthermore, chapter and section titles do not stand out very well. A
 local Perl script called Tidytxt is used to post-process the output. First, it
 text, but this contains multiple sequences of blank lines that make it look
 awkward. Furthermore, chapter and section titles do not stand out very well. A
 local Perl script called Tidytxt is used to post-process the output. First, it


@@ -440,6 +539,14 @@ preceded by an extra two blank lines and a line of equals characters. An extra
 newline is inserted before each section heading, and they are underlined with
 hyphens.
 
 newline is inserted before each section heading, and they are underlined with
 hyphens.
 

+The output of xmlto also contains non-ASCII Unicode characters that w3m passes
+through. Fortunately, they are few, and Tidytxt cleans them up as well. Some
+headings use "box drawing" characters in the range U+2500 to U+253F which are
+translated into -+| as appropriate, and U+00A0 (hard space) and U+25CF (bullet)
+are translated into plain spaces and asterisks. (It might be possible to do all
+this in the same way as I dealt with copyright - see above - but adding a few
+lines of Perl to an existing script was a lot easier.)
+

 
 CREATING INFO FILES
 
 
 CREATING INFO FILES
 


@@ -448,16 +555,18 @@ characters in the source are transliterated, and the <index> elements are
 removed. The docbook2texi script is then called to convert the XML file into a
 Texinfo file. However, this is not quite enough. The converted file ends up
 with "conceptindex" and "optionindex" items, which are not recognized by the
 removed. The docbook2texi script is then called to convert the XML file into a
 Texinfo file. However, this is not quite enough. The converted file ends up
 with "conceptindex" and "optionindex" items, which are not recognized by the

-makeinfo command. An in-line call to Perl in the Makefile changes these to
-"cindex" and "findex" respectively in the final .texinfo file. Finally, a call
-of makeinfo creates a set of .info files.
+makeinfo command. These have to be changed to "cindex" and "findex"
+respectively in the final .texinfo file. Furthermore, the main menu lacks a
+pointer to the index, and indeed the index node itself is missing. These
+problems are fixed by running the file through a script called TidyInfo.
+Finally, a call of makeinfo creates a .info file.

 
 There is one apparently unconfigurable feature of docbook2texi: it does not
 seem possible to give it a file name for its output. It chooses a name based on
 the title of the document. Thus, the main specification ends up in a file
 called the_exim_mta.texi and the filter document in exim_filtering.texi. These
 files are removed after their contents have been copied and modified by the
 
 There is one apparently unconfigurable feature of docbook2texi: it does not
 seem possible to give it a file name for its output. It chooses a name based on
 the title of the document. Thus, the main specification ends up in a file
 called the_exim_mta.texi and the filter document in exim_filtering.texi. These
 files are removed after their contents have been copied and modified by the

-inline Perl call, which makes a .texinfo file.
+TidyInfo script, which writes to a .texinfo file.

 
 
 CREATING THE MAN PAGE
 
 
 CREATING THE MAN PAGE


@@ -478,7 +587,8 @@ UNRESOLVED PROBLEMS
 
 There are a number of unresolved problems with producing the Exim documentation
 in the manner described above. I will describe them here in the hope that in
 
 There are a number of unresolved problems with producing the Exim documentation
 in the manner described above. I will describe them here in the hope that in

-future some way round them can be found.
+future some way round them can be found. Some of the problems are solved by
+using SDoP instead of xmlto/fop to produce PostScript and PDF output.

 
 (1)  When a whole chain of tools is processing a file, an error somewhere
      in the middle is often very hard to debug. For instance, an error in the
 
 (1)  When a whole chain of tools is processing a file, an error somewhere
      in the middle is often very hard to debug. For instance, an error in the


@@ -488,27 +598,31 @@ future some way round them can be found.
      targets was to help in checking out these kinds of problem.
 
 (2)  There is a mechanism in XML for marking parts of the document as
      targets was to help in checking out these kinds of problem.
 
 (2)  There is a mechanism in XML for marking parts of the document as

-     "revised", and I have arranged for xfpt markup to use it. However, at the
-     moment, the only output format that pays attention to this is the HTML
-     output, which sets a green background. There are therefore no revision
-     marks (change bars) in the PostScript, PDF, or text output formats as
-     there used to be. (There never were for Texinfo.)
+     "revised", and I have arranged for xfpt markup to use it. However, the
+     only xmlto output format that pays attention to this is the HTML output,
+     which sets a green background. If xmlto/fop is used to generate PostScript
+     and PDF, there are no revision marks (change bars). This problem
+     is not present when SDoP is used. However, the text and Texinfo output
+     format lack revision indications.

 
 (3)  The index entries in the HTML format take you to the top of the section
      that is referenced, instead of to the point in the section where the index
      marker was set.
 
 
 (3)  The index entries in the HTML format take you to the top of the section
      that is referenced, instead of to the point in the section where the index
      marker was set.
 

-(4)  The HTML output supports only a single index, so the concept and options
-     index entries have to be merged.
+(4)  The HTML output supports only a single index, so the variable, options,
+     and concept index entries have to be merged.

 
 

-(5)  The index for the PostScript/PDF output does not merge identical page
-     numbers, which makes some entries look ugly.
+(5)  The index for the PostScript/PDF output created by xmlto/fop does not
+     merge identical page numbers, which makes some entries look ugly. This is
+     not a problem when SDoP is used.

 
 

-(6)  None of the indexes (PostScript/PDF and HTML) make use of textual
-     markup; the text is all roman, without any italic or boldface.
+(6)  The HTML index and the PostScript/PDF indexes, when made with xmlto/fop,
+     make no use of textual markup; the text is all roman, without any italic
+     or boldface. For PostScript/PDF, this is not a problem when SDoP is used.

 
 

-(7)  I turned off hyphenation in the PostScript/PDF output, because it was
-     being done so badly.
+(7)  I turned off hyphenation in the PostScript/PDF output produced by
+     xmlto/fop, because it was being done so badly. Needless to say, I made
+     SDoP do a better job. These comments apply to xmlto/fop:

 
      (a) It seems to force hyphenation if it is at all possible, without
          regard to the "tightness" or "looseness" of the line. Decent
 
      (a) It seems to force hyphenation if it is at all possible, without
          regard to the "tightness" or "looseness" of the line. Decent


@@ -517,9 +631,10 @@ future some way round them can be found.
          hyphenations, often for several lines in succession.
 
      (b) It uses an algorithmic form of hyphenation that doesn't always produce
          hyphenations, often for several lines in succession.
 
      (b) It uses an algorithmic form of hyphenation that doesn't always produce

-         acceptable word breaks. (I prefer to use a hyphenation dictionary.)
+         acceptable word breaks. (I prefer to use a hyphenation dictionary,
+         which is what SDoP does.)

 
 

-(8)  The PostScript/PDF output is badly paginated:
+(8)  The PostScript/PDF output produced by xmlto/fop is badly paginated:

 
      (a) There seems to be no attempt to avoid "widow" and "orphan" lines on
          pages. A "widow" is the last line of a paragraph at the top of a page,
 
      (a) There seems to be no attempt to avoid "widow" and "orphan" lines on
          pages. A "widow" is the last line of a paragraph at the top of a page,


@@ -529,8 +644,12 @@ future some way round them can be found.
      (b) There seems to be no attempt to prevent section headings being placed
          last on a page, with no following text on the page.
 
      (b) There seems to be no attempt to prevent section headings being placed
          last on a page, with no following text on the page.
 

+     Neither of these problems occurs when SDoP is used to produce the
+     PostScript/PDF output.
+

 (9)  The fop processor does not support "fi" ligatures, not even if you put the
 (9)  The fop processor does not support "fi" ligatures, not even if you put the

-     appropriate Unicode character into the source by hand.
+     appropriate Unicode character into the source by hand. Again, this is not
+     a problem if SDoP is used.

 
 (10) There are no diagrams in the new documentation. This is something I hope
      to work on. The previously used Aspic command for creating line art from a
 
 (10) There are no diagrams in the new documentation. This is something I hope
      to work on. The previously used Aspic command for creating line art from a


@@ -541,18 +660,23 @@ future some way round them can be found.
 
 (11) The use of a "zero-width space" works well as a way of specifying that
      Exim option names can be split, without hyphens, over line breaks.
 
 (11) The use of a "zero-width space" works well as a way of specifying that
      Exim option names can be split, without hyphens, over line breaks.

-     However, when an option is not split, if the line is very "loose", the
-     zero-width space is expanded, along with other spaces. This is a totally
-     crazy thing to, but unfortunately it is suggested by the Unicode
-     definition of the zero-width space, which says "its presence between two
-     characters does not prevent increased letter spacing in justification".
-     It seems that the implementors of fop have understood "letter spacing"
-     also to include "word spacing". Sigh.

 
 

-The consequence of (7), (8), and (9) is that the PostScript/PDF output looks as
-if it comes from some of the very early attempts at text formatting of around
-20 years ago. We can only hope that 20 years' progress is not going to get
-lost, and that things will improve in this area.
+     However, when xmlto/fop is being used and an option is not split, if the
+     line is very "loose", the zero-width space is expanded, along with other
+     spaces. This is a totally crazy thing to, but unfortunately it is
+     suggested by the Unicode definition of the zero-width space, which says
+     "its presence between two characters does not prevent increased letter
+     spacing in justification". It seems that the implementors of fop have
+     understood "letter spacing" also to include "word spacing". Sigh.
+
+     This problem does not arise when SDoP is used.
+
+The consequence of (7), (8), and (9) is that the PostScript/PDF output as
+produced by xmlto/fop looks as if it comes from some of the very early attempts
+at text formatting of around 20 years ago. We can only hope that 20 years'
+progress is not going to get lost, and that things will improve in this area.
+My small contribution to this has been to write SDoP, which, though simple and
+"non-standard", does get some of these formatting issues right.

 
 
 LIST OF FILES
 
 
 LIST OF FILES


@@ -571,15 +695,17 @@ MyStyle.xsl                    Stylesheet for all output
 MyTitleStyle.xsl               Stylesheet for spec title page
 MyTitlepage.templates.xml      Template for creating MyTitleStyle.xsl
 Myhtml.css                     Experimental css stylesheet for HTML output
 MyTitleStyle.xsl               Stylesheet for spec title page
 MyTitlepage.templates.xml      Template for creating MyTitleStyle.xsl
 Myhtml.css                     Experimental css stylesheet for HTML output

-PageLabelPDF                   Script to postprocess PDF
+PageLabelPDF                   Script to postprocess xmlto/fop PDF output

 Pre-xml                        Script to preprocess XML
 TidyHTML-filter                Script to tidy up the filter HTML output
 TidyHTML-spec                  Script to tidy up the spec HTML output
 Pre-xml                        Script to preprocess XML
 TidyHTML-filter                Script to tidy up the filter HTML output
 TidyHTML-spec                  Script to tidy up the spec HTML output

+TidyInfo                       Script to sort index problems in Texinfo output

 Tidytxt                        Script to compact multiple blank lines
 filter.xfpt                    xfpt source of the filter document
 spec.xfpt                      xfpt source of the specification document
 x2man                          Script to make the Exim man page from the XML
 
 
 Tidytxt                        Script to compact multiple blank lines
 filter.xfpt                    xfpt source of the filter document
 spec.xfpt                      xfpt source of the specification document
 x2man                          Script to make the Exim man page from the XML
 
 

-Philip Hazel
-Last updated: 31 January 2006
+(Originally, and for the most part: Philip Hazel)
+The Exim Maintainers
+Last updated: 5 July 2010