UTF8/locale: document constraints on current expansions.

diff --git a/doc/doc-docbook/spec.xfpt b/doc/doc-docbook/spec.xfpt
index 81261c8f6aa3b9d4f75682fb5bbf2171768f3710..f97a430a6c74b7c2ccf747ed3fcb6a1e51960272 100644 (file)
--- a/doc/doc-docbook/spec.xfpt
+++ b/doc/doc-docbook/spec.xfpt
@@ -9637,9 +9637,10 @@ some of the braces:
 .code
 ${length_<n>:<string>}
 .endd
 .code
 ${length_<n>:<string>}
 .endd

-The result of this item is either the first <&'n'&> characters or the whole
+The result of this item is either the first <&'n'&> bytes or the whole

 of <&'string2'&>, whichever is the shorter. Do not confuse &%length%& with
 &%strlen%&, which gives the length of a string.
 of <&'string2'&>, whichever is the shorter. Do not confuse &%length%& with
 &%strlen%&, which gives the length of a string.

+All measurement is done in bytes and is not UTF-8 aware.

 
 
 .vitem "&*${listextract{*&<&'number'&>&*}&&&
 
 
 .vitem "&*${listextract{*&<&'number'&>&*}&&&


@@ -10071,6 +10072,8 @@ ${sg{1=A 4=D 3=C}{\N(\d+)=\N}{K\$1=}}
 yields &"K1=A K4=D K3=C"&. Note the use of &`\N`& to protect the contents of
 the regular expression from string expansion.
 
 yields &"K1=A K4=D K3=C"&. Note the use of &`\N`& to protect the contents of
 the regular expression from string expansion.
 

+The regular expression is compiled in 8-bit mode, working against bytes
+rather than any Unicode-aware character handling.

 
 
 .vitem &*${sort{*&<&'string'&>&*}{*&<&'comparator'&>&*}{*&<&'extractor'&>&*}}*&
 
 
 .vitem &*${sort{*&<&'string'&>&*}{*&<&'comparator'&>&*}{*&<&'extractor'&>&*}}*&


@@ -10127,11 +10130,11 @@ ${substr{3}{2}{$local_part}}
 If the starting offset is greater than the string length the result is the
 null string; if the length plus starting offset is greater than the string
 length, the result is the right-hand part of the string, starting from the
 If the starting offset is greater than the string length the result is the
 null string; if the length plus starting offset is greater than the string
 length, the result is the right-hand part of the string, starting from the

-given offset. The first character in the string has offset zero.
+given offset. The first byte (character) in the string has offset zero.

 
 The &%substr%& expansion item can take negative offset values to count
 
 The &%substr%& expansion item can take negative offset values to count

-from the right-hand end of its operand. The last character is offset -1, the
-second-last is offset -2, and so on. Thus, for example,
+from the right-hand end of its operand. The last byte (character) is offset -1,
+the second-last is offset -2, and so on. Thus, for example,

 .code
 ${substr{-5}{2}{1234567}}
 .endd
 .code
 ${substr{-5}{2}{1234567}}
 .endd


@@ -10148,7 +10151,7 @@ ${substr{-3}{2}{12}}
 yields &"1"&.
 
 When the second number is omitted from &%substr%&, the remainder of the string
 yields &"1"&.
 
 When the second number is omitted from &%substr%&, the remainder of the string

-is taken if the offset is positive. If it is negative, all characters in the
+is taken if the offset is positive. If it is negative, all bytes (characters) in the

 string preceding the offset point are taken. For example, an offset of -1 and
 no length, as in these semantically identical examples:
 .code
 string preceding the offset point are taken. For example, an offset of -1 and
 no length, as in these semantically identical examples:
 .code


@@ -10157,13 +10160,15 @@ ${substr{-1}{abcde}}
 .endd
 yields all but the last character of the string, that is, &"abcd"&.
 
 .endd
 yields all but the last character of the string, that is, &"abcd"&.
 

+All measurement is done in bytes and is not UTF-8 aware.
+

 
 
 .vitem "&*${tr{*&<&'subject'&>&*}{*&<&'characters'&>&*}&&&
         {*&<&'replacements'&>&*}}*&"
 .cindex "expansion" "character translation"
 .cindex "&%tr%& expansion item"
 
 
 .vitem "&*${tr{*&<&'subject'&>&*}{*&<&'characters'&>&*}&&&
         {*&<&'replacements'&>&*}}*&"
 .cindex "expansion" "character translation"
 .cindex "&%tr%& expansion item"

-This item does single-character translation on its subject string. The second
+This item does single-character (in bytes) translation on its subject string. The second

 argument is a list of characters to be translated in the subject string. Each
 matching character is replaced by the corresponding character from the
 replacement list. For example
 argument is a list of characters to be translated in the subject string. Each
 matching character is replaced by the corresponding character from the
 replacement list. For example


@@ -10174,6 +10179,9 @@ yields &`1b3de1`&. If there are duplicates in the second character string, the
 last occurrence is used. If the third string is shorter than the second, its
 last character is replicated. However, if it is empty, no translation takes
 place.
 last occurrence is used. If the third string is shorter than the second, its
 last character is replicated. However, if it is empty, no translation takes
 place.

+
+All character handling is done in bytes and is not UTF-8 aware.
+

 .endlist
 
 
 .endlist
 
 


@@ -10193,6 +10201,8 @@ The string is interpreted as an RFC 2822 address, as it might appear in a
 header line, and the effective address is extracted from it. If the string does
 not parse successfully, the result is empty.
 
 header line, and the effective address is extracted from it. If the string does
 not parse successfully, the result is empty.
 

+The parsing correctly handles SMTPUTF8 Unicode in the string.
+

 
 .vitem &*${addresses:*&<&'string'&>&*}*&
 .cindex "expansion" "RFC 2822 address handling"
 
 .vitem &*${addresses:*&<&'string'&>&*}*&
 .cindex "expansion" "RFC 2822 address handling"


@@ -10236,7 +10246,7 @@ It does not see the comma because it's still encoded as "=2C".  The second
 example below is passed the contents of &`$header_from:`&, meaning it gets
 de-mimed. Exim sees the decoded "," so it treats it as &*two*& email addresses.
 The third example shows that the presence of a comma is skipped when it is
 example below is passed the contents of &`$header_from:`&, meaning it gets
 de-mimed. Exim sees the decoded "," so it treats it as &*two*& email addresses.
 The third example shows that the presence of a comma is skipped when it is

-quoted.
+quoted.  The fourth example shows SMTPUTF8 handling.

 .code
 # exim -be '${addresses:From: \
 =?iso-8859-2?Q?Last=2C_First?= <user@example.com>}'
 .code
 # exim -be '${addresses:From: \
 =?iso-8859-2?Q?Last=2C_First?= <user@example.com>}'


@@ -10245,6 +10255,8 @@ user@example.com
 Last:user@example.com
 # exim -be '${addresses:From: "Last, First" <user@example.com>}'
 user@example.com
 Last:user@example.com
 # exim -be '${addresses:From: "Last, First" <user@example.com>}'
 user@example.com

+# exim -be '${addresses:フィル <フィリップ@example.jp>}'
+フィリップ@example.jp

 .endd
 
 .vitem &*${base32:*&<&'digits'&>&*}*&
 .endd
 
 .vitem &*${base32:*&<&'digits'&>&*}*&


@@ -10476,6 +10488,7 @@ This forces the letters in the string into lower-case, for example:
 .code
 ${lc:$local_part}
 .endd
 .code
 ${lc:$local_part}
 .endd

+Case is defined per the system C locale.

 
 .vitem &*${length_*&<&'number'&>&*:*&<&'string'&>&*}*&
 .cindex "expansion" "string truncation"
 
 .vitem &*${length_*&<&'number'&>&*:*&<&'string'&>&*}*&
 .cindex "expansion" "string truncation"


@@ -10489,6 +10502,7 @@ ${length{<number>}{<string>}}
 See the description of the general &%length%& item above for details. Note that
 &%length%& is not the same as &%strlen%&. The abbreviation &%l%& can be used
 when &%length%& is used as an operator.
 See the description of the general &%length%& item above for details. Note that
 &%length%& is not the same as &%strlen%&. The abbreviation &%l%& can be used
 when &%length%& is used as an operator.

+All measurement is done in bytes and is not UTF-8 aware.

 
 
 .vitem &*${listcount:*&<&'string'&>&*}*&
 
 
 .vitem &*${listcount:*&<&'string'&>&*}*&


@@ -10516,6 +10530,7 @@ matching list is returned.
 The string is interpreted as an RFC 2822 address and the local part is
 extracted from it. If the string does not parse successfully, the result is
 empty.
 The string is interpreted as an RFC 2822 address and the local part is
 extracted from it. If the string does not parse successfully, the result is
 empty.

+The parsing correctly handles SMTPUTF8 Unicode in the string.

 
 
 .vitem &*${mask:*&<&'IP&~address'&>&*/*&<&'bit&~count'&>&*}*&
 
 
 .vitem &*${mask:*&<&'IP&~address'&>&*/*&<&'bit&~count'&>&*}*&


@@ -10598,6 +10613,10 @@ example, a plus sign would not cause quoting (but it would for &%quote%&).
 If you are creating a new email address from the contents of &$local_part$&
 (or any other unknown data), you should always use this operator.
 
 If you are creating a new email address from the contents of &$local_part$&
 (or any other unknown data), you should always use this operator.
 

+This quoting determination is not SMTPUTF8-aware, thus quoting non-ASCII data
+will likely use the quoting form.
+Thus &'${quote_local_part:フィル}'& will always become &'"フィル"'&.
+

 
 .vitem &*${quote_*&<&'lookup-type'&>&*:*&<&'string'&>&*}*&
 .cindex "quoting" "lookup-specific"
 
 .vitem &*${quote_*&<&'lookup-type'&>&*:*&<&'string'&>&*}*&
 .cindex "quoting" "lookup-specific"


@@ -10761,6 +10780,7 @@ Now deprecated, a synonym for the &%base64%& expansion operator.
 .cindex "&%strlen%& expansion item"
 The item is replace by the length of the expanded string, expressed as a
 decimal number. &*Note*&: Do not confuse &%strlen%& with &%length%&.
 .cindex "&%strlen%& expansion item"
 The item is replace by the length of the expanded string, expressed as a
 decimal number. &*Note*&: Do not confuse &%strlen%& with &%length%&.

+All measurement is done in bytes and is not UTF-8 aware.

 
 
 .vitem &*${substr_*&<&'start'&>&*_*&<&'length'&>&*:*&<&'string'&>&*}*&
 
 
 .vitem &*${substr_*&<&'start'&>&*_*&<&'length'&>&*:*&<&'string'&>&*}*&


@@ -10775,6 +10795,7 @@ ${substr{<start>}{<length>}{<string>}}
 .endd
 See the description of the general &%substr%& item above for details. The
 abbreviation &%s%& can be used when &%substr%& is used as an operator.
 .endd
 See the description of the general &%substr%& item above for details. The
 abbreviation &%s%& can be used when &%substr%& is used as an operator.

+All measurement is done in bytes and is not UTF-8 aware.

 
 .vitem &*${time_eval:*&<&'string'&>&*}*&
 .cindex "&%time_eval%& expansion item"
 
 .vitem &*${time_eval:*&<&'string'&>&*}*&
 .cindex "&%time_eval%& expansion item"


@@ -10797,6 +10818,7 @@ number of larger units and output in Exim's normal time format, for example,
 .cindex "expansion" "case forcing"
 .cindex "&%uc%& expansion item"
 This forces the letters in the string into upper-case.
 .cindex "expansion" "case forcing"
 .cindex "&%uc%& expansion item"
 This forces the letters in the string into upper-case.

+Case is defined per the system C locale.

 
 .vitem &*${utf8clean:*&<&'string'&>&*}*&
 .cindex "correction of invalid utf-8 sequences in strings"
 
 .vitem &*${utf8clean:*&<&'string'&>&*}*&
 .cindex "correction of invalid utf-8 sequences in strings"


@@ -10805,6 +10827,20 @@ This forces the letters in the string into upper-case.
 .cindex "expansion" "utf-8 forcing"
 .cindex "&%utf8clean%& expansion item"
 This replaces any invalid utf-8 sequence in the string by the character &`?`&.
 .cindex "expansion" "utf-8 forcing"
 .cindex "&%utf8clean%& expansion item"
 This replaces any invalid utf-8 sequence in the string by the character &`?`&.

+.new
+In versions of Exim before 4.92, this did not correctly do so for a truncated
+final codepoint's encoding, and the character would be silently dropped.
+If you must handle detection of this scenario across both sets of Exim behavior,
+the complexity will depend upon the task.
+For instance, to detect if the first character is multibyte and a 1-byte
+extraction can be successfully used as a path component (as is common for
+dividing up delivery folders), you might use:
+.code
+condition = ${if inlist{${utf8clean:${length_1:$local_part}}}{:?}{yes}{no}}
+.endd
+(which will false-positive if the first character of the local part is a
+literal question mark).
+.wen

 
 .vitem "&*${utf8_domain_to_alabel:*&<&'string'&>&*}*&" &&&
        "&*${utf8_domain_from_alabel:*&<&'string'&>&*}*&" &&&
 
 .vitem "&*${utf8_domain_to_alabel:*&<&'string'&>&*}*&" &&&
        "&*${utf8_domain_from_alabel:*&<&'string'&>&*}*&" &&&


@@ -11027,7 +11063,8 @@ the header name must be terminated by a colon if white space does not follow.
 .cindex "&%eqi%& expansion condition"
 The two substrings are first expanded. The condition is true if the two
 resulting strings are identical. For &%eq%& the comparison includes the case of
 .cindex "&%eqi%& expansion condition"
 The two substrings are first expanded. The condition is true if the two
 resulting strings are identical. For &%eq%& the comparison includes the case of

-letters, whereas for &%eqi%& the comparison is case-independent.
+letters, whereas for &%eqi%& the comparison is case-independent, where
+case is defined per the system C locale.

 
 .vitem &*exists&~{*&<&'file&~name'&>&*}*&
 .cindex "expansion" "file existence test"
 
 .vitem &*exists&~{*&<&'file&~name'&>&*}*&
 .cindex "expansion" "file existence test"


@@ -11090,6 +11127,7 @@ The two substrings are first expanded. The condition is true if the first
 string is lexically greater than or equal to the second string. For &%ge%& the
 comparison includes the case of letters, whereas for &%gei%& the comparison is
 case-independent.
 string is lexically greater than or equal to the second string. For &%ge%& the
 comparison includes the case of letters, whereas for &%gei%& the comparison is
 case-independent.

+Case and collation order are defined per the system C locale.

 
 .vitem &*gt&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*& &&&
        &*gti&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*&
 
 .vitem &*gt&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*& &&&
        &*gti&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*&


@@ -11101,6 +11139,7 @@ The two substrings are first expanded. The condition is true if the first
 string is lexically greater than the second string. For &%gt%& the comparison
 includes the case of letters, whereas for &%gti%& the comparison is
 case-independent.
 string is lexically greater than the second string. For &%gt%& the comparison
 includes the case of letters, whereas for &%gti%& the comparison is
 case-independent.

+Case and collation order are defined per the system C locale.

 
 .vitem &*inlist&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*& &&&
        &*inlisti&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*&
 
 .vitem &*inlist&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*& &&&
        &*inlisti&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*&


@@ -11109,6 +11148,7 @@ case-independent.
 Both strings are expanded; the second string is treated as a list of simple
 strings; if the first string is a member of the second, then the condition
 is true.
 Both strings are expanded; the second string is treated as a list of simple
 strings; if the first string is a member of the second, then the condition
 is true.

+For the case-independent &%inlisti%& condition, case is defined per the system C locale.

 
 These are simpler to use versions of the more powerful &*forany*& condition.
 Examples, and the &*forany*& equivalents:
 
 These are simpler to use versions of the more powerful &*forany*& condition.
 Examples, and the &*forany*& equivalents:


@@ -11175,6 +11215,7 @@ The two substrings are first expanded. The condition is true if the first
 string is lexically less than or equal to the second string. For &%le%& the
 comparison includes the case of letters, whereas for &%lei%& the comparison is
 case-independent.
 string is lexically less than or equal to the second string. For &%le%& the
 comparison includes the case of letters, whereas for &%lei%& the comparison is
 case-independent.

+Case and collation order are defined per the system C locale.

 
 .vitem &*lt&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*& &&&
        &*lti&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*&
 
 .vitem &*lt&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*& &&&
        &*lti&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*&


@@ -11186,6 +11227,7 @@ The two substrings are first expanded. The condition is true if the first
 string is lexically less than the second string. For &%lt%& the comparison
 includes the case of letters, whereas for &%lti%& the comparison is
 case-independent.
 string is lexically less than the second string. For &%lt%& the comparison
 includes the case of letters, whereas for &%lti%& the comparison is
 case-independent.

+Case and collation order are defined per the system C locale.

 
 
 .vitem &*match&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*&
 
 
 .vitem &*match&~{*&<&'string1'&>&*}{*&<&'string2'&>&*}*&


@@ -11212,6 +11254,8 @@ metacharacter, but if there is no circumflex, the expression is not anchored,
 and it may match anywhere in the subject, not just at the start. If you want
 the pattern to match at the end of the subject, you must include the &`$`&
 metacharacter at an appropriate point.
 and it may match anywhere in the subject, not just at the start. If you want
 the pattern to match at the end of the subject, you must include the &`$`&
 metacharacter at an appropriate point.

+All character handling is done in bytes and is not UTF-8 aware,
+but we might change this in a future Exim release.

 
 .cindex "numerical variables (&$1$& &$2$& etc)" "in &%if%& expansion"
 At the start of an &%if%& expansion the values of the numeric variable
 
 .cindex "numerical variables (&$1$& &$2$& etc)" "in &%if%& expansion"
 At the start of an &%if%& expansion the values of the numeric variable

diff --git a/doc/doc-txt/ChangeLog b/doc/doc-txt/ChangeLog
index bc521a6a3bf70bf7d142575508963b48bbf3c5f3..583603f19ac0e043e73d54a65ba0a78377f35d74 100644 (file)
--- a/doc/doc-txt/ChangeLog
+++ b/doc/doc-txt/ChangeLog
@@ -96,6 +96,8 @@ PP/03 Make ${utf8clean:} expansion operator detect incomplete final character.
       Previously if the string ended mid-character, we did not insert the
       promised '?' replacement.
 
       Previously if the string ended mid-character, we did not insert the
       promised '?' replacement.
 

+PP/04 Documentation: current string operators work on bytes, not codepoints.
+

 
 Exim version 4.91
 -----------------
 
 Exim version 4.91
 -----------------