This is an unofficial snapshot of the ISO/IEC JTC1 SC22 WG21 Core Issues List revision 116a. See http://www.open-std.org/jtc1/sc22/wg21/ for the official list.

2024-12-19

2639. new-lines after phase 1

[Accepted at the November, 2022 meeting.]

Translation phases 2 and 3 assume that lines are terminated by "new-line characters". However, the current specification of phase 1 does not guarantee that to be true. In particular, for a UTF-8 file the verbatim sequence of source file characters forms the input for phase 2, even on systems where the line terminator is a carriage return. The non-UTF-8 specification is also defective in that it speaks of "introducing" new-line characters, even for encodings like Latin-1 where new-lines might already be present and no "introduction" is needed or appropriate.

Proposed resolution [SUPERSEDED]:

Change in 5.2 [lex.phases] paragraph 1.1 as follows:

... If an input file is determined to be a UTF-8 file, then it shall be a well-formed UTF-8 code unit sequence and it is decoded to produce a sequence of UCS scalar values that constitutes the sequence of elements of the translation character set, representing each line-termination character or character sequence as a new-line character.

For any other kind of input file supported by the implementation, characters are mapped, in an implementation-defined manner, to a sequence of translation character set elements (5.3.1 [lex.charset]) (introducing new-line characters for representing end-of-line indicators as new-line characters).

Proposed resolution (approved by CWG 2022-11-08):

Change in 5.2 [lex.phases] paragraph 1.1 as follows:

... If an input file is determined to be a UTF-8 file, then it shall be a well-formed UTF-8 code unit sequence and it is decoded to produce a sequence of UCS scalar values that constitutes the sequence of elements of the translation character set. In the resulting sequence, each pair of characters in the input sequence consisting of U+000D CARRIAGE RETURN followed by U+000A LINE FEED, as well as each U+000D CARRIAGE RETURN not immediately followed by a U+000A LINE FEED, is replaced by a single new-line character.

For any other kind of input file supported by the implementation, characters are mapped, in an implementation-defined manner, to a sequence of translation character set elements (5.3.1 [lex.charset]) (introducing new-line characters for , representing end-of-line indicators as new-line characters ).