diff --git a/papers/P2728.md b/papers/P2728.md index 7c8b6df..4e85ac1 100644 --- a/papers/P2728.md +++ b/papers/P2728.md @@ -1,7 +1,7 @@ --- title: "Unicode in the Library, Part 1: UTF Transcoding" -document: D2728R14 -date: 2026-06-07 +document: P2728R14 +date: 2026-06-10 audience: - SG-16 Unicode - SG-9 Ranges @@ -205,8 +205,15 @@ range in order to bounds check its beginning and end (which is required for correctness, not just safety). The `to_utf_view::@*iterator*@` maintains a small buffer (`buf_`) -containing between one and four code units, which comprise the current -character in the target encoding. +containing the current character, transcoded into the target encoding. If the +underlying range models `forward_range`, the buffer may additionally contain +the transcoded code units of one or more characters following the current one: +an implementation is permitted to transcode a whole chunk of input at a time +(for example, using SIMD instructions) and serve subsequent increments — or, +when iterating backward, decrements — out of the buffer. If the underlying +range is single-pass, the buffer contains the code units of exactly one +character (between one and four code units), because reading ahead in a +single-pass range is destructive and therefore observable. It also maintains an index (`buf_index_`) into this buffer, which it increments or decrements when `operator++` or `operator--` is invoked, respectively. If @@ -232,6 +239,10 @@ the fourth character to the third character by invoking contain most of the actual transcoding logic, updating `current_` and filling `buf_` up with the transcoded characters. +(The diagram depicts the minimal buffer, holding one character at a time. As +described above, an implementation may instead fill `buf_` with the code units +of several consecutive characters when the underlying range is multipass.) + Iterating a bidirectional transcoding view backwards produces, in reverse order, the exact same sequence of characters or `expected` values as are produced by iterating the view forwards. @@ -787,10 +798,10 @@ private: iterator_t<@*Base*@> current_{}; // @*exposition only*@ sentinel_t<@*Base*@> end_; // @*exposition only*@ - inplace_vector buf_{}; // @*exposition only*@ + inplace_vector buf_{}; // @*exposition only*@ - int8_t buf_index_{}; // @*exposition only*@ - uint8_t to_increment_{}; // @*exposition only*@ + ptrdiff_t buf_index_{}; // @*exposition only*@ + size_t to_increment_{}; // @*exposition only*@ template requires view && @*code-unit*@> @@ -823,13 +834,8 @@ private: } public: - constexpr const iterator_t<@*Base*@>& base() const& noexcept - requires forward_range<@*Base*@> - { return current_; } - - constexpr iterator_t<@*Base*@> base() && - requires forward_range<@*Base*@> - { return std::move(current_); } + constexpr iterator_t<@*Base*@> base() const + requires forward_range<@*Base*@>; constexpr value_type operator*() const; @@ -863,10 +869,15 @@ public: constexpr @*iterator*@& operator--() requires bidirectional_range<@*Base*@> { - if (!buf_index_) + if (!buf_index_) { @*read-reverse*@(); - else + } else { --buf_index_; + if constexpr (E == to_utf_view_error_kind::expected && is_same_v) { + if (!@*success*@()) + buf_index_ -= 2; + } + } return *this; } @@ -877,10 +888,8 @@ public: return retval; } - friend constexpr bool operator==(const @*iterator*@& lhs, const @*iterator*@& rhs) requires equality_comparable> - { - return lhs.current_ == rhs.current_ && lhs.buf_index_ == rhs.buf_index_; - } + friend constexpr bool operator==(const @*iterator*@& lhs, const @*iterator*@& rhs) + requires equality_comparable>; private: constexpr sentinel_t<@*Base*@> @*end*@() const { // @*exposition only*@ @@ -911,6 +920,18 @@ private: }; ``` +`@*buffer-capacity*@` is an unspecified constant that is at least +`4 / sizeof(ToType)`. If `@*Base*@` does not model `forward_range`, +`@*buffer-capacity*@` is `4 / sizeof(ToType)`. + +::: note + +A `@*buffer-capacity*@` greater than `4 / sizeof(ToType)` permits +`@*read*@` to transcode several input subsequences per invocation, for example +a chunk at a time using SIMD instructions. + +::: + ::: note `to_utf_view::@*iterator*@` does its work by adapting an underlying @@ -921,7 +942,10 @@ corresponding to `@*from-type*@`. If the underlying range contains ill-formed UTF, the code units are divided into input subsequences according to Substitution of Maximal Subparts, and each ill-formed input subsequence is transcoded into a `U+FFFD`. `c` is then encoded to `ToType`'s corresponding -encoding, into an internal code unit buffer `buf_`. +encoding, into an internal code unit buffer `buf_`. `buf_` may contain the +transcoded code units of more than one input subsequence; the *current* input +subsequence is the input subsequence whose transcoded code units include +`buf_[buf_index_]`. ::: @@ -950,6 +974,53 @@ In that case, `to_utf_view::@*iterator*@::iterator_category` is defined as follo - Otherwise, if `C` models `derived_from`, then `iterator_category` denotes `forward_iterator_tag`. - Otherwise, `iterator_category` denotes `C`. +```cpp +constexpr iterator_t<@*Base*@> base() const + requires forward_range<@*Base*@>; +``` + +_Returns_: If `*this` is at the end of the range being adapted, an iterator +equal to the end of the range being adapted. Otherwise, an iterator pointing +to the first code unit of the current input subsequence. + +::: note + +An implementation whose `@*read*@` transcodes a single input +subsequence per invocation can return `current_`. An implementation that +transcodes several input subsequences per invocation can recompute this +position from `current_` and `buf_index_`; the recomputation takes time +bounded by `@*buffer-capacity*@`, a constant. + +::: + +```cpp +friend constexpr bool operator==(const @*iterator*@& lhs, const @*iterator*@& rhs) + requires equality_comparable>; +``` + +_Returns_: If `@*Base*@` models `forward_range`, `true` if and only if either +`lhs` and `rhs` are both at the end of the range being adapted, or the current +input subsequences of `lhs` and `rhs` begin at the same position in the +underlying range and `*lhs` and `*rhs` denote the code unit at the same offset +within the transcoded code units of that input subsequence. Otherwise, +`lhs.current_ == rhs.current_ && lhs.buf_index_ == rhs.buf_index_`. + +::: note + +For an implementation whose `@*read*@` and `@*read-reverse*@` transcode a +single input subsequence per invocation, the first comparison is also +equivalent to +`lhs.current_ == rhs.current_ && lhs.buf_index_ == rhs.buf_index_`. For an +implementation that transcodes several input subsequences per invocation, the +stored members alone do not identify a position: two iterators denoting the +same element can hold different `current_` values if their buffers were filled +starting from different positions (for example, when one of them was filled +moving forward by `@*read*@` and the other moving backward by +`@*read-reverse*@`, or when their buffers were filled with chunks of +different extents). + +::: + ```cpp constexpr value_type operator*() const; ``` @@ -1006,27 +1077,88 @@ constexpr void @*read*@(); // @*exposition only*@ _Effects_: -Decodes the input subsequence starting at position `current_` into a code point -`c`, using the UTF encoding corresponding to `@*from-type*@`, and setting `c` -to U+FFFD if the input subsequence is ill-formed. It sets `to_increment_` to -the number of code units read while decoding `c`. encodes `c` into `buf_` in -the UTF encoding corresponding to `ToType`, and sets `buf_index_` to `0`. If +Let `n` be a number of consecutive input subsequences, chosen by the +implementation, such that: + +- `n >= 1`; +- `n` does not exceed the number of input subsequences remaining in the + underlying range, starting at position `current_`; +- the total number of code units produced by transcoding those `n` input + subsequences as described below is at most `@*buffer-capacity*@`; and +- if `forward_range<@*Base*@>` is not modeled, `n == 1`. + +Clears `buf_`. Then, for each of the `n` consecutive input subsequences +starting at position `current_`, in order: decodes the input subsequence into +a code point `c`, using the UTF encoding corresponding to `@*from-type*@`, +setting `c` to U+FFFD if the input subsequence is ill-formed; and appends the +code units of `c`, encoded in the UTF encoding corresponding to `ToType`, to +`buf_`. Sets `to_increment_` to the total number of code units comprising +those `n` input subsequences, and sets `buf_index_` to `0`. If `forward_range<@*Base*@>` is modeled, `current_` is set to the position it had before `@*read*@` was called. +::: note + +The choice of `n` is not observable: the division of the input into +input subsequences does not depend on it, so the sequence of elements produced +by the view is the same for every valid choice. `n` need not be the same on +each invocation and can depend on the contents of the underlying range: an +implementation typically transcodes a fixed-size window of code units, +trimmed back to a whole number of input subsequences, so the number of input +subsequences per chunk varies with how many code units each occupies. +Choosing `n > 1` permits an +implementation to transcode a chunk of input per invocation of `@*read*@`, +for example using SIMD instructions. + +::: + ```cpp constexpr void @*read-reverse*@(); // @*exposition only*@ ``` _Effects_: -Decodes the input subsequence ending at position `current_` into a code point -`c`, using the UTF encoding corresponding to `@*from-type*@`, and setting `c` -to U+FFFD if the input subsequence is ill-formed. It sets `to_increment_` to -the number of code units read while decoding `c`; encodes `c` into `buf_` in -the UTF encoding corresponding to `ToType`; and sets `buf_index_` to -`buf_.size() - 1`, or to `0` if this is an `or_error` view and we read an -invalid subsequence. +Let `n` be a number of consecutive input subsequences, chosen by the +implementation, such that: + +- `n >= 1`; +- `n` does not exceed the number of input subsequences in the underlying + range preceding position `current_`; and +- the total number of code units produced by transcoding those `n` input + subsequences as described below is at most `@*buffer-capacity*@`. + +Clears `buf_`. Then, for each of the `n` consecutive input subsequences +ending at position `current_`, in order: decodes the input subsequence into a +code point `c`, using the UTF encoding corresponding to `@*from-type*@`, +setting `c` to U+FFFD if the input subsequence is ill-formed; and appends the +code units of `c`, encoded in the UTF encoding corresponding to `ToType`, to +`buf_`. Sets `to_increment_` to the total number of code units comprising +those `n` input subsequences, and sets `current_` to the position of the +beginning of the first of those `n` input subsequences. Sets `buf_index_` to +the index in `buf_` of the first code unit produced by transcoding the last +of those `n` input subsequences if `E` is `to_utf_view_error_kind::expected` +and that input subsequence is ill-formed, and to `buf_.size() - 1` otherwise. + +::: note + +The `n` consecutive input subsequences ending at position `current_` are +well-defined: the division of the code units preceding `current_` into input +subsequences under Substitution of Maximal Subparts does not depend on the +code units at or after `current_`. As with `@*read*@`, the choice of `n` is +not observable, and `n` need not be the same on each invocation: it can +depend on the contents of the underlying range. In particular, an +implementation need not decide on a number of input subsequences in advance. +It can instead step backward over a fixed number of code units (chosen so +that the transcoded result is guaranteed to fit in `buf_`), locate the first +input subsequence boundary at or after that position by examining a bounded +number of neighboring code units, and transcode forward from there to +`current_`; `n` is then however many input subsequences that window happens +to contain — fewer when the code points are encoded with more code units +apiece. An implementation may also always choose `n == 1`, since backward +iteration is typically used for short, local movements that would not +amortize the cost of transcoding a large chunk. + +::: #### 25.7.?.7 Class `to_utf_view::@*sentinel*@` [range.transcoding.sentinel] {-} @@ -1161,8 +1293,11 @@ at that point. ## Why We Don't Cache `begin()` -When we invoke `begin()`, constructing the transcoding iterator may read up to four -elements from the underlying view if it's transcoding from UTF-8. A previous revision of +When we invoke `begin()`, constructing the transcoding iterator may read a +bounded number of elements from the underlying view — up to four if it's +transcoding from UTF-8 and buffering a single code point, or up to the +(constant) capacity of its internal buffer if it transcodes a chunk of input +at a time. A previous revision of this paper implemented `begin()` caching, based on the idea that iterating the underlying range could have unbounded complexity. @@ -1584,6 +1719,99 @@ above. An experimental implementation of `.base_code_units()` is available on the `enolan_basecodeunits2` branch of `beman.utf_view`. +## SIMD support + +The wording has been updated to allow implementations to read code points in chunks rather +than one at a time, which enables support for SIMD. + +To my knowledge, this would be the first view that reads its input in chunks for reasons +of performance rather than correctness. + +The transcoding iterator must be increased in size in order to fit a larger buffer which +contains the output of the SIMD transcoding kernel. On the other hand, there is no need to +store the input buffer in the iterator. + +The decision of which output buffer size to select is left up to the implementation, but +once chosen, an implementation can't change it without breaking ABI. (This also implies +that, as a process note, it's not possible for us to standardize a version of this +facility that *doesn't* allow SIMD and then patch SIMD support onto it in a future paper.) + +The invariant of `.base()` is that it points to the beginning of the code unit range for +the current code point in the underlying view. Transcoding more than one code point at a +time slightly complicates the implementation of `.base()` relative to the scalar +implementation. For example, an implementation might want to store an iterator to the +beginning of the current chunk in the underlying view and then, when `.base()` is invoked, +iterate it forward from the beginning of the chunk to the start of the current code +point. This means that, whereas the scalar implementation of `.base()` is a simple accessor +to a data member, the chunked implementation may need to perform `CHUNK_SIZE` iterator +increments internally (which is still O(1)). + +Since it's challenging to implement fully conformant Substitution of Maximal Subparts +error handling in a SIMD transcoding kernel, we expect that implementers will add a +validation step to their SIMD transcoding kernels and fall back to the serial +implementation on invalid UTF. To fully benefit from the fast path, you need valid input. + +SIMD support is enabled for forward ranges only. Here is an example of why the chunking +behavior breaks input ranges. Say we have a video game that reads the player's name but +only allows space for five code points: + +```cpp +std::u32string get_player_name() { + std::ranges::subrange input_view( + std::istreambuf_iterator(std::cin), + std::istreambuf_iterator{}); + // get 5 code points of input + return input_view + | beman::utf_view::as_char8_t + | beman::utf_view::to_utf32 + | std::views::take(5) + | std::ranges::to(); +} +``` + +(This is actually somewhat more complicated to do than depicted here.) + +With a chunked implementation, the user types "GAM3R" but then needs to keep typing until +the chunk gets filled up, only for the rest of the chunk to get discarded. + +The reference implementation's benchmark transcodes the +[unicode_lipsum](https://github.com/lemire/unicode_lipsum) corpora from UTF-16 to +UTF-8. The SIMD implementation uses a prototype kernel written against C++26 `std::simd` +and a buffer capacity of 128 code units. For comparison, the last column is a single bulk +`simdutf` call over the whole corpus, with no view involved. Numbers are GiB/s of input +consumed (GCC 16.1, `-O3 -march=native`, x86-64 AVX2, AMD Ryzen 9 5950X). + +| Corpus | Scalar view | Prototype SIMD view | Bulk `simdutf` | +|---|---:|---:|---:| +| Latin | 1.82 | 4.01 | 60.4 | +| Arabic | 0.83 | 1.05 | 10.9 | +| Chinese | 0.73 | 0.99 | 8.9 | +| Japanese | 0.70 | 0.76 | 8.7 | +| Korean | 0.72 | 0.76 | 8.0 | + +In the current prototype, we see >2x speedup on the most favorable case, which is ASCII +input, and moderate to small speedups on other corpuses. (Note that the prototype is +currently in an incomplete state, and only implements the UTF16-UTF8 direction, so results +with other directions may differ). `simdutf` smokes us, mainly due to benefiting from the +bulk API, and we can't approach its speed with a view; we would need to do an algorithm to +achieve comparable performance.\* + +To put these numbers into perspective, the article text of English Wikipedia is roughly +40 GiB in UTF-8, or about 80 GiB in UTF-16, and so would take approximately 44 seconds +to transcode on a single core with the scalar implementation and 20 seconds to +transcode with SIMD, assuming its properties are roughly similar to the Latin corpus +above. + +\* (This is because a view delivers its output one code unit at a time: every element +costs an iterator increment, a dereference, and a buffer-index check, no matter how +cheaply the buffer was filled. SIMD accelerates only the buffer refill; once that cost +is amortized away, throughput is bounded by the per-element delivery loop — on the +Latin row above, the SIMD view is already running at about two cycles per code unit, +which is the cost of the loop itself rather than of transcoding. A bulk API has no +per-element step at all: it reads and writes entire vectors. Closing the gap therefore +requires an interface that writes directly to an output range, not a faster kernel +inside the view.) + # Changelog ## Changes since R13 @@ -1591,6 +1819,17 @@ An experimental implementation of `.base_code_units()` is available on the - Fix typo in Table 3-9 of error handling diagram pointed out during SG16 review (`ED A0` had been misspelled as `E0 A0`). - Add design discussion about `.base_code_units()`. +- Loosen the wording to admit chunked (e.g. SIMD) implementations when the + underlying range models `forward_range`: make `buf_`'s capacity an + unspecified constant, allow `@*read*@` and `@*read-reverse*@` to transcode + several input subsequences per invocation, and respecify `base()` and + iterator equality positionally instead of in terms of the exposition-only + members. Make `operator--` skip an ill-formed input subsequence's + replacement-character code units as a unit in `_or_error` views, mirroring + `operator++`, since with chunking they can now appear in the middle of + `buf_`. Specify explicitly that `@*read-reverse*@` leaves `current_` at the + beginning of the first input subsequence it transcoded. Add corresponding + design discussion. ## Changes since R12