I've been working on a UTF-8 iterator adapter. By which, I mean an adapter that turns an iterator to a
unsigned char sequence into an iterator to a
char32_t sequence. My work here was inspired by this iterator I found online.
However, as I looked through the standard when I was beginning my own implementation, I came to a realization: it does not appear to be possible to implement such an adaptor while still conforming to the requirements that C++ places on iterators.
For example, could you create a UTF-8 iterator that satisfies the InputIterator requirements? Yes, but only so long as the iterator you are given is not itself an InputIterator. Why?
Because InputIterator requires the ability to dereference the same iterator more than once. You can also dereference multiple copies of that iterator, so long as they all compare equal.
Of course, dereferencing a UTF-8 iterator adaptor requires both dereferencing and potentially incrementing the base iterator. And if that iterator is an InputIterator, then you can't get the original value back after you increment it. And the fact that copies have to work means that you can't locally store a
char32_t that represents the previously-decoded value. You could have done this:
auto it = ... auto it2 = it; //Copies an empty `char32_t`. *it; //Accesses base iterator, storing `it.ch`. *it; //Doesn't access the base iterator; simply returns `it.ch`. *it2; //Cannot access `it.ch`, so must access base iterator.
OK, fine, so you can't use InputIterators. But what about ForwardIterator? Is it possible to create a ForwardIterator adaptor that can adapt ForwardIterators over UTF-8 character sequences?
That is problematic as well, because the operation
*it is required to produce
const value_type&. InputIterators can spit out anything which is convertible to
value_type, but a
ForwardIterator is required to provide an actual reference [forward.iterators]/1.3:
Xis a mutable iterator,
referenceis a reference to
Xis a constant iterator,
referenceis a reference to
The only recourse here is for every such iterator to carry around a
char32_t, which exists solely to provide the storage for that reference. And even then, that value will have to be updated every time the the iterator instance is incremented and dereferenced. This effectively invalidates the old reference, and the standard doesn't explicitly permit that (invalidation can only happen when an iterator is destroyed, or if the container says so).
The aforementioned code I found online isn't valid due to this, as it returns a
uint32_t (written pre-C++11) by value rather than a proper reference.
Is there any recourse here? Have I overlooked something in the standard, or some implementation technique I could use to bypass these issues? Or is this simply not possible with the current wording of the standard?
Note: the odd thing is that it seems to be possible to write a conforming OutputIterator for UTF-8 conversion. That is, a type which takes
char32_t and writes UTF-8 to a
unsigned char OutputIterator.