I'm wondering why we don't have some string classes that represent a string of Unicode grapheme clusters instead of code points or characters. It seems to me that in most applications it would be easier for programmers to access components of a grapheme when necessary than to have to organize them from code points, which appears necessary even if only to avoid casually breaking a string in "mid-grapheme" (at least in theory). Internally a string class might use a variable length encoding such as UTF-8, UTF-16, or in this context even UTF-32 is variable length; or implement subclasses for all of them (and optionally configure the choice at run-time so that different languages could use their optimal encodings). But if programmers could "see" grapheme units when inspecting a string, wouldn't string handling code in general be closer to achieving correctness, and without much extra complexity?

  • I guess a bit of time passed, and now we have a couple of languages which actually do this. :D
    – Hakanai
    Commented Nov 2, 2017 at 0:09
  • @Hakanai mind to share which do? Commented Apr 11, 2022 at 14:38
  • 1
    @0xC0000022L Swift and Elixir at least, but probably even more by now.
    – Hakanai
    Commented Apr 13, 2022 at 2:12
  • @Hakanai Thanks. I only looked up Elixir and you're right it provides access to the grapheme clusters. However, if I see it right, you have to specifically request that info. Which makes sense. It's an expensive operation that only ever matters in the presentation layer of software. To me this still differs from a "string class based on grephemes". Sorry for the nitpicking. (Didn't check out Swift.) Commented Apr 13, 2022 at 7:50
  • 1
    @0xC0000022L They also take them into account when doing things like string equality comparison. Yeah, at a lower level that isn't how they're stored... but in the case of Elixir, I'd almost guess that was inevitable because they wanted to maintain compatibility with Erlang, which had already decided how to store strings. I haven't looked too closely at Swift, but I imagine they are going to be the same, because they wanted to interoperate with ObjC.
    – Hakanai
    Commented Apr 14, 2022 at 2:56

1 Answer 1


Seems like the best way to get correctness is to keep programmers from doing "string hacking" ... it just isn't OK to write your own word wrap, hyphenation, word count, justification, cursor movement, etc. routines. All the modern UI frameworks will do this stuff for you these days.

That is, the abstraction you'd usually work with is more of a "paragraph display object," such as for GTK: http://library.gnome.org/devel/pango/stable/pango-Layout-Objects.html

rather than a grapheme string, such as: http://library.gnome.org/devel/pango/stable/pango-Glyph-Storage.html

To get to a string of glyphs you need info that's only available at the "view" level, so most uses of strings might not have this info. For example, you have to know the font, because fonts can have different ligatures.

Aside from that kind of practical matter, glyphs probably aren't what you want.

In many contexts, you want to use the proper Unicode attributes, shown in this API for example: http://library.gnome.org/devel/pango/stable/pango-Text-Processing.html#PangoLogAttr

As you can see from that struct (which reflects the Unicode algorithms) doing various things at glyph boundaries is not any more correct than doing them at character boundaries.

These two specs describe the algorithms to find different kinds of boundaries:

Doing text processing involves finding those boundaries with the algorithms and then working with the boundaries.

If you start digging in on just how hard it is to handle all languages correctly, you'll very quickly realize you need a library that looks at whole paragraphs and handles them properly. Windows, Mac, Linux (Qt and GTK), and Java all come with facilities for this, plus there's http://site.icu-project.org/ for example.

When writing web apps, unfortunately you pretty much have to let the browser (probably helped by the OS) do this stuff, as far as I know. All you can do in JavaScript or on the server side is mess it up.

Maybe I'd sum up the answer as: most string manipulation on natural language text is broken, so not much point worrying about the string class, other than maybe to have one with no methods on it ;-)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.