Perl’s 1990s Unicode Support in Function Names
I can’t say that Perl was “the first” programming language to allow Unicode in function names — this seems unlikely to me — but I can definitively document when it first began to do so.
First Repository Commit: 834a4ddd8309fbf6aabbbc51bb6fcbe056e7963f on 1998-10-23
It was in this commit from 1998 that Larry Wall fixed the bugs standing in the way of using “UTF-8 identifiers” in Perl:
commit 834a4ddd8309fbf6aabbbc51bb6fcbe056e7963f
Author: Larry Wall <[email protected]>
Date: Fri Oct 23 18:00:41 1998 +0000
Program with utf8 identifiers fails to compile
Hindsight shows that it proved a blessing that it took us that long, for it was to our great good fortune that we held off (some might say dilly-dallied) for years before finally adding any Unicode support to the core Perl distribution until an earlier commit of Larry’s in July of 1998:
commit a0ed51b321531af4b47cce24205ab9656f043f0f
Author: Larry Wall <[email protected]>
Date: Fri Jul 24 05:44:33 1998 +0000
Here are the long-expected Unicode/UTF-8 modifications.
Notice that that was a couple years after Unicode 2.0’s release in 1996.
Delaying Unicode strings in the core language allowed us to avoid the sea of troubles that flooded the early-adopter languages that had jumped right into Unicode 1.0’s 16-bit characters, and which all gut stuck with that. By waiting to get our feet wet in that ancient sea until “modern” Unicode’s 21-bit code point repertoire, we were able to use UTF-8 as our internal representation for all our strings.
First Development Release: version 5.005_54 on 1998-11-30
Tracking Perl’s historical minor releases towards the end of the last millennium can be tricky to the uninitiated owing to the peculiar version-numbering conventions it was still at that time using for its parallel development and maintenance tracks.
First Stable/Production Release: v5.6 on 2000-03-22
Suffice it to say that being able to use Unicode in function names first appeared in the 5.005_54 minor development release, which was building to eventually producing the v5.6 release in March 2000. The v5.6 release notes contain this passage:
Unicode and UTF-8 support
Perl now uses UTF-8 as its internal representation for character strings. The utf8 and bytes pragmas are used to control this support in the current lexical scope. See perlunicode, utf8 and bytes for more information.
By that time, Unicode itself was up to its v3.0 release.
Lessons Learned: Always Throw the First One Out!
Even though Perl never got caught by the 16-bit trap (still!) afflicting so many others, its initial internal model for Unicode support quickly proved itself to be problematical one, so we rewrote all those internals for the v5.8 release on 2002-07-18.
Fortunately this internals rewrite affected very few actual end-users, because it was restricted to the internal C API. As far as end-programmer use, the crucial logical separation of an abstract code point (a 21-bit number) on one hand and the hidden underlying physical memory layout inherent to UTF-8 representation on the other was preserved, and so folks writing Perl scripts (rather than folks updating the perl compiler and interpreter) nearly never need to think of anything but the abstract characters (more like Go’s runes), not machine layouts in bits and bytes.
Reworking our initial representational model not only fixed a broken design, it also made it easy to role in updates from the Unicode Consortium; by Perl v5.8.1 of 2003-09-25 we supported v4.0 of the Unicode Character Database, and have tracked updates to the UCD and the many closely related annexes and technical reports (like TR#18 on Unicode Regular Expressions) fairly tightly ever since then.
Then and Now
And sure, there are still various “Unicode-y” things we don’t do it as well in Perl as we could. Examples that quickly come to mind include normalizing the identifiers for variables and function names through something like NFKC normalization, or making better guarantees about how Unicode in module names get represented externally in the filesystem.
Even so, I still find it easier to use Unicode in Perl than in any other programming language of its venerable vintage. After all, they all had to have Unicode support tacked on after the language was first created, and that’s always its own kind of nightmare. Many Unicode-related tasks that are hard in languages originally designed a long time ago but which like Perl and Python are still around today are sometimes easier in “recent” languages designed with abstract Unicode support built into their design from the git-go, as the saying runs. But that’s a topic for another day.
