Everything you need to know about Punycode

Language revolution in the address bar

From its inception, the internet was designed to be a global network; however, there was one notable limitation: domain names could only use Latin characters. This alphabet barrier was not intentional; it was a consequence of technical design decisions made at the time when the global standards for digital representations of the characters were still in development. The basic building block of the entire system, the Domain Name System (DNS), was conceived in 1983 and worked exclusively with a character subset of ASCII, allowing only lowercase Latin characters (a–z), numbers (0–9), and the comma¹.

That technical choice, which initially provided functionality and universality in the early phase of development, became a significant obstacle over time. Though the content of web pages and emails could be in any language, the domain name (part of the URL) still had to be written in the Latin alphabet. Such an „alphabet barrier" contributed to the so-called „digital divide" in particular in countries, where languages which does not use Latin, such as English, arent widely used. For such a user, it was often easier to memorize a chain of numbers (IP address) rather than a series of unknown glyphs². Ironically, what was meant to be easy to remember for some became a cultural barrier for the rest of the world. The essential international standard for non-Latin languages, Unicode, was initiated four years later, in 1991, which illustrates how design choices from an early phase of development can have unintended, global impacts³. The Punycode development begins with an effort to bridge such a divide.

A smart solution

Efforts about „internationalization of domain names" started to appear since the mid-1990s, though after years of debates and a lot of competing proposals, a standard solution⁴⁵.In March 2003, the IETF (Internet Engineering Task Force) approved RFC 3492, a standard that described the Punycode algorithm ⁶. Its author, Adam Costello, designed it as a neat and efficient solution, which was able to losslessly and reversibly transform any Unicode string to the plain ASCII subset.

Punycode is not sa brand new alghoritm. Rather, it is a specific implementation of a more generic algorithm named Bootstring, which enables the representation of any string from a larger character set (Unicode in our case) using a smaller set of characters (a subset of ASCII in our case)⁷. Such a concept was designed to be universal and functional across most scripts, while striving to be self-optimized and adapt to a character set in a particular string[^punycode-optimization].

##Origin of the name: How is it "puny"?

The name is a catchy word play, which rhymes with Unicode and refers to three meanings of how the alghotitm is „puny“:

• Small subset of characters: only lowercase letters, numbers and comma is allowed¹.

• Short encoded version: encoded strings arent much longer than original. This is not only elegant, but also important, as DNS limits the length of the domain tag to 63 chars⁸.

• implementation is small

The power of Punycode lies in its „puniness“ — its simplicity. It managed to achieve maximum significance (universal for all characters, therefore applicable to all languages) with minimal requirements.

The alghoritm: Whats behind the xn--?

The Punycode is denoted by a special prefix: „xn--", so everything encoded, such as a domain name, starts with it. The prefix is defined within the ACE (ASCII-Compatible Encoding), in standards IDNA 2003 and IDNA 2008⁴⁵.

The encoding process has multiple phases:

1. ASCII char separation: All ASCII characters (those that don't need to be encoded) from the input string are copied to the start of the output string.

2. Add the hyphen (minus) separator -:

If there were any ASCII characters, the separator "-" is added after them (e.g., for "čáslav", the ASCII characters will be followed by a trailing hyphen like "slav-")⁹.

We have to realize that the hyphen itself is an ASCII character. Thus, the hyphens can be part of the input string, and if they are present, they will be appended to the output as other characters. That does not create any ambiguity, as the dash added last is the one that was added, as it denotes the end of the ASCII characters.

3. Encoding non‑ASCII chars: The characters beyond ASCII are encoded using the Bootstring algorithm with parameters for Punycode, resulting in a sequence of a‑z and 0‑9⁹.

4. Adding the ACE prefix xn--: In domain names, the Punycode‑encoded label is prefixed with xn-- to denote ACE (ASCII‑Compatible Encoding)¹⁰.

So, for example, if we want to encode the string "čáslav" (a Czech town):

1. ASCII char separation:

From the "čáslav", the ASCII chars "slav" are placed at the start of the output.

1. Add the hyphen (minus) separator -:

As the input contains both ASCII and non-ASCII chars, a hyphen is then added, so the output is slav-

3. Encoding non‑ASCII chars:

The č and á chars are encoded using the Bootstring algorithm into 4na7x and appended to the end of the output, so the resulting Punycode output is slav-4na7x

4. Adding the ACE prefix xn--:

To denote the punycode encoded text in the domain name, we need to prepend the xn-- prefix. It is called ACE (ASCII Compatible Encoding).

So the string we can use in our DNS (Domain Name System setup) is xn--slav-4na7x

Examples

The table below shows how different types of input are transformed generated using tr46 UTS #46 processing, by a Punycode lib tr46