Unicode in URLs: Internationalized Domain Names Explained
The Domain Name System was designed for ASCII. Internationalised Domain Names (IDNs) extend it to support non-ASCII characters, allowing domain names in Arabic, Chinese, Cyrillic, Devanagari, and dozens of other scripts. The mechanism that makes this possible—Punycode—is one of the more elegant engineering solutions in the web's history.
Punycode: ASCII-Compatible Encoding
Punycode (RFC 3492) encodes arbitrary Unicode strings as ASCII. Each internationalised label (the part between dots) is converted to a Punycode string prefixed with xn--. For example:
- münchen.de → xn--mnchen-3ya.de
- 日本語.jp → xn--wgv71a309e.jp
- παράδειγμα.δοκιμή → xn--hxajbheg2az3al.xn--jxalpdlp
DNS infrastructure sees only the ASCII Punycode form; browsers display the Unicode form to users when it's deemed safe. Use CharLookup search to look up characters used in IDN labels.
The Homograph Attack (Revisited)
IDN homograph attacks exploit the fact that characters from different scripts look identical. The Cyrillic а (U+0430) and Latin a (U+0061) are indistinguishable in most fonts, but they're different code points and can produce different Punycode. Browsers implement security heuristics: mixed-script domains (Latin + Cyrillic in the same label) are shown in Punycode form to alert users. The confusable characters article covers the underlying Unicode security mechanisms.
Percent-Encoding in URLs (IRIs)
URLs can only contain ASCII characters. Non-ASCII characters in URL paths and query strings must be percent-encoded: the UTF-8 bytes of the character, each preceded by %. The euro sign € (U+20AC) encodes as %E2%82%AC (its three UTF-8 bytes). An Internationalized Resource Identifier (IRI, RFC 3987) is the formal specification that allows Unicode characters in URLs, with percent-encoding applied automatically when the IRI is converted to a URI for transmission.
Most modern HTTP clients handle this automatically. The important thing to know is that a URL containing %E2%82%AC and one containing the literal € character refer to the same resource—they're just two representations of the same IRI.