Understanding UTF-8: How Unicode Characters Are Encoded
UTF-8 is the dominant character encoding on the web—over 98% of all web pages use it. It's a variable-width encoding that maps Unicode code points to between one and four bytes. Understanding how it works helps you debug encoding bugs, avoid security pitfalls, and write more efficient code.
The Encoding Rules
UTF-8 uses a clever prefix scheme so that any byte tells you exactly where you are in a multi-byte sequence:
- 1 byte (U+0000–U+007F): The high bit is 0. These cover all ASCII characters—the letter A (U+0041) encodes as the single byte
0x41. - 2 bytes (U+0080–U+07FF): First byte starts with
110, second with10. Covers Latin extended, Greek, Cyrillic, Hebrew, and Arabic. - 3 bytes (U+0800–U+FFFF): First byte starts with
1110. Covers most CJK characters and the rest of the BMP. - 4 bytes (U+10000–U+10FFFF): First byte starts with
11110. Used for emoji, historic scripts, and supplementary characters like 🔥 (U+1F525).
Why UTF-8 Won
UTF-8 is backward-compatible with ASCII—any ASCII document is also a valid UTF-8 document. This was crucial for adoption because enormous amounts of existing code, protocols, and data assumed ASCII. No conversion was needed for plain English text, and the overhead for other scripts was manageable.
Compare this to UTF-16, which uses at least two bytes for every character. A UTF-16 file containing only ASCII-range text would be twice the size of its UTF-8 equivalent, and it's not backward-compatible with ASCII. You can see how characters compare at the character comparison tool.
Common UTF-8 Pitfalls
The most frequent mistake is treating a string's byte count as its character count. In PHP, strlen("€") returns 3 because the euro sign U+20AC is encoded as three bytes in UTF-8. Use mb_strlen() for Unicode-aware length calculations. Similarly, taking a byte-level substring of a UTF-8 string can split a multi-byte sequence and produce invalid data. Always use multibyte-aware string functions in any language you work in.