Character Encoding: ASCII, UTF-8, and Why Text Turns to Gibberish

Text on a computer is stored as bytes, and a character encoding is the rule that maps those bytes back to letters. Most of the time this is invisible, until a name shows up as café instead of café, or a pasted document fills up with replacement glyphs where emoji used to be, and then it is the only thing that matters. Mojibake — garbled text — is almost always a mismatch between the encoding used to write the bytes and the encoding used to read them. Understanding the small family of encodings behind that mismatch explains the gibberish and tells you how to fix it.

ASCII: the 128-character foundation

ASCII is a 7-bit encoding that assigns a byte value to each of 128 characters: the English alphabet in upper and lower case, the digits, common punctuation, and a set of control characters inherited from teletype machines. The letter A is byte 65, a is 97, the space is 32, the newline is 10. Every ASCII character fits in seven bits, which is why the eighth bit of every byte was free for other uses — and that freedom is exactly where the trouble started.

'A'   = 65  = 0x41
'a'   = 97  = 0x61
' '   = 32  = 0x20
'\n'  = 10  = 0x0A

ASCII is sufficient for English and almost nothing else. Accented Latin characters, Cyrillic, Arabic, Chinese, Japanese, Korean, emoji — none of them exist in ASCII. As computing spread beyond English, every region filled the unused upper half of the byte differently, producing a sprawl of incompatible code pages: Western European text in one, Cyrillic in another, and no consistent way to mix them. A document written on one machine looked like garbage on another, because the same byte mapped to different letters depending on the local code page.

Unicode and UTF-8: one set to encode them all

Unicode solves the code-page problem by assigning every character in every script a single number, called a code point. The letter A is U+0041, the Hebrew aleph is U+05D0, the rocket emoji is U+1F680. UTF-8 is the dominant way to encode those code points as bytes, and its design is the reason it won.

UTF-8 is variable-width: it uses one byte for the ASCII range and between two and four bytes for everything else. The first 128 code points in UTF-8 are byte-for-byte identical to ASCII, which means a pure-ASCII document is already valid UTF-8 with no conversion. That single property is why UTF-8 is backward compatible with decades of existing text and why it displaced the older encodings so thoroughly — adopting it cost nothing for ASCII-only content and gained compatibility with every script on Earth.

'A'     = 0x41                        (1 byte, ASCII)
'é'     = 0xC3 0xA9                   (2 bytes)
'中'    = 0xE4 0xB8 0xAD              (3 bytes)
'🚀'    = 0xF0 0x9F 0x9A 0x80          (4 bytes)

UTF-8 is now the default encoding for HTML, JSON, XML, source code, modern operating system file names, and the overwhelming majority of network protocols. When in doubt, use UTF-8; the cases where you genuinely need something else are rare and specialized.

Why text turns to gibberish

Mojibake is what happens when bytes written under one encoding are read under another. The classic case is UTF-8 text decoded as Latin-1 or Windows-1252: each UTF-8 byte above 127 gets interpreted as its own character, so é — encoded as the two bytes 0xC3 and 0xA9 — shows up as é. The reverse direction produces different but equally ugly results. A third variant is the replacement character, a glyph a decoder emits when it encounters a byte sequence that is invalid in the target encoding and it gives up rather than guess.

The fixes are uniform in principle and fiddly in practice. Identify the encoding the bytes were written in, decode them with that encoding, and re-encode or display them in UTF-8. The hard part is identifying the original encoding, because nothing in the bytes themselves labels them — a heuristic or a charset declaration in a header or document is all you have. Web pages declare their encoding through a Content-Type header or a meta charset declaration in the document head, and getting that declaration wrong is the cause of most browser mojibake.

Committing to UTF-8 everywhere

The robust answer is to use UTF-8 at every layer and never let another encoding in. Set it on database connections and table columns, on HTTP responses and request parsing, on file reads and writes, on the source files themselves, and on the HTML pages that render them. Most encoding bugs are really consistency bugs — one component assumed UTF-8 while another defaulted to Latin-1 — and the fix is usually a one-line configuration change rather than a code change.

A few practical tools help when you are stuck. If text arrives base64-encoded over a transport that mangles binary, decode it with the Base64 Encode/Decode tool and confirm the bytes are intact before chasing character issues. If text is travelling through a URL, percent-encode it as UTF-8 with the URL Encode/Decode tool so non-ASCII characters survive the trip. And if you are normalizing text for comparison or matching, run it through the Case Converter after fixing the encoding, because case folding on mojibake produces nonsense. Encoding stops being mysterious the moment you treat it as a contract between writer and reader — and UTF-8 is the contract the entire web has already signed.