Article revision Version 1 of 1 Current

Decode UTF-8 Safely in C Before Lexing Source

Originally published by @nibblebits Nov 20, 2026 at 14:49
View current article
Initial version

Original article publication

This is the article as it appeared before any published revisions.

Snapshot

Article at version 1

Compilers & Toolchains

A compiler front end needs to track where a token came from. With UTF-8 source, one decoded value can occupy several bytes. If the lexer advances one byte for every decoded value, its next read and its diagnostics will eventually point to the wrong place.

The C11 decoder below returns one Unicode scalar value together with the number of bytes consumed. The caller keeps the original source buffer and advances its byte offset only after a successful decode.

Check the sequence before accepting it

UTF-8 uses one to four bytes per scalar. Continuation bytes have values from 80 to BF in hexadecimal. The leading byte determines the width, and four leading-byte cases impose a tighter range on the second byte. These ranges enforce the shortest encoding and exclude surrogate values and values beyond U+10FFFF. RFC 3629, sections 3–4

First byte Width Second-byte range when present
00–7F 1 —
C2–DF 2 80–BF
E0 3 A0–BF
E1–EC, EE–EF 3 80–BF
ED 3 80–9F
F0 4 90–BF
F1–F3 4 80–BF
F4 4 80–8F

All remaining bytes in a multibyte sequence use 80–BF. For example, E0 80 80 has continuation-byte shapes, but it fails the second-byte restriction. Accepting it would allow a longer encoding of a value that has a shorter form. Unicode Standard, section 3.9.3 and Table 3-7

The interface reports one of three outcomes.

Result Meaning Caller action
UTF8_OK One scalar decoded Advance by result.width
UTF8_NEED_MORE The available prefix could continue with more bytes Obtain more input, or report truncation at the end of the source
UTF8_INVALID The available bytes already violate a range Report an error at the current byte offset

Outputs change only on success. A failed call leaves the caller's offset and result object available for an error report.

Decode within a bounded byte span

Save this as utf8_decode.c. The input pointer must address the stated number of bytes, and the result pointer must be valid. The program requires eight-bit bytes, which the static assertion checks. The input uses explicit hexadecimal bytes so the demo does not depend on the compiler's source encoding.

#include <inttypes.h>
#include <limits.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>

_Static_assert(CHAR_BIT == 8, "This decoder requires eight-bit bytes");

enum Utf8Status { UTF8_OK, UTF8_NEED_MORE, UTF8_INVALID };

struct Decoded {
    uint32_t scalar;
    size_t width;
};

/* Input spans n bytes. Outputs change only on UTF8_OK. */
static enum Utf8Status decode_utf8(const unsigned char *bytes, size_t n,
                                   struct Decoded *result)
{
    if (n == 0) {
        return UTF8_NEED_MORE;
    }

    const unsigned first = bytes[0];
    size_t width;
    uint32_t scalar;
    if (first <= 0x7f) {
        width = 1;
        scalar = first;
    } else if (first >= 0xc2 && first <= 0xdf) {
        width = 2;
        scalar = first & 0x1f;
    } else if (first >= 0xe0 && first <= 0xef) {
        width = 3;
        scalar = first & 0x0f;
    } else if (first >= 0xf0 && first <= 0xf4) {
        width = 4;
        scalar = first & 0x07;
    } else {
        return UTF8_INVALID;
    }

    for (size_t i = 1; i < width; ++i) {
        if (i >= n) {
            return UTF8_NEED_MORE;
        }
        unsigned low = 0x80;
        unsigned high = 0xbf;
        if (i == 1) {
            if (first == 0xe0) low = 0xa0;
            if (first == 0xed) high = 0x9f;
            if (first == 0xf0) low = 0x90;
            if (first == 0xf4) high = 0x8f;
        }
        const unsigned next = bytes[i];
        if (next < low || next > high) {
            return UTF8_INVALID;
        }
        scalar = (scalar << 6) | (next & 0x3f);
    }

    *result = (struct Decoded){.scalar = scalar, .width = width};
    return UTF8_OK;
}

int main(void)
{
    const unsigned char source[] = {
        0x41, 0xc2, 0xa2, 0xe2, 0x82, 0xac, 0xf0, 0x9f, 0x98, 0x80, 0x00
    };
    size_t offset = 0;
    while (offset < sizeof source) {
        struct Decoded decoded;
        const enum Utf8Status status =
            decode_utf8(source + offset, sizeof source - offset, &decoded);
        if (status != UTF8_OK) {
            fprintf(stderr, "Invalid or unfinished UTF-8 at byte %zu\n", offset);
            return 1;
        }
        printf("byte %zu -> U+%04" PRIX32 " (%zu bytes)\n",
               offset, decoded.scalar, decoded.width);
        offset += decoded.width;
    }
    return 0;
}

Each continuation-byte read follows a length check. The decoder validates that byte before shifting its six payload bits into the scalar. The special second-byte limits are checked as soon as that byte is available. Thus E0 80 reports invalid input immediately, while E2 82 reports that another byte is needed.

Preserve the original byte offsets

I built and ran the program with MinGW-w64 GCC targeting x86_64-w64-mingw32. These commands run in PowerShell with GCC on the path.

gcc -std=c11 -O2 -Wall -Wextra -Wconversion -Werror -pedantic utf8_decode.c -o utf8_decode.exe
.\utf8_decode.exe

The output shows where each decoded scalar begins.

byte 0 -> U+0041 (1 bytes)
byte 1 -> U+00A2 (2 bytes)
byte 3 -> U+20AC (3 bytes)
byte 6 -> U+1F600 (4 bytes)
byte 10 -> U+0000 (1 bytes)

The offset advances by 1, 2, 3, and 4 for the first four values. The final byte is decoded as U+0000, and the loop stops when its offset reaches the supplied buffer length. This distinguishes an encoded zero from the absence of another byte. The lexer can then apply the source language's rules for embedded zero, identifiers, and whitespace.

A decoder also accepts well-formed encodings of unassigned scalar values and noncharacters. Whether a scalar is allowed in a token depends on the language and its Unicode policy. Unicode Standard, section 3.9

Exercise the boundaries

The code passed 20 boundary and interface checks. They covered stray continuation bytes, forbidden leading bytes, longer-than-needed encodings, surrogate encodings, values above the Unicode limit, truncated valid prefixes, and invalid short prefixes. The checks also verified that a failure preserves the output object and that a successful call consumes only the first scalar when more bytes follow.

For an independent encoding check, I used Windows' UTF-8 encoder to generate bytes for each of the 1,112,064 scalar values. The C decoder recovered the expected value and width for all of them. That includes U+0000, the boundaries between encoding widths, and U+10FFFF.

In a lexer, retain the byte offset from before the call as the start of the decoded value. Advance it after success and attach byte ranges to tokens. This gives later diagnostics a location in the original source file while the lexer works with decoded scalars.

My compiler course develops the surrounding front end, including loading source, lexical analysis, and building tokens. Creating a C Compiler From Scratch Video Course!

compilersc11utf-8unicodelexical-analysis