Article Community improvements enabled

Protected Mode In OSDEV

Daniel McCarthy published 3 hours ago 15 min read 11 views
New post
Daniel McCarthy remains the original author. Improvements are attributed to their editors and reviewed by the original author before publication.

Protected Mode

Protected Mode is an operating mode available on x86 processors beginning with the Intel 80286. The 80286 introduced a 16-bit form of Protected Mode, while the 80386 expanded it into the 32-bit environment commonly associated with classic IA-32 operating systems.

Unlike Real Mode, Protected Mode gives an operating system much more control over memory, privilege levels, hardware access, and program isolation. On the 80386 and newer processors, a normal 32-bit Protected Mode environment can use a 4 GiB linear address space.

A PC booting through the traditional BIOS normally starts your bootloader in 16-bit Real Mode. If you want to run a 32-bit kernel, your bootloader therefore has to switch the processor into Protected Mode itself.

Important: Traditional BIOS interrupt services such as int 0x10 and int 0x13 are designed for Real Mode. You generally cannot continue calling them normally after entering Protected Mode.


Why Use Protected Mode?

Protected Mode provides facilities that are missing or heavily restricted in Real Mode:

  • 32-bit instructions and registers on the 80386 and newer
  • Up to 4 GiB of linear address space
  • Memory segmentation with permissions
  • CPU privilege levels, usually called rings
  • Separation between kernel and user-space code
  • Controlled hardware I/O access
  • Support for paging and virtual memory
  • Better support for multitasking and process isolation

These features make Protected Mode a useful environment for learning how a traditional 32-bit x86 operating system works.


Real Mode vs Protected Mode

In Real Mode, segment registers such as DS, ES, SS, and CS take part directly in calculating an address.

For example, a Real Mode address is approximately:

physical address = segment * 16 + offset

So if:

DS = 0x1000
offset = 0x0020

then:

0x1000 * 16 = 0x10000
0x10000 + 0x20 = 0x10020

Protected Mode changes this.

Segment registers no longer simply contain the upper part of an address. Instead, they contain segment selectors.

A selector points to a descriptor inside a descriptor table, normally the Global Descriptor Table, or GDT.

A GDT descriptor tells the processor things such as:

  • Where the segment begins
  • How large it is
  • Whether it contains code or data
  • Whether data can be written
  • Whether code can be executed
  • Which privilege level can use it
  • Whether it is a 16-bit or 32-bit segment
  • Whether the descriptor is present

This gives the processor a way to validate memory access.


What We Need to Do

A basic Real Mode to Protected Mode transition looks like this:

  1. Start in 16-bit Real Mode.
  2. Disable interrupts.
  3. Enable the A20 address line.
  4. Create a GDT.
  5. Load the GDT with LGDT.
  6. Set the PE bit in CR0.
  7. Perform a far jump into the 32-bit code segment.
  8. Load the data segment registers.
  9. Create a 32-bit stack.
  10. Continue executing 32-bit code.

We will go through these steps individually and then put everything together into one working example.


Step 1: Start in Real Mode

A BIOS boot sector is normally loaded at physical address 0x7C00.

When using NASM, a very small boot sector often begins like this:

[ORG 0x7C00]
[BITS 16]

start:
    cli

    xor ax, ax
    mov ds, ax
    mov es, ax
    mov ss, ax
    mov sp, 0x7C00

[BITS 16] tells NASM to generate 16-bit code.

We disable interrupts with:

cli

because we do not want an interrupt to occur in the middle of changing the CPU's execution environment.

The stack is also initialized so that instructions such as push, pop, and call have somewhere valid to store data.


Step 2: Enable the A20 Line

What Is A20?

Very old x86 processors could only address 1 MiB of memory.

For compatibility with old software, later PCs introduced a mechanism that could force addresses above 1 MiB to wrap around.

That mechanism is associated with the A20 address line.

A modern operating system normally enables A20 before using memory above the first megabyte.

There are several ways to enable it.

One relatively simple method on many machines is the Fast A20 Gate through I/O port 0x92.

Example:

enable_a20:
    in al, 0x92

    test al, 0x02
    jnz .already_enabled

    or al, 0x02
    and al, 0xFE
    out 0x92, al

.already_enabled:
    ret

You could call it with:

call enable_a20

The important bit here is bit 1:

bit 1 = A20 enable

We also clear bit 0 because on some systems it is associated with a fast reset signal.

The Fast A20 Gate is convenient for learning, but a production-quality bootloader should handle machines where this method is unavailable and should ideally verify that A20 really became enabled.


Step 3: Create a Global Descriptor Table

Before the CPU can safely use Protected Mode segments, it needs a Global Descriptor Table.

For a very small kernel, we can begin with only three entries:

Index Selector Purpose 0 0x00 Null descriptor 1 0x08 Kernel code 2 0x10 Kernel data

The first descriptor must be a null descriptor.

The second will describe our 32-bit code segment.

The third will describe our 32-bit data segment.


Understanding a GDT Descriptor

Each normal GDT descriptor is 8 bytes long.

Conceptually, it contains:

63                              32 31                               0
+--------------------------------+----------------------------------+
| Base / Flags / Limit           | Base / Access / Limit           |
+--------------------------------+----------------------------------+

For a beginner kernel, we usually create a flat memory model.

That means:

base  = 0x00000000
limit = 0xFFFFFFFF

Both the code and data descriptors therefore cover the full 32-bit linear address space.

The descriptors still matter because they describe permissions and CPU behavior even when their bases are both zero.


Example GDT in NASM

Here is a complete minimal GDT:

gdt_start:

gdt_null:
    dq 0x0000000000000000

gdt_code:
    dw 0xFFFF
    dw 0x0000
    db 0x00
    db 10011010b
    db 11001111b
    db 0x00

gdt_data:
    dw 0xFFFF
    dw 0x0000
    db 0x00
    db 10010010b
    db 11001111b
    db 0x00

gdt_end:

Let's break this down.


The Null Descriptor

gdt_null:
    dq 0

The first GDT entry is intentionally invalid.

Selector 0x00 refers to this descriptor.

Trying to use the null selector as a normal code or data segment will result in a processor exception.


The Code Descriptor

Our code descriptor is:

gdt_code:
    dw 0xFFFF
    dw 0x0000
    db 0x00
    db 10011010b
    db 11001111b
    db 0x00

The segment base is zero.

The limit is configured to cover the 4 GiB address space when 4 KiB granularity is enabled.

The access byte is:

10011010b

which represents a present, Ring 0, executable, readable code segment.

A simplified interpretation is:

1 00 1 1 0 1 0
| |  | | | | |
| |  | | | | +-- Accessed
| |  | | | +---- Readable
| |  | | +------ Conforming
| |  | +-------- Executable
| |  +---------- Code/data descriptor
| +------------- Privilege level 0
+--------------- Present

For the first kernel code segment, you do not need to memorize every bit immediately. The important idea is that this descriptor tells the CPU:

"This is executable Ring 0 code, and it is a 32-bit segment."


The Data Descriptor

The data descriptor is almost the same:

gdt_data:
    dw 0xFFFF
    dw 0x0000
    db 0x00
    db 10010010b
    db 11001111b
    db 0x00

The access byte is:

10010010b

This creates a present Ring 0 data segment that can be read and written.


Why Are the Selectors 0x08 and 0x10?

A GDT entry is 8 bytes long.

The null descriptor begins at offset:

0 * 8 = 0x00

The code descriptor begins at:

1 * 8 = 0x08

The data descriptor begins at:

2 * 8 = 0x10

So we can define:

CODE_SEG equ gdt_code - gdt_start
DATA_SEG equ gdt_data - gdt_start

NASM will calculate:

CODE_SEG = 0x08
DATA_SEG = 0x10

This is nicer than scattering magic numbers throughout the bootloader.


Step 4: Create the GDT Pointer

LGDT does not take the address and size as separate operands.

Instead, it reads a small structure containing both.

We can create it like this:

gdt_descriptor:
    dw gdt_end - gdt_start - 1
    dd gdt_start

The first field is the GDT size minus one:

dw gdt_end - gdt_start - 1

The second is the address of the GDT:

dd gdt_start

We can then load it with:

lgdt [gdt_descriptor]

After this instruction, the CPU knows where our GDT is.

Loading the GDT does not enter Protected Mode by itself.


Step 5: Set the PE Bit in CR0

The CR0 register is one of the x86 control registers.

Bit 0 is called PE, meaning Protection Enable.

To set it:

mov eax, cr0
or eax, 0x1
mov cr0, eax

The important operation is:

CR0 = CR0 | 1

We use OR instead of simply replacing CR0 because the register contains other important control bits that should be preserved.

After:

mov cr0, eax

the PE bit is enabled.

But we are not finished.


Step 6: Perform a Far Jump

Immediately after enabling Protected Mode, we perform a far jump:

jmp CODE_SEG:protected_mode_entry

A normal jump changes the instruction pointer.

A far jump changes both:

CS
EIP

CODE_SEG is our GDT code selector:

0x08

The jump therefore loads the Protected Mode code descriptor into CS and begins executing at protected_mode_entry.


Step 7: Tell NASM We Are Now Writing 32-bit Code

The label after the far jump should be assembled as 32-bit code:

[BITS 32]

protected_mode_entry:

[BITS 32] does not switch the CPU into Protected Mode.

It only tells NASM:

"Encode the following instructions using 32-bit defaults."

The actual CPU mode change came from setting CR0.PE and loading the Protected Mode code segment.


Step 8: Reload the Data Segment Registers

The old segment-register values were created for Real Mode.

We should now load our Protected Mode data selector:

mov ax, DATA_SEG

mov ds, ax
mov es, ax
mov fs, ax
mov gs, ax
mov ss, ax

Here:

DATA_SEG = 0x10

Every one of these segment registers now refers to our Ring 0 data descriptor in the GDT.


Step 9: Set Up a 32-bit Stack

Now create a suitable stack for the 32-bit kernel:

mov esp, 0x90000

The stack grows downward.

So after this instruction:

ESP = 0x00090000

a push eax would reduce ESP and store the value below that address.

For a tiny experimental bootloader, a hard-coded location can be enough.

A larger kernel should deliberately reserve and manage its kernel stack rather than choosing an address blindly.


Full Real Mode to Protected Mode Example

The following is a complete minimal boot sector demonstrating the transition.

It:

  • Starts in Real Mode
  • Creates a stack
  • Enables A20 with the Fast A20 Gate
  • Loads a GDT
  • Sets CR0.PE
  • Far jumps into a 32-bit code segment
  • Loads the data selectors
  • Sets up a 32-bit stack
  • Writes OK directly to VGA text memory
[ORG 0x7C00]
[BITS 16]

CODE_SEG equ gdt_code - gdt_start
DATA_SEG equ gdt_data - gdt_start

start:
    cli

    ; ---------------------------------------------------------
    ; Set up basic Real Mode segments and stack
    ; ---------------------------------------------------------

    xor ax, ax
    mov ds, ax
    mov es, ax
    mov ss, ax
    mov sp, 0x7C00

    ; ---------------------------------------------------------
    ; Enable A20 using the Fast A20 Gate
    ; ---------------------------------------------------------

    in al, 0x92
    test al, 0x02
    jnz .a20_enabled

    or al, 0x02
    and al, 0xFE
    out 0x92, al

.a20_enabled:

    ; ---------------------------------------------------------
    ; Load our Global Descriptor Table
    ; ---------------------------------------------------------

    lgdt [gdt_descriptor]

    ; ---------------------------------------------------------
    ; Enable Protected Mode
    ; ---------------------------------------------------------

    mov eax, cr0
    or eax, 0x01
    mov cr0, eax

    ; ---------------------------------------------------------
    ; Load the new code segment with a far jump
    ; ---------------------------------------------------------

    jmp CODE_SEG:protected_mode_entry


; =============================================================
; Global Descriptor Table
; =============================================================

gdt_start:

gdt_null:
    dq 0x0000000000000000

gdt_code:
    ; Base:  0x00000000
    ; Limit: 0xFFFFFFFF
    ; Ring 0, executable, readable, 32-bit

    dw 0xFFFF
    dw 0x0000
    db 0x00
    db 10011010b
    db 11001111b
    db 0x00

gdt_data:
    ; Base:  0x00000000
    ; Limit: 0xFFFFFFFF
    ; Ring 0, readable, writable, 32-bit

    dw 0xFFFF
    dw 0x0000
    db 0x00
    db 10010010b
    db 11001111b
    db 0x00

gdt_end:


gdt_descriptor:
    dw gdt_end - gdt_start - 1
    dd gdt_start


; =============================================================
; 32-bit Protected Mode code
; =============================================================

[BITS 32]

protected_mode_entry:

    ; Load our data selector into every data segment register.

    mov ax, DATA_SEG
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax

    ; Create a 32-bit stack.

    mov esp, 0x90000

    ; ---------------------------------------------------------
    ; Write "OK" directly to VGA text memory.
    ;
    ; VGA text memory begins at 0xB8000.
    ;
    ; Each character uses two bytes:
    ;   byte 0 = ASCII character
    ;   byte 1 = colour attribute
    ; ---------------------------------------------------------

    mov byte [0xB8000], 'O'
    mov byte [0xB8001], 0x0F

    mov byte [0xB8002], 'K'
    mov byte [0xB8003], 0x0F

.hang:
    cli
    hlt
    jmp .hang


; =============================================================
; Boot signature
; =============================================================

times 510 - ($ - $$) db 0
dw 0xAA55

Building the Example

Save the file as:

boot.asm

Assemble it with NASM:

nasm -f bin boot.asm -o boot.bin

This produces a raw 512-byte boot sector named:

boot.bin

You can run it with QEMU:

qemu-system-i386 -drive format=raw,file=boot.bin

If everything works, the screen should display:

OK

At that point, the CPU is executing your own 32-bit Protected Mode code.


What Actually Happened?

The most important part of the example is this sequence:

lgdt [gdt_descriptor]

mov eax, cr0
or eax, 1
mov cr0, eax

jmp CODE_SEG:protected_mode_entry

In plain English:

lgdt
    ↓
Tell the CPU where our segment definitions are.

CR0.PE = 1
    ↓
Enable Protected Mode.

far jump
    ↓
Load the new Protected Mode code segment into CS.

protected_mode_entry
    ↓
Begin normal 32-bit execution.

Then:

mov ax, DATA_SEG
mov ds, ax
mov es, ax
mov fs, ax
mov gs, ax
mov ss, ax

loads our data descriptor into the remaining segment registers.

Finally:

mov esp, 0x90000

gives the 32-bit code a stack.


Calling a 32-bit Kernel

Eventually, you probably do not want your entire kernel written inside the boot sector.

Your Protected Mode assembly entry point could instead prepare the CPU and call a kernel function.

For example:

[BITS 32]

extern kernel_main

protected_mode_entry:
    mov ax, DATA_SEG

    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax

    mov esp, 0x90000

    call kernel_main

.hang:
    cli
    hlt
    jmp .hang

A simple C kernel entry function might look like:

#include <stdint.h>

void kernel_main(void)
{
    volatile uint16_t* video = (volatile uint16_t*)0xB8000;

    video[0] = ((uint16_t)0x0F << 8) | 'H';
    video[1] = ((uint16_t)0x0F << 8) | 'i';

    for (;;)
    {
        __asm__ volatile ("hlt");
    }
}

This example assumes that your bootloader has already loaded the kernel into memory and that your build system links the assembly and C code correctly.

Calling C therefore comes after solving several additional bootloader problems, including loading the kernel and creating an appropriate linker script.


What Changes After Entering Protected Mode?

Once Protected Mode is active, several Real Mode assumptions stop being true.


Segment Registers Now Contain Selectors

In Real Mode, a segment value contributes directly to an address.

In Protected Mode:

DS = 0x10

does not mean:

data segment base = 0x100

Instead, 0x10 selects a descriptor in the GDT.

That descriptor then determines the base address, limit, permissions, and other properties of the segment.


BIOS Interrupts Are Generally Unavailable

In Real Mode, printing a character using the BIOS might involve:

mov ah, 0x0E
mov al, 'A'
int 0x10

After entering Protected Mode, you generally cannot continue doing this.

That is why our example writes directly to:

0xB8000

instead:

mov byte [0xB8000], 'A'
mov byte [0xB8001], 0x0F

Bootloaders usually perform BIOS-related work before entering Protected Mode.

Examples include:

  • Reading sectors from a disk
  • Obtaining the system memory map
  • Choosing a video mode
  • Loading a kernel image

Interrupts Need an IDT

Protected Mode uses the Interrupt Descriptor Table, or IDT, to describe interrupt and exception handlers.

We used:

cli

before entering Protected Mode.

This prevents normal maskable interrupts from arriving before we have an IDT ready.

Later, your kernel will normally do something similar to:

lidt [idtr]

and only then eventually:

sti

to allow normal hardware interrupts again.

Do not simply execute sti immediately after entering Protected Mode unless your interrupt environment is ready.


Paging Is Not Automatically Enabled

Protected Mode and paging are different CPU features.

This:

mov eax, cr0
or eax, 1
mov cr0, eax

sets:

CR0.PE

It does not set:

CR0.PG

So you can run a 32-bit Protected Mode kernel without paging.

Later, when you learn virtual memory, you can build page tables and enable paging separately.


Privilege Rings

Protected Mode supports four CPU privilege levels:

Ring Typical use 0 Kernel 1 Rarely used 2 Rarely used 3 User applications

Ring 0 has the highest privilege.

Ring 3 has the lowest.

Most modern operating systems primarily use:

Ring 0 = kernel
Ring 3 = applications

The GDT, IDT, paging system, and other processor mechanisms help enforce the separation.


Common Mistakes

Invalid GDT

A malformed descriptor can cause an exception as soon as the CPU tries to load it.

If the exception itself cannot be handled because you do not yet have a valid IDT, the machine may reset due to a triple fault.


Incorrect GDT Pointer

This is wrong if the address or size does not describe your actual GDT:

lgdt [gdt_descriptor]

Make sure:

gdt_descriptor:
    dw gdt_end - gdt_start - 1
    dd gdt_start

actually refers to the table you created.


Wrong Selector

If the table is:

index 0 = null
index 1 = code
index 2 = data

then the normal Ring 0 selectors are:

code = 0x08
data = 0x10

Using the wrong selector can cause a General Protection Fault.


Forgetting the Far Jump

This is incomplete:

mov eax, cr0
or eax, 1
mov cr0, eax

You should then transfer control through the new code selector:

jmp CODE_SEG:protected_mode_entry

Forgetting to Reload DS, ES, SS, FS, and GS

After the far jump, reload your data-related segment registers:

mov ax, DATA_SEG
mov ds, ax
mov es, ax
mov fs, ax
mov gs, ax
mov ss, ax

Enabling Interrupts Too Early

This is dangerous immediately after switching modes:

sti

unless you already have a valid Protected Mode IDT and interrupt setup.

Keep interrupts disabled until your kernel is ready.


Calling BIOS Functions

A Real Mode BIOS call such as:

int 0x10

should not be treated as a normal Protected Mode API.

Do the BIOS work before switching modes, or write your own drivers after the switch.


Forgetting About A20

If A20 is disabled, addresses around and above 1 MiB can wrap unexpectedly.

Enable it before relying on high memory.


A Useful Mental Model

For a beginner, think of entering Protected Mode as giving the processor a new rulebook.

In Real Mode:

segment register
      ↓
helps directly calculate an address

In Protected Mode:

segment selector
      ↓
GDT descriptor
      ↓
base + limit + permissions + CPU rules

The transition is therefore not simply:

16-bit → 32-bit

It is also a transition from the old Real Mode segmentation system into a protected descriptor-based environment.


Recommended Boot Order

A simple BIOS kernel might eventually use this sequence:

BIOS loads boot sector
        ↓
16-bit Real Mode
        ↓
Set up Real Mode stack
        ↓
Use BIOS to load kernel
        ↓
Obtain memory information
        ↓
Enable A20
        ↓
Create GDT
        ↓
LGDT
        ↓
Set CR0.PE
        ↓
Far jump
        ↓
32-bit Protected Mode
        ↓
Load data selectors
        ↓
Create 32-bit stack
        ↓
Call kernel
        ↓
Create IDT
        ↓
Configure interrupts
        ↓
Optionally enable paging
        ↓
Continue kernel initialization

That is the basic path from a BIOS bootloader to a traditional 32-bit x86 kernel.


Related Topics

Once you understand this transition, the most useful topics to study next are:

  • Global Descriptor Table (GDT)
  • GDT descriptors
  • x86 segmentation
  • A20 line
  • Interrupt Descriptor Table (IDT)
  • CPU exceptions
  • Programmable Interrupt Controller (PIC)
  • x86 control registers
  • Paging
  • Virtual memory
  • Task State Segment (TSS)
  • Ring 3 / user mode
  • ELF kernel loading
  • Linker scripts
  • Long Mode

Further Reading

I recommend Daniel McCarthy's kernel development courses:

Discussion 0

No comments yet. Start a thoughtful discussion.

Join the discussion

You need an account to contribute.

Sign in