Article Community improvements enabled

Getting Started with Operating System Development

Daniel McCarthy published 6 hours ago 15 min read 24 views
New post
Daniel McCarthy remains the original author. Improvements are attributed to their editors and reviewed by the original author before publication.

Getting Started with Operating System Development

Writing an operating system is one of the most demanding — and rewarding — programming projects you can take on.

Unlike normal application development, there is very little underneath you. There is no operating system providing files, processes, threads, memory management, timers, drivers, or a standard terminal unless you build those facilities yourself.

That is exactly what makes OS development interesting.

When your kernel prints its first message, handles its first interrupt, allocates its first page of memory, launches its first process, or reads its first file from disk, you are seeing something work because you built the machinery that made it possible.

There is no single correct route to building an operating system. The important thing is to start with a small, understandable system, get it working, and expand it one subsystem at a time.


The Reality of OS Development

Operating system development is not a quick project.

A useful kernel can easily become a multi-year undertaking. Even seemingly simple features can require knowledge from several different areas of computer science at once.

You may need to understand:

  • CPU architecture
  • Assembly language
  • C or another systems programming language
  • Executable formats such as ELF
  • Linkers and linker scripts
  • Interrupts and exceptions
  • Virtual and physical memory
  • Concurrency
  • Filesystems
  • Storage devices
  • PCI and other hardware buses
  • Networking
  • Compilers and ABIs
  • Debugging without the safety net of a normal operating system

Progress can sometimes feel slow because low-level bugs are unforgiving. A bad pointer may crash the entire machine. A broken page table may cause a triple fault. An incorrectly configured interrupt controller may make the system appear to freeze.

That is normal.

The goal is not to understand every topic before writing your first line of kernel code. The goal is to know enough to begin, then learn each new subject as your kernel reaches it.


Knowledge You Should Have First

You do not need to be an expert in everything, but you should already be comfortable programming before beginning serious kernel development.

C

C remains one of the most common languages for kernel development.

You should understand:

  • Pointers
  • Pointer arithmetic
  • Arrays
  • Structures
  • Unions
  • Bitwise operations
  • Integer types and integer sizes
  • Function pointers
  • Memory layout
  • Stack and heap concepts
  • Header files
  • Separate compilation
  • Undefined behaviour
  • volatile
  • Basic compiler optimisation concepts

Kernel programming exposes weaknesses in your understanding of C very quickly. There is no runtime environment to protect you from incorrect assumptions.

Assembly Language

You do not need to write your entire kernel in assembly, but you should be able to read and write small amounts of it.

You should understand concepts such as:

  • Registers
  • The stack
  • Calling conventions
  • push and pop
  • Jumps and conditional branches
  • Function calls and returns
  • CPU flags
  • Interrupt entry and return
  • Privileged instructions

For x86 or x86-64 development, understanding the processor's execution modes and control registers becomes especially important.

Computer Architecture

Before going too far, become familiar with:

  • How a CPU executes instructions
  • Memory addressing
  • Endianness
  • Privilege levels
  • Interrupts and exceptions
  • Memory-mapped hardware
  • I/O ports where applicable
  • Virtual memory
  • Caches
  • Multiprocessor systems

You do not need to memorise an entire processor manual. You do need to become comfortable reading architecture documentation when your kernel requires it.


Decide What You Are Trying to Build

Before writing thousands of lines of code, decide what you want from the project.

Possible goals include:

  • Learning how computers work at a low level
  • Learning kernel development
  • Experimenting with a new kernel architecture
  • Building a Unix-like operating system
  • Building an embedded operating system
  • Researching scheduling or memory-management ideas
  • Creating a small educational kernel
  • Building an operating system capable of running user programs
  • Eventually creating a self-hosting operating system

Your goal affects almost every technical decision you make.

For example, a small educational kernel does not need the same filesystem, compatibility layer, driver ecosystem, or security model as a general-purpose desktop operating system.

It is perfectly acceptable for your first goal to be:

Boot a kernel, print text, and understand exactly how it works.

That is already a meaningful achievement.


Choose an Architecture

Most hobby operating-system developers begin with an architecture that is well documented and easy to emulate.

Common choices include:

  • x86-64 — extremely well documented, widely emulated, and supported by many bootloaders and development tools.
  • x86 (32-bit) — historically popular for beginner kernels and still useful for learning protected mode and classic PC architecture.
  • AArch64 — a strong choice if you are interested in modern ARM systems.
  • RISC-V — a clean and increasingly popular architecture for education and experimentation.

For a first desktop-style hobby kernel, x86-64 is usually a practical choice because documentation, emulators, debuggers, bootloaders, and examples are widely available.

Do not try to support several architectures at the beginning. Get one architecture working first.


Set Up a Development Environment

You will need more than a compiler.

A typical OS development environment contains:

  • A compiler
  • A linker
  • An assembler
  • A build system
  • An emulator or virtual machine
  • A debugger
  • Binary-inspection tools
  • A text editor or IDE
  • Version control

Compiler and Binutils

For C-based kernels, GCC and Clang are both widely used.

GNU Binutils provides important tools such as:

  • ld
  • as
  • objdump
  • objcopy
  • readelf
  • nm

These tools become extremely useful when debugging linker problems, executable formats, symbols, and generated machine code.

Assembler

Depending on your project, you may use:

  • NASM
  • GNU Assembler (GAS)
  • LLVM's assembler

Assembly is commonly required for early boot code, interrupt stubs, context switching, and architecture-specific instructions.

Build System

Even a tiny kernel quickly grows beyond a few files.

Use a build system from the beginning.

Common choices include:

  • Make
  • CMake
  • Meson
  • Ninja
  • Custom build scripts

Make is still very common in small kernels because it is simple, widely available, and easy to inspect.

Emulator

You should normally develop inside an emulator before testing on physical hardware.

Popular choices include:

  • QEMU
  • Bochs
  • VirtualBox
  • VMware

QEMU is particularly useful because it supports many CPU architectures and integrates well with GDB.

An emulator allows you to reset a broken kernel instantly without repeatedly rebooting a real computer.

Debugger

Learn to use GDB or LLDB early.

Kernel debugging is much easier when you can:

  • Set breakpoints
  • Step through instructions
  • Inspect registers
  • Examine memory
  • View stack traces
  • Inspect symbols

Serial logging is also extremely valuable. A kernel that cannot display graphics may still be able to send diagnostic messages through a serial port.


Use a Cross-Compiler

A normal system compiler is configured to build programs for the operating system you are currently using.

That is not necessarily what you want when building a new kernel.

For example, your Linux compiler expects Linux conventions, libraries, startup files, and ABI assumptions. Your new operating system is not Linux.

A cross-compiler targets a platform independently of the host operating system.

Typical targets include names such as:

i686-elf
x86_64-elf

Using a cross-compiler helps prevent your host environment from accidentally leaking into your kernel build.

Another option is to use a carefully configured Clang/LLVM toolchain with an explicit freestanding target.

Whichever approach you choose, understand what your compiler is targeting.


Freestanding C Is Different

Normal C programs run in a hosted environment.

They expect an operating system and usually have access to a complete standard library.

A kernel begins in a freestanding environment.

You should not assume that functions such as these already exist:

printf();
malloc();
fopen();
exit();

If your kernel needs them, you eventually need to implement the underlying functionality yourself.

Compilers may also expect a small number of basic memory routines in freestanding environments, depending on what code they generate.

Common early implementations include:

memcpy();
memmove();
memset();
memcmp();

This is one reason kernel development teaches you so much about what normal programs take for granted.


Linux Development

Linux is an excellent host environment for kernel development because most low-level development tools are readily available.

A typical installation may include:

gcc
clang
make
nasm
binutils
gdb
qemu-system-x86
git
python3

Package names vary between distributions.

Ubuntu, Debian, Fedora, Arch Linux, openSUSE, and many other general-purpose distributions work perfectly well.

You do not need a special "OS development distribution."


Windows Development

Windows can also be used successfully.

Common approaches include:

  • WSL2
  • MSYS2
  • Native LLVM/Clang
  • Cross-compilers built for Windows

WSL2 is often convenient because it provides a Linux-style development environment while still allowing you to use Windows as your desktop operating system.

MSYS2 is another useful option when you want GNU-style tools directly within Windows.

Avoid assuming that a normal Visual Studio application configuration will automatically produce a suitable freestanding kernel. Kernel builds require control over compiler options, runtime assumptions, executable format, and linking.


macOS Development

macOS is also usable as a host.

Clang is available through Apple's developer tools, while additional packages can be installed using package managers such as Homebrew.

You will still need an appropriate target toolchain and an emulator such as QEMU.

As with Linux and Windows, the important distinction is between the host operating system and the target environment your kernel is being built for.


Use Version Control Immediately

Do not wait until the project becomes "big enough."

Create a Git repository on day one.

Kernel development regularly involves experiments that break previously working code. Version control allows you to compare changes, create branches, revert mistakes, and identify when a bug was introduced.

A basic workflow is enough:

git init
git add .
git commit -m "Initial kernel"

Commit whenever you reach a meaningful working state.

Good examples include:

Bootloader loads kernel successfully
Add basic terminal output
Install GDT
Add IDT and exception handlers
Enable physical page allocator
Enable paging
Add kernel heap
Add PIT timer
Add keyboard driver

Those checkpoints are extremely useful when later changes break the system.


Do Not Write Everything Yourself on Day One

A common beginner mistake is trying to write:

  • A bootloader
  • A kernel
  • A compiler
  • A C library
  • A filesystem
  • Drivers

all at the same time.

You can eventually build every component yourself if that is your goal, but doing everything simultaneously makes debugging unnecessarily difficult.

For a first kernel, using an existing bootloader is usually the better choice.

Modern options include bootloaders and boot protocols such as:

  • Limine
  • GRUB
  • UEFI

Let the bootloader get your kernel into memory so you can concentrate on kernel development.

You can always write your own bootloader later.


Your First Milestone: Boot a Kernel

Your first milestone should be extremely small.

For example:

  1. Build a kernel executable.
  2. Create a bootable disk image or ISO.
  3. Start it in QEMU.
  4. Reach your kernel entry point.
  5. Print a message.
  6. Halt safely.

Something as simple as:

Hello from my kernel!

is important.

It proves that several pieces of your toolchain are working together:

  • Compiler
  • Assembler
  • Linker
  • Linker script
  • Bootloader
  • Kernel executable
  • Emulator

Once that works, commit it to Git before changing anything else.


A Sensible Kernel Development Roadmap

There is no universal order, but the following progression works well for many hobby kernels.

Stage 1 — Boot and Output

Build enough infrastructure to:

  • Boot the kernel
  • Establish a known CPU state
  • Print diagnostic information
  • Halt or panic safely

Stage 2 — CPU Tables and Exceptions

On x86/x86-64, learn about:

  • GDT
  • IDT
  • CPU exceptions
  • Interrupt handlers

Make sure faults produce useful debugging information instead of silently resetting the machine.

Stage 3 — Memory Management

Implement:

  • Physical memory detection
  • Physical page allocation
  • Virtual memory
  • Page tables
  • Kernel memory allocation

Memory management becomes the foundation for many later subsystems.

Stage 4 — Interrupts and Timers

Add:

  • Interrupt-controller support
  • A timer
  • Basic timekeeping

A timer is essential for scheduling and preemption.

Stage 5 — Devices and Input

Begin with simple devices such as:

  • Serial ports
  • Keyboard input
  • Framebuffer output

Do not begin by trying to support every graphics card, USB controller, and network adapter.

Stage 6 — Threads and Scheduling

Create:

  • Kernel threads
  • Saved CPU contexts
  • Context switching
  • A scheduler

Start simple. A basic round-robin scheduler is enough to learn the fundamentals.

Stage 7 — User Mode

Once your kernel is stable enough, add:

  • User address spaces
  • User/kernel privilege separation
  • System calls
  • Process creation
  • Program loading

For ELF-based systems, this usually involves writing an ELF loader.

Stage 8 — Filesystems

Start with something manageable.

You could:

  • Use an initial RAM filesystem
  • Implement a simple custom filesystem
  • Implement FAT
  • Build a virtual filesystem layer

Eventually you can add mounting, path resolution, permissions, caching, and multiple filesystem types.

Stage 9 — Storage Drivers

Typical areas include:

  • ATA
  • AHCI
  • NVMe
  • VirtIO

Virtual devices are often easier to support first when developing under QEMU.

Stage 10 — Networking

Networking introduces another large collection of subsystems:

  • Network-device drivers
  • Ethernet
  • ARP
  • IPv4/IPv6
  • ICMP
  • UDP
  • TCP
  • Sockets

Treat networking as its own major project.


Build One Thing at a Time

Kernel projects become difficult when too many unfinished systems depend on each other.

A better development loop is:

  1. Choose one small feature.
  2. Read the relevant architecture or hardware documentation.
  3. Implement the smallest possible version.
  4. Test it.
  5. Add assertions and debugging output.
  6. Commit the working state.
  7. Move to the next feature.

For example, do not begin a scheduler, filesystem, USB stack, and network stack in the same week unless you already have substantial kernel-development experience.

Small, working increments are much easier to reason about.


Learn to Read Specifications

Tutorials are useful for learning concepts, but specifications are what let you build systems independently.

Eventually you should become comfortable reading documentation such as:

  • Intel Software Developer's Manual
  • AMD64 Architecture Programmer's Manual
  • ARM Architecture Reference Manual
  • RISC-V specifications
  • UEFI specification
  • PCI specifications
  • ACPI specification
  • ELF documentation
  • Hardware datasheets

Do not try to read these documents from beginning to end.

Use them as references.

When you need to configure a page table, read the page-table section. When you need to handle an exception, read the exception documentation. When you need to enumerate PCI devices, read the relevant PCI material.

That is how systems programmers use large technical manuals in practice.


Debugging Is Part of the Project

Do not treat debugging tools as something you will add later.

Build observability into the kernel from the beginning.

Useful facilities include:

  • Serial output
  • Kernel logging
  • Panic messages
  • Register dumps
  • Stack traces
  • Assertions
  • Memory diagnostics
  • Emulator debug logs
  • GDB integration

When something fails, reduce the problem.

Ask:

  • What was the last known working state?
  • What changed?
  • Did control reach this function?
  • Are the arguments correct?
  • Is the CPU in the mode I think it is?
  • Is this virtual address mapped?
  • Is this interrupt enabled?
  • Did the linker place this section where I expected?
  • What does the generated assembly actually do?

Kernel debugging becomes much easier when you learn to verify assumptions rather than guess.


Efficiency and Correctness Matter

Kernel code sits underneath everything else.

A poor design decision in an application may affect one program. A poor design decision in a kernel can affect every process in the system.

That does not mean every function must be perfectly optimised from the beginning.

It means you should care about:

  • Correctness
  • Predictable behaviour
  • Clear ownership of resources
  • Concurrency
  • Memory safety
  • Error handling
  • Reasonable abstractions
  • Avoiding unnecessary work in hot paths

Measure before performing complicated optimisations, but do not deliberately build expensive mechanisms into critical paths without understanding their cost.


Read Other Kernels

One of the best ways to improve is to study working operating systems.

Useful projects include:

  • Linux
  • FreeBSD
  • OpenBSD
  • NetBSD
  • SerenityOS
  • Haiku
  • xv6
  • seL4
  • MINIX
  • Redox

You do not need to understand an entire production kernel.

Choose one subsystem and follow it.

For example:

  • How is a process represented?
  • How are page faults handled?
  • How are files represented?
  • How does pathname lookup work?
  • How does a scheduler choose the next thread?
  • How are system calls entered?

Reading real kernel code helps bridge the gap between theory and implementation.


Ask Good Questions

OS development communities can be extremely helpful, but low-level problems usually require technical detail.

When asking for help, include:

  • What you are trying to achieve
  • Target architecture
  • Bootloader or boot protocol
  • Compiler/toolchain
  • Relevant source code
  • Compiler errors or warnings
  • Emulator output
  • Register values when applicable
  • What you expected
  • What actually happened
  • What you have already tested

Avoid posting only:

My kernel crashes. Why?

A minimal reproducible example is far easier for other developers to investigate.


Do Not Measure Progress by Lines of Code

A larger kernel is not automatically a better kernel.

A clean 10,000-line educational kernel that you fully understand may teach you more than a 200,000-line project assembled from copied tutorials.

Measure progress by capabilities and understanding.

Examples:

  • I understand how my kernel enters long mode.
  • I can explain every mapping in my initial page tables.
  • Exceptions produce useful diagnostics.
  • Physical memory allocation survives stress testing.
  • Two threads can context switch reliably.
  • User programs cannot directly access kernel memory.
  • My filesystem survives remounting and corruption tests.

Those are meaningful milestones.


A Good First Project Structure

Your project might begin with a structure similar to:

myos/
├── arch/
│   └── x86_64/
├── kernel/
├── include/
├── scripts/
├── sysroot/
├── Makefile
├── linker.ld
└── README.md

As the kernel grows, you might later add:

drivers/
fs/
mm/
net/
lib/
user/
tests/

Do not obsess over the perfect directory structure before the kernel boots. Refactor when the project becomes large enough to justify it.


Suggested Beginner Checklist

  • Choose one CPU architecture.
  • Pick a host development environment.
  • Install a compiler, assembler, linker, debugger, Git, and QEMU.
  • Configure a suitable cross-compilation or freestanding toolchain.
  • Create a Git repository.
  • Choose a bootloader or boot protocol.
  • Build a bootable kernel.
  • Print diagnostic text.
  • Add exception handling.
  • Implement memory management.
  • Add interrupts and a timer.
  • Add basic device support.
  • Implement threads and scheduling.
  • Enter user mode.
  • Add system calls.
  • Load a user program.
  • Add filesystem support.
  • Add storage drivers.
  • Expand into networking, graphics, USB, SMP, or other areas that interest you.

Do not think of this checklist as a race. Each item can contain weeks or months of learning.


Where to Learn More

The OS development community has accumulated decades of useful documentation.

Good starting resources include:

I recommend getting started with the simple bootloader tutorial in this community. Then you can move on to Daniel McCarthy's kernel development courses

Books about operating systems, computer architecture, compilers, and systems programming are also extremely valuable.


Final Advice

Start small.

Do not begin by trying to create the next Windows or Linux. Begin by creating a kernel that boots reliably and does one thing correctly.

Then add another thing.

Then another.

Eventually those small pieces become memory management, drivers, schedulers, filesystems, processes, system calls, networking, graphical environments, and user applications.

The most important skill in operating system development is not memorising every CPU register or knowing every hardware specification.

It is learning how to break a complicated machine into understandable pieces, investigate each piece carefully, and keep building even when the system does not work the first time.

That is how operating systems are built.

Discussion 0

No comments yet. Start a thoughtful discussion.

Join the discussion

You need an account to contribute.

Sign in