uBixOS 3.0 — It Compiles Itself Now, and It Boots on a Raspberry Pi

Picking Up Where the Last Post Left Off

Last time I wrote about uBixOS, the headline was that a project I started in 2002 had a second CPU architecture, a ZFS-style filesystem, symmetric multiprocessing, and a desktop that finally idled like a desktop should. I ended that post with “here’s where it’s going.”

That was about a month ago. Roughly 360 commits later, “where it’s going” turned into “where it went,” and the OS is different enough that it earned a version number: 3.0.0-BETA. Here’s what actually changed.

Two Architectures, Both 64-Bit, Fully Equal

The last post was honest that uBixOS was aarch64-forward — ARM64 was where the new work landed first, and x86 was playing catch-up. That gap is closed.

x86_64 reached full parity. In the span of a couple of days it gained everything the ARM side had been showing off: execve passing real argv/envp into a login shell, preemptive scheduling, copy-on-write fork (it was still deep-copying every page), shared file-cache pages so there’s one libc in memory instead of one per process, a full networking stack (virtio-net + lwIP + DHCP + sockets), the NetSurf browser rendering, AC’97 audio on /dev/audio, and booting off the UbixFS pool as root. The Disk Utility and Activity Monitor data layers got real implementations. By the time it was done, x86_64 wasn’t the poor cousin anymore — it’s the abstraction anchor that keeps the machine-independent / machine-dependent split honest. When both 64-bit arches have to boot through the same code, you can’t quietly let architecture-specific assumptions leak into the core.

And the flip side of that: 32-bit i386 is retired. Master is now 64-bit only. I pulled the isolated i386 kernel tree, the ISA/PCI plumbing, and the old build targets out entirely — 32-bit lives on frozen at releng/2, and TARGET=i386 now errors out with a pointer to that branch. Twenty-four years of x86-32 was the right thing to let go of. The two targets that remain — aarch64 and x86_64 — earn their keep by keeping each other honest.

SMP That Actually Uses the Cores

The June post had SMP infrastructure — CPUs enumerated, per-CPU structures in place, application processors coming up and parking. Since then the secondaries became first-class citizens.

The application processors now run the scheduler and execute threads in parallel — real parallel scheduling, not one core doing all the work while the others idle. Getting there was a stack of correctness work that’s the kind of thing you only learn by breaking it:

Per-task FPU/SIMD save and restore on both arches. The context switch was only saving general-purpose registers, so a preempted task could resume with another task’s XMM/FPSIMD state — silent floating-point corruption that SMP migration would have made routine.
The run-queue lock is now held across switch_to, released by whatever context the scheduler resumes into. Previously the lock dropped before the register save completed, which is fine on one CPU and a data race on two.
Per-CPU run queues with a load balancer and a migration cooldown, behind a config flag and then switched on by default.
Cross-CPU TLB shootdown via IPI for shared address spaces, reschedule IPIs so idle cores hlt and wake on enqueue, and a scheduler that will never resume, migrate, or free a task whose context is still live on another CPU.

SMP bugs remain the most educational bugs in the whole system. Every one of them is invisible on a uniprocessor and instant the moment a second core touches the same memory.

The Big One: uBixOS Compiles Itself

This is the milestone I’m proudest of. uBixOS is now self-hosting — it can build its own userland, on the device, with no host machine in the loop.

That took a real toolchain, and I went with LLVM. I ported Clang, LLVM, lld, and a freestanding libc++/libc++abi/libunwind to uBixOS, and clang/lld became the default compiler for both the world and the kernel on both architectures — gcc is now the legacy path. Alongside it, the pieces a build actually needs:

bmake — cross-builds first, then built and ran inside uBixOS itself.
/bin/sh (a port of oksh), which meant fixing musl’s vfork ABI and the exec bit along the way.
22-plus busybox coreutils — sed, awk, printf, diff, cmp, install, expr, ln, and the rest — because a Makefile that can’t run sed isn’t going to build anything.

The payoff moment: bin/cat compiles and runs on-device, produced by clang and lld running under uBixOS, driven by bmake reading the tree with SRCTOP walk-up so subdirectory builds resolve the root. The kernel build mechanisms are de-host-ified too. An OS that can rebuild itself is a fundamentally different kind of project than one that needs a Mac or a Linux box to exist — it stands on its own now.

Getting the self-host loop green surfaced a beautiful chain of low-level bugs: a syscall uap struct-layout mismatch from non-deterministic PADL_/PADR_ endianness, a getrlimit/setrlimit syscall-number collision with sigprocmask, and a signal-after-COW-fork SIGSEGV that came down to execve not resetting caught signal handlers to SIG_DFL. Each one was a wall until it wasn’t.

It Runs on Real Hardware

Everything above runs in QEMU. But an OS that only exists in an emulator is a simulation of an OS. So I brought uBixOS up on an actual Raspberry Pi 3 — and every step of this is hardware-confirmed on the board, not modeled:

Serial “hello” → EL2-to-EL1 drop → DTB /memory parse → timer interrupts via the BCM2837 local controller.
A VideoCore framebuffer at 1080p — the graphical desktop renders on real hardware.
A BCM2837 EMMC SD-card driver, so the full OS boots off the SD card with the UbixFS pool as root.
The DWC2 USB host controller coming up, enumerating the hub, and walking down to the onboard SMSC LAN9514 Ethernet — which then binds an address over DHCP. Networking, on the Pi, over USB, end to end.

There’s a board-abstraction layer now (g_board, an interrupt-controller vtable, DTB-driven RAM base) so the Pi and the QEMU virt machine share the same kernel with the board differences factored out — and Pi 4 / Pi 400 (BCM2711) bring-up has already started on top of it. Watching a desktop I wrote paint itself on a monitor plugged into a $35 board that I’m SSH-ing into is a very different feeling than watching it in a window on my Mac.

A Real Ports Ecosystem

None of the self-hosting or hardware work would compose without a way to bring third-party software in cleanly, so uBixOS grew a proper ports framework (mk/ports.mk) and a growing catalog on top of it:

git 2.39.5 — with HTTPS transport working, via a libcurl 8.9.1 port on a BearSSL backend and a CA bundle. You can clone over the network from inside the OS.
dropbear — an SSH server with password auth and an interactive shell over a real pty, which is how I’m getting into the Pi.
bmake, oksh, and the LLVM toolchain, all as ports.

The ports recipe is now the default way to add a dependency, which keeps the base system honest about what’s uBixOS and what’s someone else’s code wearing a uBixOS build.

The Desktop Kept Getting Nicer, Too

Not everything this month was plumbing. The graphical shell picked up the kind of polish that makes a system feel deliberate: a Windows 11-style Start menu grid with app icons and a stacked clock, an Alt-Tab task switcher with live window thumbnails, hover previews of windows in the taskbar, and an acrylic backdrop blur for translucent surfaces like the taskbar. objGFX learned anti-aliased polygon and triangle fills. The login screen got a proper redesign — sans-serif type, a monogram avatar, a blinking caret, a corner clock. And I wrote Cubitaire, a Klondike solitaire game for the window system with a procedural deck and audio, because every real desktop needs one and it’s a great objGFX stress test.

Where It’s Going (Again)

The through-line of this month is self-sufficiency. A month ago uBixOS could boot two architectures and run a desktop. Now it’s a 64-bit OS that runs SMP across real cores, compiles its own userland with its own toolchain, boots off an SD card on a Raspberry Pi with graphics and networking, and pulls in third-party software — git, ssh, a shell — through a ports system. It doesn’t need the host it was born on anymore.

The next frontier is closing the self-hosting loop all the way — a full on-device world build, then the kernel — and pushing the Pi 4 bring-up to parity with the Pi 3. There’s a native GUI editor (“Vellum”) and a Node.js port on the design board.

The source is at github.com/cwolsen7905/UbixOS, and the project lives at ubixos.org. Twenty-four years in, it went from “back in active development” to “compiles itself and runs on hardware I can hold” in about four weeks. I don’t think I’ve ever had more fun with a codebase in my life.