Every once in a while I dust off the code base of a toy OS kernel I'm working on. In the most-working state it's ever been in, if I remember correctly, it's been able to boot from GRUB, write to a text mode console and read the keyboard, enable paging (i.e. almost use virtual memory without swap), handle interrupts and switch tasks (although just to the one that exists already as it can't create them).

The idea is to build a microkernel. Instead of putting things like virtual memory, scheduling, drivers, file systems, networking, users, etc. in the kernel like most operating systems today, these are handled in separate processes. This means that the amount of code running with the ability to completely blow up your computer is very, very small, and the rest can do things like alternate implementations more easily.

Traditional monolithic kernels do everything in kernel mode for a variety of reasons. For instance, the point of the separation between user and kernel mode is to make sure user processes can't mess up each other or the system, so they just ask the kernel to do things like read hard drives or draw to the screen.

Another idea is capability-based security. Instead of using modes like a monolithic kernel, this kernel would use paging with memory mapped IO, the IO port permission system and some kind of reference-based messaging system between processes to restrict access to exactly what has been given to a process. At boot, the kernel starts out with full permissions and hands things out to the drivers and other services that need them.

If I could build a processor as well I would completely do away with kernel mode and make all IO memory mapped (the latter is done quite commonly outside of x86). I'm not sure how I would implement references to other address spaces and/or IPC handlers, but it might be nice to do it through virtual memory like IO or through some other hardware-based reference system.