Its headline business features—Active Directory, Group Policy, Internet Information Services, Management Console, Windows Management Instrumentation—have become industry standards. But most importantly, Windows NT served as the technological basis for what can fairly be described as the most successful and well-known software product of all time: Windows XP.

But there’s always been a dirty little secret hiding underneath that iconic field of green grass. From an engineering and security standpoint, the foundation of Windows 2000 and Windows XP is absolutely horrible.

The many security shortfalls of Windows have been discussed at length over the years. Terms like “Nimda,” “Code Red,” and “Messenger Service” can still give battle-hardened IT administrators involuntary twitches. Malware evolved from being the hobbyhorse of bored teenagers into being a profitable industry. It was the sorry state of security that led Microsoft Chairman Bill Gates to announce a company-wide "Trustworthy Computing" initiative; he told the company's software developers that "when we face a choice between adding features and resolving security issues, we need to choose security. This resulted in Windows Vista and Server 2008 being delayed by years, so that Microsoft could focus instead on fixing many of the worst security problems in Windows XP and Server 2003. While the improvements made in XP Service Pack 2 such as the on-by-default firewall, less permissive Internet Explorer, and tightened permissions were both necessary and welcome (despite the slow uptake), it didn’t take long for it to become obvious that Windows had received first aid, not surgery—the rate at which new security vulnerabilities were found barely abated. Even now, more than five years since the release of XP SP2, the rate at which new security vulnerabilities are found remains steady.

Microsoft’s problems with Windows development extended well beyond security. By the time Windows Server 2003 was released, Microsoft had come to accept that while it was pretty good at producing a fully-featured NT-based operating system with an extensive graphical user interface, it was wholly unable to produce a fully-featured NT-based operating system without one. Evidence of reliance on the presence of a graphical user interface can be seen all through older Windows 2000 and Windows XP. For example, there were no command-line tools to configure and install patches and software from Windows Update, nor was it even possible to select the updates to install without using Internet Explorer. Another problem was the tendency of some low-level system services to present graphical interfaces to the user, and to make use of code inside of DLLs that were primarily concerned with displaying a user interface. Worse still for Microsoft’s thousands of Windows developers, Windows had become such a large project that it was extremely difficult for any group to make changes to one part of the system and know exactly how those changes would affect the rest of the system.

Readers who are familiar with Windows Embedded (and third-party tools like nLite) are probably scratching their heads right about now, wondering what all the fuss is about. Hasn’t the ability to build a small version of Windows for a specific purpose been around for a very long time? Yes, it has, but the approach used by these tools is roughly equivalent to taking a hacksaw to a large Persian rug in order to make it fit in a smaller room. Sure, it’s still a rug, but it’s a pretty nasty way of getting the job done. Windows Embedded, nLite, and similar tools simply chop the unwanted DLL files from the system, twiddle a few things here and there in the Registry, and call it a day. With Windows Embedded, this may not be such a bad approach, because the intended purpose of Windows Embedded is to run a single application, such as a kiosk, ATM, or outdoor display. But for a general-purpose operating system, it’s just not an option.

Getting to the Core of the problem

With these problems in mind, Microsoft assembled a team of core operating system architecture experts that were tasked with mapping out the many interdependencies between the 5,500+ components of Windows, and devising a plan to separate them into “layers.” The lowest layer would become a self-contained system that has no understanding of, or dependence on, the large majority of Windows. Successive layers would include more functionality, but would only be dependent on layers underneath it. This project, which was underway by the end of 2003, was given an internal name of “MinWin.”

The first real manifestation of this componentization effort was the Windows Server Core installation option in Windows Server 2008. The Server Core pitch is simple and compelling enough—it’s still Windows, but without the vast majority of the desktop components that have no place on a dedicated server. You get something that can still run your Active Directory domain controllers, your file shares, and your DNS servers, without having to be bothered by installing patches for Internet Explorer, Windows Media and a host of other components that were welded into prior releases of Windows. Fully two-thirds of the security patches released for Windows Server 2003 offered no actual increase in security for dedicated servers, but still required software to be installed and reboots to be performed on a near-monthly basis.

When you install Server Core, you are greeted with the familiar Windows login screen, complete with high-resolution graphics. Log in, and you’re presented with a command prompt. Server Core in 2008 R2 is a vast improvement on 2008 in this area; just type “sconfig” and you can step through some simple menus to configure networking, remote management, Windows Update, and domain participation. A couple of graphical applications and configuration panels like Notepad, Task Manager, and Date and Time are still present, but all other tasks on the machine must either be done through command-line utilities, or via remote management. What is offered in terms of UI capabilities is very basic. Because Windows Explorer and many of the components of the Shell aren’t included in a Server Core install, applications like Notepad, which would normally show an “Open” dialog box.

Yes, that’s really the Open dialog that was introduced with Windows 3.1 in 1992! Printing doesn’t work because there is no print subsystem. Help doesn’t work, either, because neither the help files nor the help engine are installed. Twelve fonts are available instead of the 50 or so that a full Server 2008 install will have.

Sounds good so far, yes? There’s a big problem—a Server 2008 Core installation, without any installed roles or extra features, still has disk and memory requirements that handily exceed a full Windows 2000 install. Server 2008 R2’s memory consumption is somewhat lower (I estimate about 30% lower than Server 2008 before roles and features are installed), but disk requirements remain high. Microsoft says there are still about 600 interconnected DLLs and other binaries, totaling hundreds of megabytes. None of these could be safely removed without causing other parts of the system to break. Even Mark Russinovich, widely regarded as one of the world’s top experts on Windows internals, admits that they still can’t predict what exactly would break.

Clearly, if Microsoft wants to demonstrate that its idea of a “slimmed-down” server operating system resembles something a bit smaller than, say, Canada with the northern territories chopped off, there is still a lot more to be done.

More panes, less pain

While the MinWin project had been discussed by some Microsoft representatives as far back as 2005, it wasn’t until late 2007 that its existence and true goals became widely known. Eric Traut, one of Microsoft’s lead operating system engineers demonstrated a very small version of Windows running completely without a graphical user interface, and doing little else aside from running a simple Web server. It was a short demonstration, just a few minutes long, but the news hit the tech world like a tsunami. MinWin was described variously as a “new kernel,” a “replacement for the bloated Windows Vista,” a “text-based Windows,” “Mini-me,” and so on. Truth be told, MinWin is none of these. It's a slice of the core Windows operating system that includes virtual memory management, a model for running processes and threads, file system support, networking, and some other basics that are common in all modern operating system kernels. What it doesn’t include are a number of other components that are still very much at the core of what Windows is, such as the shell, graphics and audio systems, Windows Management Instrumentation, support for many classes of devices… even the venerable command prompt was removed. The rest of Windows then sits on top of MinWin.

Let’s have a closer look at some of the work that’s been done to make this possible.

Assumptions and internals

Software written for Windows has always been written with a certain set of assumptions. It is expected that the basic Windows API functions are located in KERNEL32.DLL, USER32.DLL, ADVAPI32.DLL and GDI32.DLL. These four DLLs contain over 6,000 function calls that cover an enormous variety of common tasks, from creating new processes, to drawing bits of the Windows user interface, to displaying pages from Windows Help. The origins of these DLLs date back some 20 years, and each successive release of Windows has piled on new bits of functionality. For better or worse, Microsoft is stuck with these DLLs until the end of time. The size and diversity of functionality in these (and other) components also means that many functions that one could reasonably include in “MinWin” are sitting in the same DLL as functions that one certainly would not. One example: advapi32.dll contains functions to manage the installed services on the local computer, as well as functions to interact with a Windows domain. Services are an ingrained part of Windows (even in MinWin), but domains are not.

Fortunately, most software doesn’t need to be too concerned with the internal implementation details of any given API call, or where exactly the real implementation is. As long as the functions continue to do what they’re expected to do, and as long as there are files with names like kernel32.dll that contain references to those functions, software will continue to work from one version of Windows to the next.

(A perhaps necessary aside here: despite its name, the file “kernel32.dll” doesn’t actually contain the Windows kernel itself. Its purpose is to give user-mode applications access to functionality that is contained in the kernel.)

In order to solve this problem, Microsoft looked at the complete contents of the core DLL files and grouped related functions into something it calls “API sets.” For instance, the 40 functions that work with the Registry on the local computer have been grouped together into an API set called “API-MS-WIN-CORE-LOCALREGISTRY-L1-1-0.” Note the last part of the name — the “L1” refers to the architectural layer the API set is a part of. L1 is the lowest level, i.e. MinWin. The two numbers following the layer number are the version number of the API (i.e. 1.0). Presumably the version number will change if the functions contained in the API set change at some point in the future, making it possible for the functionality of older versions to be retained.

Other API sets cover process and thread management, file operations, localization, system information, services, memory management, string manipulation, and a few other things—34 API sets in all, almost all of which are part of layer 1. The library loader component of Windows (which has itself been put into the API-MS-WIN-CORE-LIBRARYLOADER-L1-1-0 API set) has been updated to recognize these API set names, but the actual functionality lives inside a DLL file with another name. The mappings are stored in a new DLL file called “apisetschema.dll.” This DLL, which is loaded when the system is booted and is automatically mapped into every process on the machine, can tell the library loader that those 40 Registry functions that make up the API-MS-WIN-CORE-LOCALREGISTRY-L1-1-0 set are actually stored in sechost.dll. These are called “logical DLLs”, and in Windows 7 and Server 2008 R2, there are four: kernel32.dll, ntdll.dll, sechost.dll, and KernelBase.dll. That last one is brand new—it contains the portions of kernel32.dll and advapi32.dll that are now part of MinWin.

Moving groups of functions into other DLLs is all well and good, but what about backwards compatibility? In order to ensure that applications that aren’t aware of these changes (i.e. pretty much every application ever written) will still work, the original DLLs still have the function calls in them. The function call will simply be forwarded to the new DLL.

One last thing before we move on. A variety of tools that do DLL analysis (such as Dependency Walker) expect that an actual DLL file exists, so Microsoft created a series of “stub” DLL files in the System32 directory that match the names of the API sets. You can see these on any Windows 7 or Server 2008 R2 system: turn on “Show hidden files” in the Folder options of Windows Explorer, navigate to the \windows\system32 directory, and search for the files starting with “api-ms-win.” These files are not actually used by Windows when loading libraries; this explains why, when you use a tool like Sysinternals’ Process Monitor, you will never see references to those DLL files.

Go small or go home

Six years ago, Microsoft assembled a team and set out how to “not make things worse with Vista”, as one executive put it. Today, we find them still very much in the middle of this long project to bring an end to the long-standing, well-earned reputation of Windows being a large, bloated, inflexible operating system.

Those who watched Eric Traut’s original MinWin demonstration, or have attended talks on the new kernel by Dave Probert, Mark Russinovich, and other core operating system team members this year, are surely wondering why Server Core still has a graphical mode at all. Microsoft hasn’t untangled all the graphical components of Windows from the non-graphical ones, so Server Core as it ships today is the best the company can do for now. A more flexible, more componentized Windows Server paves the way for completely GUI-less operation, a smaller install footprint, lower memory usage, fewer security patches, and lower power usage.

These benefits aren’t limited to server workloads, of course. A Windows kernel that can be sliced and diced to include or exclude any number of capabilities could elbow its way into the world of smaller, low-power embedded devices that are currently the stomping grounds of QNX and Linux, as well as the company’s own Windows CE product. While Microsoft has not come right out and said that it sees a future where the Windows CE kernel (and, by extension, Windows Mobile) would be supplanted by the Windows NT kernel, it’s obvious that having one core operating system development team at Microsoft instead of two would be a lot more efficient. It’s not like there isn’t a precedent in the industry—Apple’s iPhone OS, Google’s Android, and Palm’s webOS have already proven that a general-purpose kernel works as the basis for a phone.

So, it’s good for servers, good for embedded devices, and good for phones. But what about the desktop? The benefits of modularization with Windows 7’s successors are less obvious. When Peter Bright wrote about the modularization of Windows last year, he rightly concluded that this could empower Microsoft’s accounting department to slice and dice Windows in ways that could effectively screw the customer out of their money in the long run. Aside from that, Windows users of the future will have a more stable and well-tested operating system.