Xindows 360

Tom Clay

Tom Clay

— 7 min read
Xindows 360

In "The Audacity of Static Recompilation," we explored how XenonRecomp performs the seemingly magical act of converting PowerPC assembly into compilable (is that a word?) C++. Then in "I Need a Runtime," we confronted the sobering reality: your freshly recompiled code is absolutely useless without a runtime to handle all those system calls that games depend on.

Today, we're diving deep into one specific system call - NtCreateFile - to understand why runtime implementation is so much more complex than it first appears. Because here's the thing: when you see a simple file operation in an Xbox 360 game, you're not just dealing with one function. You're dealing with layers upon layers of abstraction that Microsoft carefully engineered over decades.

Windows NT: The Foundation Nobody Talks About

Here's a fun fact that'll make you the life of any party: the Xbox 360 runs a modified version of the Windows NT kernel.

Microsoft's strategy was simple: make Xbox 360 development feel like Windows development. Game studios could leverage their existing Windows programming expertise, use familiar development tools, and port code between PC and console with minimal friction. This approach significantly reduced the learning curve compared to competing platforms that required mastering entirely new programming paradigms.

cough PlayStation 3 cough

Lost in Translation

Creating a runtime that bridges these architectural differences is like being a simultaneous translator at the UN, except both parties are speaking variants of the same language with completely different accents and idioms. The runtime needs to handle multiple translation layers simultaneously.

Let's use NtCreateFile as our case study for understanding how Xbox 360 system calls work and why runtime implementation is so complex. This single function perfectly illustrates the layered architecture, multiple entry points, and translation challenges that define the Xbox 360 kernel.

UnleashedRecomp/kernel/imports.cpp

GUEST_FUNCTION_HOOK(__imp__NtCreateFile, NtCreateFile);

We can see it is setting up a PowerPC function to call the host method NtCreateFile. Great. Let's go check it out:

Great. Let's go check it out.
UnleashedRecomp/kernel/imports.cpp

uint32_t NtCreateFile
(
    be<uint32_t>* FileHandle,
    uint32_t DesiredAccess,
    XOBJECT_ATTRIBUTES* Attributes,
    XIO_STATUS_BLOCK* IoStatusBlock,
    uint64_t* AllocationSize,
    uint32_t FileAttributes,
    uint32_t ShareAccess,
    uint32_t CreateDisposition,
    uint32_t CreateOptions
)
{
    return 0;
}

What the hell? It just returns 0... Where is the implementation?

Well, this is where recompilation really shines. The ability to inject host functions into the recompiled code.

Let's take a look at UnleashedRecomp/kernel/io/file_system.cpp to see how the guest file system calls are handled.

GUEST_FUNCTION_HOOK(sub_82BD4668, XCreateFileA);
// more guest hooks
GUEST_FUNCTION_HOOK(sub_831CE3F8, XCreateFileA);

What is actually happening is the UnleashedRecomp runtime is hooking the high-level version(s) of it, CreateFileA . But why are there 2 guest functions being hooked into the same host function?

Let's hop into IDA to dig up more intel. We will refer to sub_82BD4668 as XCreateFileA_1 and sub_831CE3F8 as XCreateFileA_2 from here on out.

A Quick Note on XCreateFileA vs CreateFile vs CreateFileA

The XCreateFileA naming in UnleashedRecomp's runtime was likely chosen to prevent symbol collisions with Windows' native CreateFileA when compiling. The Xbox 360 SDK would have exposed this as the native Win32 API, CreateFile, CreateFileA, or CreateFileW.

In the Win32 API, most string-handling functions come in two flavors:

  • ANSI versions (suffix A): Use 8-bit char strings - CreateFileA, MessageBoxA, etc.
  • Unicode versions (suffix W): Use 16-bit wchar_t strings - CreateFileW, MessageBoxW, etc.

When CreateFile is used, it's actually a preprocessor macro that expands to either CreateFileA or CreateFileW depending on whether you've defined UNICODE in your build.

The Xbox 360, like most game consoles of its era, primarily used ANSI strings for performance reasons. Unicode support was available, but the A variants were the dominant path.

The NT Rabbit Hole Deepens

If you haven't noticed by now, there are multiple abstraction layers to a seemingly simple function.

Why Did Microsoft Design It This Way?

This wasn't accidental complexity - it was a deliberate architectural decision that dates back to Windows NT's origins in 1988. When Dave Cutler and his team from Digital Equipment Corporation (DEC) joined Microsoft, they brought with them the lessons learned from building VMS, one of the most portable operating systems of its era.

Windows NT was initially developed on the Intel i860 RISC processor, then moved to MIPS R3000, then i386, and eventually supported Alpha, PowerPC, and other architectures. The subsystem design with a "Native API" at its core meant the kernel could remain processor-independent while multiple OS personalities (Win32, OS/2, POSIX) sat on top as user-mode subsystems.

Multiple OS Personalities

The Native API is intentionally undocumented and unstable. Applications aren't supposed to call it directly. Instead, environment subsystems like Win32 translate their respective API calls into Native API calls. This means programs written for different operating systems could theoretically run on the same NT kernel. While this was partly a marketing play (POSIX compliance was required for US government contracts), it demonstrated the architecture's flexibility.

The Xbox 360 leverages this exact subsystem architecture. Microsoft created a custom Xbox subsystem that sits alongside (and partially includes) Win32 components. Game developers called familiar APIs like CreateFile, which the Xbox subsystem translated into appropriate kernel operations - but with Xbox-specific behaviors like the virtual filesystem (game:\, update:\, etc.) and security restrictions. This is why you see NtCreateFile (Native API) and CreateFile (subsystem API) coexisting in Xbox 360 binaries - they're different layers of the same architectural stack.

The Real Implementation

Anyways, back to the runtime code...

The XCreateFileA implementation in UnleashedRecomp shows us exactly how this translation magic works:

static FileHandle* XCreateFileA
(
    const char* lpFileName,
    uint32_t dwDesiredAccess,
    uint32_t dwShareMode, 
    void* lpSecurityAttributes,
    uint32_t dwCreationDisposition,
    uint32_t dwFlagsAndAttributes,
    uint32_t hTemplateFile
)
{
    // Path resolution with mod support
    std::filesystem::path filePath = FileSystem::ResolvePath(lpFileName, true);
    
    std::fstream fileStream;
    std::ios::openmode fileOpenMode = std::ios::binary;
    
    if (dwDesiredAccess & (GENERIC_READ | FILE_READ_DATA))
    {
        fileOpenMode |= std::ios::in;
    }

    if (dwDesiredAccess & GENERIC_WRITE)
    {
        fileOpenMode |= std::ios::out;
    }

    fileStream.open(filePath, fileOpenMode);
    if (!fileStream.is_open())
    {
        return GetInvalidKernelObject<FileHandle>();
    }

    FileHandle *fileHandle = CreateKernelObject<FileHandle>();
    fileHandle->stream = std::move(fileStream);
    fileHandle->path = std::move(filePath);
    return fileHandle;
}

Notice what's happening here? Instead of attempting to recreate the Xbox 360's exact file system semantics, UnleashedRecomp takes a pragmatic approach:

  • Path Translation: Xbox paths like "game:\textures\sonic.dds" get resolved to actual PC file paths.
  • Standard Library: Uses std::fstream instead of low-level Windows APIs for cross-platform compatibility.
  • Kernel Object Management: Maintains a handle-based paradigm through custom kernel objects similar to how the kernel would actually handle them.

The Dichotomy Revealed

There are two different implementations of the same file creation API that both get hooked to XCreateFileA in UnleashedRecomp. Both functions are nearly identical - same prologue, same parameter processing, same error handling logic. They both eventually call NtCreateFile to do the actual kernel work. But there's one crucial difference in how they make that call.

The Critical Difference: Direct vs. Indirect Dispatch

XCreateFileA_1:

.text:82BD47A8    lwz       r11, -0x112C(r6)
.text:82BD47C0    lwz       r11, 0xC(r11)
.text:82BD47C4    mtctr     r11
.text:82BD47C8    bctrl

This loads a function pointer from a table, moves it to the count register, and performs an indirect branch-and-link. Classic dynamic dispatch through a function pointer table.

XCreateFileA_2:

.text:831CE548    bl        NtCreateFile

This directly branches to NtCreateFile. No indirection, no function pointer lookup, just a straight call.

Why Two Paths to the Same Destination?

The cross-reference analysis reveals the usage patterns and gives us a little bit of insight:

XCreateFileA_1 (17+ cross-references):

  • Called extensively by C runtime library functions like _tsopen_nolock, _wfopen
  • Standard file I/O throughout the game code
  • Uses dynamic dispatch, likely because it's part of a relocatable library where NtCreateFile addresses need runtime resolution

XCreateFileA_2 (3-4 cross-references):

  • Limited usage in specific subsystems
  • Statically linked code where NtCreateFile's address is known at compile time

The real distinction appears to be linking strategy rather than functional behavior:

  • XCreateFileA_1: Dynamically linked CRT function using a function pointer table (IAT-style)
  • XCreateFileA_2: Statically linked variant with direct calls

Both ultimately execute the same kernel operation, but the indirection in XCreateFileA_1 suggests it's part of the Xbox 360's standard C runtime library, while XCreateFileA_2 may be an optimized path for code that was statically linked against the kernel.

Why not just hook the import stub?

This is a valid route to take but attempting to intercept these calls at a high level is always the optimal approach to eliminate unneeded mangling and conforming of the function arguments to adhere to the expectations of lower-level APIs. Hooking import stubs should be viewed as the nuclear option.

The Bigger Picture

The XCreateFileA implementation is just one tiny piece of a massive puzzle. A complete runtime includes (but not limited to):

  • Memory Management: XAllocMem, XFreeMem, heap operations
  • Threading: Thread creation, synchronization primitives, atomic operations
  • Graphics: D3D9 interception and translation to modern APIs
  • Audio: XMA audio decoding and mixing
  • Input: Controller input translation and rumble support

Conclusion: Why This Matters

Understanding how XCreateFileA_1 and XCreateFileA_2 both represent different linking strategies for the same underlying CreateFile operation reveals a deeper insight into the Xbox 360 software architecture. The console ran actual NT kernel code with a custom subsystem layer that games called through familiar Win32-like APIs.

Knowing this information is a good starting point for a large majority of the functions that are required to be implemented; however, it is important to note that there are specific quirks and features to these seemingly familiar APIs. In addition, there are numerous kernel APIs that are not documented as they are Xbox 360 specific. The Xenia project can be a great source for information on the specific quirks and implementations.

By understanding these principals, it is possible to create runtimes that are both compatible with the original game's expectations and optimized for modern hardware. It's not just about making old games run - it's about making them run better than they ever could on their original hardware.

Until next time!
Tom

"Bill Gates says abstraction is like condoms, you should wear 3 of them." - Terry Davis

References:

  1. Russinovich, M., Solomon, D., & Ionescu, A. (2017). Windows Internals, Seventh Edition. Microsoft Press.
  2. Russinovich, M. (1998). Windows NT and VMS: The Rest of the Story. IT Pro Today.
  3. White, B. M. (2023). The History of Windows NT 3.1. Abort Retry Fail.
  4. Windows Native API. Wikipedia.
  5. Inside Native Applications. Microsoft Sysinternals.

Read more