Writing Windows Applications in x64 Assembly in 2026

Windows Development

Low Level Programming

X86 64

Software Development

Writing Windows Applications in x64 Assembly in 2026

Stefan Hofmann

11 min read·

Just now

--

The compiler’s tax, the practical Win32-to-Win64 migration traps — with a 55 KB tray application as the running example.

ClipFlow, the running example for this article, is a Windows tray application: press a hotkey and it loads a text template onto the clipboard; templates can carry $$Name$$ placeholders, a small dialog asks for the values, and the personalized text is ready to paste anywhere. The binary is 55 KB, icon included — one executable, no installer, no runtime, no framework, imports from four system DLLs, and its source contains no networking code.

But the app is not the subject. The subject is the layer underneath, the one every C compiler handles for you. If you wrote 32-bit Windows assembly back in the MASM32 days, this article is your migration path: the x64 delta is smaller than rumored and sharper than expected. The calling convention is the famous half of it. The quiet half — instructions that vanished, addressing that changed meaning, a segment register that moved, exceptions that became metadata, and a type system where half the types widened and the other half did not — is where 32-bit habits break silently.

The assembler is JWasm, the headers are the WinInc set, and the linker comes from the Windows SDK. A complete build:

jwasm -nologo -I%USERPROFILE%\WinInc\Include -win64 -Zp8 clipflow.asm rc /nologo clipflow.rc link /nologo /subsystem:windows /entry:start clipflow.obj clipflow.res

-Zp8 matters more than it looks: the SDK structures on x64 assume natural alignment capped at 8 bytes, and a mismatch silently changes every offset in every struct you touch.

There is no C runtime in the binary — the entry point is genuinely yours. For clarity, here is an explicitly stack-managed entry-point sketch (the real build should follow one consistent assembler stack-management strategy):

start: and rsp, -16 ; align for calls; entry is not a normal proc sub rsp, 20h ; reserve 32 bytes of outgoing shadow space call RunApplication mov ecx, eax ; ExitProcess takes a UINT in ECX call ExitProcess ; does not return

Here the linker explicitly selects start with /entry:start. The manual sub rsp, 20h is essential: alignment alone does not provide the callee's shadow space. In normal framed procedures, let your chosen JWasm invoke strategy manage outgoing arguments rather than mixing it with ad hoc adjustments.

No CRT startup objects, no hidden constructors, no atexit machinery. The first instruction that runs is one you wrote.

The following options illustrate the relevant assembler settings (choose one win64 mode, not both):

In C, the compiler pays a tax on your behalf — on every call, in every function. In assembly you are the compiler, and the tax collector still comes. These are the rules that actually bite.

The first four integer or pointer arguments travel in RCX, RDX, R8, R9 (floats use XMM0–XMM3). Everything from the fifth parameter onward goes on the stack, one argument per 8-byte slot — a DWORD argument still occupies a full slot. And the callee never cleans up: every procedure ends with a plain ret, never ret 8. In 32-bit Windows you had stdcall, cdecl, fastcall and thiscall coexisting; ordinary Windows x64 API calls use a unified default convention (specialized conventions such as __vectorcall also exist). The ret n habit from stdcall is the one that bites first — nothing crashes immediately; the caller's stack is just quietly wrong.

Before every call, the caller reserves 32 bytes of stack directly above the return address. The callee is entitled to spill RCX, RDX, R8 and R9 there — its “home slots”. From the callee’s perspective at entry:

[rsp+0x28] 5th parameter (first stack argument) [rsp+0x20] home slot for R9 [rsp+0x18] home slot for R8 [rsp+0x10] home slot for RDX [rsp+0x08] home slot for RCX [rsp+0x00] return address

Skip the shadow space and the crash does not happen in your code — it happens inside the callee, when it writes its parameters home. A crash far from the bug is the signature of this mistake.

RSP must be 16-byte aligned immediately before the call; the callee therefore enters with RSP 8 mod 16, and the prologue's pushes restore alignment as they go. The reason is mundane and brutal: SSE. movaps and friends fault on unaligned operands, and plenty of system code uses them. Get this wrong and your process dies inside a DLL you did not write, with a stack trace that is a lie. (It is also why the entry point above aligns RSP before reserving outgoing shadow space.)

RCX, RDX, R8 and R9 are volatile. A procedure that needs its parameters after making calls of its own must put them in memory. With the appropriate JWasm Win64 homing option and a framed procedure, the generated prologue can home them into the incoming shadow space (schematically):

push rbp mov rbp, rsp push rsi push rdi sub rsp, N ; locals + outgoing shadow space, 16-aligned mov [rbp+0x10], rcx ; parameter 1 → its home slot mov [rbp+0x18], rdx mov [rbp+0x20], r8 mov [rbp+0x28], r9

The parameters are stable memory references from then on — and the fifth parameter, if there is one, sits at [rbp+0x30]. This is why a window procedure can still read its LPARAM deep inside a message handler.

x64 structured exception handling is table-based. Every non-leaf function should carry unwind data (.pdata/.xdata) so that debuggers walk honest stacks and exceptions can pass through your frames. option frame:auto together with proc frame can generate it; a true leaf function that needs no unwind metadata is exempt. Without it, everything runs fine — until the day it doesn't, and the crash dump blames the wrong function.

Volatile: RAX, RCX, RDX, R8–R11, XMM0–XMM5. Nonvolatile general-purpose registers (RBX, RBP, RSI, RDI, R12–R15) and XMM6–XMM15 must be preserved if modified. In the right framed-procedure mode, JWasm USES can help save and restore these; it is not a blanket guarantee for every register or context.

Functions such as printf(format, ...) accept a variable number of arguments. Windows x64 uses the normal calling convention: the first four arguments occupy RCX, RDX, R8 and R9 for integer or pointer values, or XMM0–XMM3 for floating-point values. Arguments beyond the fourth go on the stack in 8-byte slots.

The important difference is how floating-point arguments are passed. For variadic calls, a floating-point value in one of the first four positions must appear in both its XMM register and the corresponding general-purpose register, using the same bit representation.

For example, printf("%f %d", 1.5, 7) passes the format pointer in RCX, the double 1.5 in both XMM1 and RDX, and the integer 7 in R8. The register numbers follow argument positions, not separate integer and floating-point sequences.

The caller reserves the usual 32-byte shadow space. The callee can save register arguments into their home slots, allowing a va_list to traverse them together with stack arguments. Standard C default argument promotions still apply: float becomes double, and narrow integer types are promoted to int or unsigned int.

32-bit register writes zero the upper half of the 64-bit register; 8- and 16-bit writes do not:

mov eax, byteCount ; upper half of RAX is now guaranteed zero sete al ; AL = 0/1 — the rest of EAX is stale movzx eax, al ; so widen explicitly

And the upper halves of return values are only defined where the type is 64 bits wide:

invoke GetMessageW, addr msg, 0, 0, 0 .if eax == -1 ; BOOL: 32-bit — test EAX, never RAX ...

invoke DialogBoxParamW, ... .if rax == -1 ; INT_PTR: 64-bit ...

Testing the wrong half of RAX is a bug that assembles, runs, and simply never takes its error branch.

The calling convention above is the famous half of the migration. The quiet half is below — the things that don’t announce themselves, because half of them still assemble and then mean something different.

Long mode removed more than most people notice:

pushad ; gone entirely — save registers individually push ecx ; gone — push always moves 8 bytes now push 80000000h ; still assembles — but sign-extends!

That last one is the quiet trap: push with a 32-bit immediate pushes the value sign-extended to 64 bits. push 80000000h puts 0xFFFFFFFF80000000 on the stack, not 0x0000000080000000. If you meant the unsigned value, load through a 32-bit register first — mov eax, 80000000h zero-extends RAX — then push rax. For signed widening from a memory DWORD there is movsxd; for unsigned, mov into the 32-bit register and rely on the zero-extension.

Two more removals worth knowing: the single-byte INC/DEC encodings (0x40–0x4F) were reassigned as REX prefixes, so INC/DEC still exist but with different encodings — invisible unless you hand-assembled opcode tables. And there is no push imm64 at all: the only instruction that holds a full 64-bit constant is mov reg, imm64.

mov has a related encoding trap. mov r64, imm64 can load a full 64-bit constant into a register, but there is no mov m64, imm64: a direct immediate store to a 64-bit memory operand uses a sign-extended 32-bit immediate. For example, mov qword ptr [rbx], -1 writes 0xFFFFFFFFFFFFFFFF, which is intentional. For an unsigned value such as 0x0000000080000000, make the width explicit:

mov eax, 80000000h ; zero-extends into RAX mov qword ptr [rbx], rax

This avoids relying on an assembler’s handling of an out-of-range or ambiguous immediate.

In 32-bit code, mov eax, offset MyString puts an absolute address in a register. In 64-bit code, memory references with a plain displacement assemble to RIP+disp32 instead — relative to the instruction pointer. lea rax, [MyString] and mov ecx, [counter] are position-independent by construction.

The consequences are pleasant: almost no .reloc fixups for your code, ASLR for free, and your source syntax barely changes. The rule to internalize: ordinary static data references are generally RIP-relative. Restricted absolute forms still exist (including certain mov accumulator forms); a general 64-bit address can be loaded with mov rax, imm64 and then dereferenced.

If your 32-bit code read the Thread Environment Block — the self-pointer, stack limits, TLS array, last error — it did so through FS:

mov eax, fs:[18h] ; 32-bit: linear address of the TEB mov eax, fs:[30h] ; 32-bit: the PEB

On x64, all of that lives under GS instead:

mov rax, gs:[30h] ; 64-bit: TEB self-pointer mov rax, gs:[60h] ; 64-bit: the PEB

The trap: FS-relative addressing still assembles in 64-bit mode. It just no longer points at anything useful in a 64-bit process. Code that reads fs:[18h] compiles, runs, and returns garbage — another member of the "looks fine, means something else" family. In practice you will rarely need the TEB directly; the API covers almost all of it. But if you port code that used it, this is the line to fix.

The classic 32-bit SEH prolog — installing a handler at run time by pushing it and re-pointing the FS chain — simply does not exist on x64:

; the MASM32-era pattern — impossible on x64 push handler push dword ptr fs:[0] mov dword ptr fs:[0], esp

The 64-bit dispatcher does not use the old FS-linked handler chain. It looks up function-table entries in .pdata for the function containing the faulting RIP and reads unwind codes from .xdata that describe the prolog: which registers were pushed, where the frame is, how far to step. You don't install a handler — you declare your prolog to the assembler, before execution ever starts:

MyProc proc frame push rbp .pushreg rbp mov rbp, rsp .setframe rbp, 0 sub rsp, 28h .allocstack 28h .endprolog ; after this, the prolog must not change

JWasm’s option frame:auto emits all of this for you — the directives above are what it produces. The mental shift: in 32-bit, SEH was code; in 64-bit, SEH is metadata. A prolog that lies about itsel