Article Community improvements enabled

Where Does the Fifth Argument Go? Calling NASM from C on Windows x64

Daniel McCarthy published 1 week ago 4 min read 118 views
New post
Daniel McCarthy remains the original author. Improvements are attributed to their editors and reviewed by the original author before publication.

A C call with five integer arguments is a useful way to see the boundary between registers and the stack. Under the Windows x64 calling convention, the first four arguments in this example arrive in registers. The fifth is on the stack at [rsp + 40] when the assembly routine begins.

We'll build a leaf function in NASM and call it from a Windows C program. It computes 2*a + 3*b + 5*c + 7*d + 11*e. The different weights help expose an argument read from the wrong place.

Read the stack at function entry

For this prototype, each parameter is a uint64_t. The ABI puts a in RCX, b in RDX, c in R8, and d in R9. The caller reserves four eight-byte home slots for those parameter positions and places e immediately after them. The home area is also called shadow space. The register arguments still arrive in their registers, and the caller need not copy them into those slots. Microsoft x64 calling convention

At the call site, the fifth argument is 32 bytes above the caller's current stack pointer. CALL pushes an eight-byte return address, so the same memory location is 40 bytes above the callee's entry stack pointer.

Offset from entry RSP Contents or reserved slot
0x00 Return address
0x08 Home slot for argument 1
0x10 Home slot for argument 2
0x18 Home slot for argument 3
0x20 Home slot for argument 4
0x28 Argument 5

The caller aligns the stack to 16 bytes before the call. The return address shifts the entry stack pointer by eight bytes. This routine can keep that entry value throughout because it is a leaf that makes no calls, allocates no stack storage, and changes no nonvolatile registers. Microsoft x64 stack usage

Implement the NASM function

Save this as weighted_sum5.asm. The routine accumulates into RAX, the return register for this integer result, and uses R10 as a temporary. Both are volatile under the ABI. Because there is no stack adjustment, the offset in the final multiply stays relative to entry RSP. Microsoft x64 calling convention

bits 64
section .text
global weighted_sum5

weighted_sum5:
    imul rax, rcx, 2              ; 2 * a
    imul r10, rdx, 3              ; 3 * b
    add  rax, r10
    imul r10, r8, 5               ; 5 * c
    add  rax, r10
    imul r10, r9, 7               ; 7 * d
    add  rax, r10
    imul r10, qword [rsp + 40], 11 ; 11 * e
    add  rax, r10
    ret

The three-operand IMUL keeps the low 64 bits of each product. Those bits are the same for signed and unsigned multiplication, so the result matches unsigned arithmetic modulo 2⁶⁴. The routine does not use the overflow flags. Intel instruction-set reference, IMUL

Link it to C

Save this caller as weighted_demo.c. Its declaration gives the compiler the parameter widths and return type it needs at the call site.

#include <inttypes.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>

extern uint64_t weighted_sum5(uint64_t a, uint64_t b, uint64_t c,
                              uint64_t d, uint64_t e);

int main(void)
{
    const uint64_t sum = weighted_sum5(1, 2, 3, 4, 5);
    printf("weighted_sum5(1, 2, 3, 4, 5) = %" PRIu64 "\n", sum);
    return sum == UINT64_C(106) ? EXIT_SUCCESS : EXIT_FAILURE;
}

Use NASM and GCC targeting 64-bit Windows, such as a MinGW-w64 toolchain where gcc -dumpmachine reports x86_64-w64-mingw32. Run the following in a Windows shell with those tools on the path. The win64 format produces the x64 object file the linker needs. NASM output formats

nasm -f win64 weighted_sum5.asm -o weighted_sum5.obj
gcc -std=c11 -O2 -Wall -Wextra weighted_demo.c weighted_sum5.obj -o weighted_demo.exe
.\weighted_demo.exe
weighted_sum5(1, 2, 3, 4, 5) = 106

Inspect both sides of the call

GNU objdump can show how the compiler and the NASM routine agree on the same stack slot.

objdump -d -Mintel --disassemble=main weighted_demo.exe
objdump -d -Mintel --disassemble=weighted_sum5 weighted_demo.exe

In the optimized build used here, main stores 5 at [rsp+0x20] before calling weighted_sum5. Inside the routine, imul r10,QWORD PTR [rsp+0x28],0xb multiplies the value at the same address by 11. The hexadecimal offset grows by eight because the stack pointer moved when the return address was pushed.

Nine test cases passed at both -O0 and -O2. They exercised each parameter position individually and included fifth arguments above the 32-bit range. In particular, a fifth argument of 0x100000000 with the other arguments zero produced 0xb00000000, which catches an accidental 32-bit load of the stack parameter.

If you extend this routine to allocate stack space or call another function, revisit the stack offsets and the Windows x64 rules for frame functions. A load using the entry offset continues to work only when its base still represents entry RSP. Microsoft x64 stack usage

My assembly course covers the stack, CALL and RET, and a separate example that calls a NASM sum function from C. Mastering 64-bit Assembly Language Programming with NASM and C Integration

Discussion 0

No comments yet. Start a thoughtful discussion.

Join the discussion

You need an account to contribute.

Sign in