Converting C++ to Masm 64-bit In Visual Studio

  • This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.
  • ► You must register to download attachments!
Swift Games Accounts
Jun 4, 2014
This tutorial will cover how to convert a simple C++ project to Masm 64-bit in Visual Studio 2012. Some knowledge of assembly is required.

I'll start with this 64-bit app:
#include <Windows.h>

const char g_szHelloWorld[] = "Hello Masm64 World";

int main()
   DWORD dwUnused;

   //display Hello World

   //pause screen
      &dwUnused,  //doesn't matter what buffer we use
      0,          //because we aren't writing anything to it

   return 0;

It uses project properties from this tutorial to make it smaller. It's alot easier to convert when you don't have excess code.

We can have the compiler do most of the work for us, by going to the project Property Pages -> Configuration Properties -> C/C++ -> Output Files -> Assembler Output. I generally choose the "Assembly, Machine Code and Source (/FAcs)" option. After rebuilding the project once you selected this option, the compiler will generate .cod files in the solution\project\x64\release folder. These can be opened in Visual Studio.

The compiler will generate something like this
; ; Listing generated by Microsoft (R) Optimizing Compiler Version 17.00.61030.0 



EXTRN    __imp_GetStdHandle:PROC
EXTRN    __imp_ReadFile:PROC
EXTRN    __imp_WriteFile:PROC
g_szHelloWorld DB 'Hello Masm64 World', 00H
PUBLIC    main
;    COMDAT pdata
pdata    SEGMENT
$pdata$main DD    imagerel $LN4
    DD    imagerel $LN4+100
    DD    imagerel $unwind$main
pdata    ENDS
;    COMDAT xdata
xdata    SEGMENT
$unwind$main DD    010401H
    DD    06204H
xdata    ENDS
; Function compile flags: /Ogtpy
; File c:\users\temp\programs\hellomasm64\hellomasm64\main.cpp
;    COMDAT main
dwUnused$ = 64
main    PROC                        ; COMDAT

; 7    : {

  00000    48 83 ec 38     sub     rsp, 56            ; 00000038H

; 8    :    DWORD dwUnused;
; 9    : 
; 10   :    //display Hello World
; 11   :    WriteFile(
; 12   :       GetStdHandle(STD_OUTPUT_HANDLE),
; 13   :       g_szHelloWorld,
; 14   :       sizeof(g_szHelloWorld),
; 15   :       &dwUnused,
; 16   :       NULL
; 17   :    );

  00004    b9 f5 ff ff ff     mov     ecx, -11        ; fffffff5H
  00009    ff 15 00 00 00
    00         call     QWORD PTR __imp_GetStdHandle
  0000f    4c 8d 4c 24 40     lea     r9, QWORD PTR dwUnused$[rsp]
  00014    48 8d 15 00 00
    00 00         lea     rdx, OFFSET FLAT:g_szHelloWorld
  0001b    41 b8 13 00 00
    00         mov     r8d, 19
  00021    48 8b c8     mov     rcx, rax
  00024    48 c7 44 24 20
    00 00 00 00     mov     QWORD PTR [rsp+32], 0
  0002d    ff 15 00 00 00
    00         call     QWORD PTR __imp_WriteFile

; 18   : 
; 19   : 
; 20   :    //pause screen
; 21   :    ReadFile(
; 22   :       GetStdHandle(STD_INPUT_HANDLE),
; 23   :       &dwUnused,  //doesn't matter what buffer we use
; 24   :       0,          //because we aren't writing anything to it
; 25   :       &dwUnused,
; 26   :       NULL
; 27   :    );

  00033    b9 f6 ff ff ff     mov     ecx, -10        ; fffffff6H
  00038    ff 15 00 00 00
    00         call     QWORD PTR __imp_GetStdHandle
  0003e    4c 8d 4c 24 40     lea     r9, QWORD PTR dwUnused$[rsp]
  00043    48 8d 54 24 40     lea     rdx, QWORD PTR dwUnused$[rsp]
  00048    45 33 c0     xor     r8d, r8d
  0004b    48 8b c8     mov     rcx, rax
  0004e    48 c7 44 24 20
    00 00 00 00     mov     QWORD PTR [rsp+32], 0
  00057    ff 15 00 00 00
    00         call     QWORD PTR __imp_ReadFile

; 28   : 
; 29   :    return 0;

  0005d    33 c0         xor     eax, eax

; 30   : }

  0005f    48 83 c4 38     add     rsp, 56            ; 00000038H
  00063    c3         ret     0
main    ENDP

Since there is no native Masm project type in Visual Studio, you have to use Visual C++. Create a new project and use Visual C++ -> Win32 Console Application (since we're converting a console app). Click Next, check the "Empty project" checkbox, and click finish. The project should be completely empty. Select the project name in the solution explorer and then, from the main menu, go to Project -> Build Customizations... (it's Custom Build Rules... in 2008). Check the masm checkbox and click OK.
Set it to 64-bit from the Configuration Manager, click the dropdown box under Platform. Set New platform to x64, and click OK, and click Close.

In case that wasn't clear, here is good tutorial on setting up masm in 64-bit.
You do not have permission to view link Log in or register now.

Make sure to copy the project linker properties from the C++ project to this one.
Linker -> Input -> Addtional Dependencies kernel32.lib
Linker -> Input -> Ignore All Default Libraries Yes (/NODEFAULTLIB)
Linker -> Advanced -> Entry Point main
Linker -> Manifest File -> Generate Manifest No (/MANIFEST:NO)
Linker -> Advanced -> Randomized Base Address No (/DYNAMICBASE:NO)
Linker -> Advanced -> Fixed Base Address Yes (/FIXED)
Linker -> Advanced -> Merge Sections .rdata=.text

To add assembly files to the project, add a new item. Click the C++ file type, and give it a .asm extension instead of a .cpp extension. In this case, I named it main.asm.

If you didn't add masm before you added a .asm file. Right-click the .asm file go to Properties -> Configuration Properties -> General -> Item Type. Set it to Microsoft Macro Assembler and click OK.

Take a look at the main.cod again, we'll be using it as a template for most of the conversion. You'll notice it does not have
TITLE C:\Users\temp\Desktop\HelloMasm\HelloMasm32\main.cpp
.model flat
like it does in 32-bit. This is probably because there only one
You do not have permission to view link Log in or register now.

Can be translated in masm as:

Imports are easy, you use EXTRN keyword, then the symbol name from the .lib, followed by :pROC. Imports from the window's 64-bit libraries often begin with __imp_.

I prefer adding the align 16 in at the begining of each segment. This will align the data/instruction, under it, on a 16 byte boundary.

The .cod is a bit confusing when it comes to read only data, it has a CONST ENDS but no CONST SEGMENT to begin the segment. The g_szHelloWorld is the name. DB (Define Byte) is the data type. 'Hello Masm64 World', 00H is the data, a ascii string with a null terminator.

The pdata, and xdata comdats are used with exception handling. Since we aren't using any, we can ignore them.

Next up is the code section which is also known as the
You do not have permission to view link Log in or register now.
. Following the example from the .cod, each procedure (function) gets its own comdat[\URL].
dwUnused$ = 64 Here dwUnused$ is basically set as a alias for 64. So when we come across QWORD PTR dwUnused$[rsp], it really means, QWORD PTR [rsp+64]. This is where dwUnused is stored on the stack.

In order to determine how much stack space we need, we start with 20h (32) for shadow space. Then we find the function that takes the most parameters (excluding floating point parameters). If we have any functions with five or more parameters, we subtract four from it, because those are passed via registers. Multiply it by eight and add the total bytes of local variables. Finally, add 8 to

We start with 20h (32). Both WriteFile, and ReadFile take five parameters, so we're going to need to add 8 bytes. Add another 4 bytes for dwUnused. Finally add 0Ch (12) to get 38h (56).

Every .asm file needs to end with END. This can cause funky errors if you put it anywhere else.

After some tweaking and simplification:

extern __imp_GetStdHandle:proc
extern __imp_ReadFile:proc
extern __imp_WriteFile:proc

CONST segment ;.const  (.rdata)
align 16
g_szHelloWorld    db "Hello Masm64 World",00h
CONST ends

dwUnused = 28h    ;40
main    PROC
   sub   rsp, 38h 

   mov   ecx, STD_OUTPUT_HANDLE
   call  qword ptr __imp_GetStdHandle
   ;rax is the handle returned

   mov   qword ptr [rsp+20h], 0 ;NULL
   lea   r9, dwUnused[rsp] ;&dwUnused
   mov   r8d, 13h          ;sizeof(g_szHelloWorld)
   mov   rdx, offset g_szHelloWorld
   mov   rcx, rax          ;handle
   call  qword ptr __imp_WriteFile

   mov   ecx, STD_INPUT_HANDLE
   call  qword ptr __imp_GetStdHandle
   ;rax is the handle returned

   mov   qword ptr [rsp+20h], 0 ;NULL
   lea   r9, dwUnused[rsp] ;&dwUnused
   xor   r8d, r8d          ;0
   mov   rdx, r9           ;&dwUnused
   mov   rcx, rax          ;handle
   call  qword ptr __imp_ReadFile

   xor   eax, eax ;return 0
   add   rsp, 38h
main endp
_TEXT ends

This can be optimized further, but this tutorial is meant to be simple.

Below is the solution I used in this tutorial.