This tutorial will cover how to convert a simple C++ project to Masm 64-bit in Visual Studio 2012. Some knowledge of assembly is required.


I'll start with this 64-bit app:

#define WIN32_LEAN_AND_MEAN
#include <Windows.h>


const char g_szHelloWorld[] = "Hello Masm64 World";


int main()
{
DWORD dwUnused;


//display Hello World
WriteFile(
GetStdHandle(STD_OUTPUT_HANDLE),
g_szHelloWorld,
sizeof(g_szHelloWorld),
&dwUnused,
NULL
);


//pause screen
ReadFile(
GetStdHandle(STD_INPUT_HANDLE),
&dwUnused, //doesn't matter what buffer we use
0, //because we aren't writing anything to it
&dwUnused,
NULL
);


return 0;
}



It uses project properties from this tutorial to make it smaller. It's alot easier to convert when you don't have excess code.


We can have the compiler do most of the work for us, by going to the project Property Pages -> Configuration Properties -> C/C++ -> Output Files -> Assembler Output. I generally choose the "Assembly, Machine Code and Source (/FAcs)" option. After rebuilding the project once you selected this option, the compiler will generate .cod files in the solution\project\x64\release folder. These can be opened in Visual Studio.


The compiler will generate something like this
Spoiler: main.cod
]
; ; Listing generated by Microsoft (R) Optimizing Compiler Version 17.00.61030.0 


include listing.inc


INCLUDELIB OLDNAMES


EXTRN __imp_GetStdHandle:PROC
EXTRN __imp_ReadFile:PROC
EXTRN __imp_WriteFile:PROC
g_szHelloWorld DB 'Hello Masm64 World', 00H
CONST ENDS
PUBLIC main
; COMDAT pdata
pdata SEGMENT
$pdata$main DD imagerel $LN4
DD imagerel $LN4+100
DD imagerel $unwind$main
pdata ENDS
; COMDAT xdata
xdata SEGMENT
$unwind$main DD 010401H
DD 06204H
xdata ENDS
; Function compile flags: /Ogtpy
; File c:\users\temp\programs\hellomasm64\hellomasm64\main.cpp
; COMDAT main
_TEXT SEGMENT
dwUnused$ = 64
main PROC ; COMDAT


; 7 : {


$LN4:
00000 48 83 ec 38 sub rsp, 56 ; 00000038H


; 8 : DWORD dwUnused;
; 9 :
; 10 : //display Hello World
; 11 : WriteFile(
; 12 : GetStdHandle(STD_OUTPUT_HANDLE),
; 13 : g_szHelloWorld,
; 14 : sizeof(g_szHelloWorld),
; 15 : &dwUnused,
; 16 : NULL
; 17 : );


00004 b9 f5 ff ff ff mov ecx, -11 ; fffffff5H
00009 ff 15 00 00 00
00 call QWORD PTR __imp_GetStdHandle
0000f 4c 8d 4c 24 40 lea r9, QWORD PTR dwUnused$[rsp]
00014 48 8d 15 00 00
00 00 lea rdx, OFFSET FLAT:g_szHelloWorld
0001b 41 b8 13 00 00
00 mov r8d, 19
00021 48 8b c8 mov rcx, rax
00024 48 c7 44 24 20
00 00 00 00 mov QWORD PTR [rsp+32], 0
0002d ff 15 00 00 00
00 call QWORD PTR __imp_WriteFile


; 18 :
; 19 :
; 20 : //pause screen
; 21 : ReadFile(
; 22 : GetStdHandle(STD_INPUT_HANDLE),
; 23 : &dwUnused, //doesn't matter what buffer we use
; 24 : 0, //because we aren't writing anything to it
; 25 : &dwUnused,
; 26 : NULL
; 27 : );


00033 b9 f6 ff ff ff mov ecx, -10 ; fffffff6H
00038 ff 15 00 00 00
00 call QWORD PTR __imp_GetStdHandle
0003e 4c 8d 4c 24 40 lea r9, QWORD PTR dwUnused$[rsp]
00043 48 8d 54 24 40 lea rdx, QWORD PTR dwUnused$[rsp]
00048 45 33 c0 xor r8d, r8d
0004b 48 8b c8 mov rcx, rax
0004e 48 c7 44 24 20
00 00 00 00 mov QWORD PTR [rsp+32], 0
00057 ff 15 00 00 00
00 call QWORD PTR __imp_ReadFile


; 28 :
; 29 : return 0;


0005d 33 c0 xor eax, eax


; 30 : }


0005f 48 83 c4 38 add rsp, 56 ; 00000038H
00063 c3 ret 0
main ENDP
_TEXT ENDS
END



Since there is no native Masm project type in Visual Studio, you have to use Visual C++. Create a new project and use Visual C++ -> Win32 Console Application (since we're converting a console app). Click Next, check the "Empty project" checkbox, and click finish. The project should be completely empty. Select the project name in the solution explorer and then, from the main menu, go to Project -> Build Customizations... (it's Custom Build Rules... in 2008). Check the masm checkbox and click OK.
Set it to 64-bit from the Configuration Manager, click the dropdown box under Platform. Set New platform to x64, and click OK, and click Close.


In case that wasn't clear, here is good tutorial on setting up masm in 64-bit.
From What's a Creel?


Make sure to copy the project linker properties from the C++ project to this one.
Linker -> Input -> Addtional Dependencies kernel32.lib
Linker -> Input -> Ignore All Default Libraries Yes (/NODEFAULTLIB)
Linker -> Advanced -> Entry Point main
Linker -> Manifest File -> Generate Manifest No (/MANIFEST:NO)
Linker -> Advanced -> Randomized Base Address No (/DYNAMICBASE:NO)
Linker -> Advanced -> Fixed Base Address Yes (/FIXED)
Linker -> Advanced -> Merge Sections .rdata=.text


To add assembly files to the project, add a new item. Click the C++ file type, and give it a .asm extension instead of a .cpp extension. In this case, I named it main.asm.


If you didn't add masm before you added a .asm file. Right-click the .asm file go to Properties -> Configuration Properties -> General -> Item Type. Set it to Microsoft Macro Assembler and click OK.


Take a look at the main.cod again, we'll be using it as a template for most of the conversion. You'll notice it does not have

TITLE C:\Users\temp\Desktop\HelloMasm\HelloMasm32\main.cpp
.686P
.XMM
.model flat

like it does in 32-bit. This is probably because there only one calling convention in x64.


#define STD_INPUT_HANDLE ((DWORD)-10)
#define STD_OUTPUT_HANDLE ((DWORD)-11)
Can be translated in masm as:
STD_INPUT_HANDLE equ -10
STD_OUTPUT_HANDLE equ -11


Imports are easy, you use EXTRN keyword, then the symbol name from the .lib, followed by :PROC. Imports from the window's 64-bit libraries often begin with __imp_.


I prefer adding the align 16 in at the begining of each segment. This will align the data/instruction, under it, on a 16 byte boundary.


The .cod is a bit confusing when it comes to read only data, it has a CONST ENDS but no CONST SEGMENT to begin the segment. The g_szHelloWorld is the name. DB (Define Byte) is the data type. 'Hello Masm64 World', 00H is the data, a ascii string with a null terminator.


The pdata, and xdata comdats are used with exception handling. Since we aren't using any, we can ignore them.


Next up is the code section which is also known as the .text section in the PE. Following the example from the .cod, each procedure (function) gets its own [URL="stackoverflow.com/questions/1834597/what-is-the-comdat-section-used-for"]comdat[\URL].
dwUnused$ = 64 Here dwUnused$ is basically set as a alias for 64. So when we come across QWORD PTR dwUnused$[rsp], it really means, QWORD PTR [rsp+64]. This is where dwUnused is stored on the stack.


In order to determine how much stack space we need, we start with 20h (32) for shadow space. Then we find the function that takes the most parameters (excluding floating point parameters). If we have any functions with five or more parameters, we subtract four from it, because those are passed via registers. Multiply it by eight and add the total bytes of local variables. Finally, add 8 to keep the stack 16-byte aligned.


We start with 20h (32). Both WriteFile, and ReadFile take five parameters, so we're going to need to add 8 bytes. Add another 4 bytes for dwUnused. Finally add 0Ch (12) to get 38h (56).


Every .asm file needs to end with END. This can cause funky errors if you put it anywhere else.


After some tweaking and simplification:

STD_INPUT_HANDLE equ -10
STD_OUTPUT_HANDLE equ -11


;imports
;kernel32
extern __imp_GetStdHandle:proc
extern __imp_ReadFile:proc
extern __imp_WriteFile:proc


CONST segment ;.const (.rdata)
align 16
g_szHelloWorld db "Hello Masm64 World",00h
CONST ends


_TEXT SEGMENT
dwUnused = 28h ;40
main PROC
sub rsp, 38h


mov ecx, STD_OUTPUT_HANDLE
call qword ptr __imp_GetStdHandle
;rax is the handle returned


mov qword ptr [rsp+20h], 0 ;NULL
lea r9, dwUnused[rsp] ;&dwUnused
mov r8d, 13h ;sizeof(g_szHelloWorld)
mov rdx, offset g_szHelloWorld
mov rcx, rax ;handle
call qword ptr __imp_WriteFile


mov ecx, STD_INPUT_HANDLE
call qword ptr __imp_GetStdHandle
;rax is the handle returned


mov qword ptr [rsp+20h], 0 ;NULL
lea r9, dwUnused[rsp] ;&dwUnused
xor r8d, r8d ;0
mov rdx, r9 ;&dwUnused
mov rcx, rax ;handle
call qword ptr __imp_ReadFile


xor eax, eax ;return 0
add rsp, 38h
ret
main endp
_TEXT ends
end



This can be optimized further, but this tutorial is meant to be simple.


Below is the solution I used in this tutorial.




λGhost