This tutorial will cover how to convert a simple C++ project to Masm 32-bit in Visual Studio 2012. Some knowledge of assembly is required.
I'll start with where I left off from this tutorial. It's alot easier to convert when you don't have excess code.
We can have the compiler do most of the work for us, by going to the project Property Pages -> Configuration Properties -> C/C++ -> Output Files -> Assembler Output. I generally choose the "Assembly, Machine Code and Source (/FAcs)" option. After rebuilding the project once you selected this option, the compiler will generate .cod files in the solution\project\release folder. These can be opened in Visual Studio.
The compiler will generate something like this
; Listing generated by Microsoft (R) Optimizing Compiler Version 17.00.61030.0
00039 59 pop ecx
0003a c3 ret 0
Since there is no native Masm project type in Visual Studio, you have to use Visual C++. Create a new project and use Visual C++ -> Win32 Console Application (since we're converting a console app). Click Next, check the "Empty project" checkbox, and click finish. The project should be completely empty. Select the project name in the solution explorer and then, from the main menu, go to Project -> Build Customizations... (it's Custom Build Rules... in 2008). Check the masm checkbox and click OK.
In case that wasn't clear, here are some good tutorials on setting up masm.
From What's a Creel?
From Romaine Carter on scriptbucket
Make sure to copy the project linker properties from the C++ project to this one.
Linker -> Input -> Addtional Dependencies kernel32.lib
Linker -> Input -> Ignore All Default Libraries Yes (/NODEFAULTLIB)
Linker -> Advanced -> Entry Point main
Linker -> Debugging -> Generate Debug Info No
Linker -> Manifest File -> Generate Manifest No (/MANIFEST:NO)
Linker -> Advanced -> Randomized Base Address No (/DYNAMICBASE:NO)
Linker -> Advanced -> Fixed Base Address Yes (/FIXED)
Linker -> Advanced -> Merge Sections .rdata=.text
To add assembly files to the project, add a new item. Click the C++ file type, and give it a .asm extension instead of a .cpp extension. In this case, I named it main.asm.
If you didn't add masm before you added a .asm file. Right-click the .asm file go to Properties -> Configuration Properties -> General -> Item Type. Set it to Microsoft Macro Assembler and click OK.
Take a look at the main.cod again, we'll be using it as a template for most of the conversion.
I'll start with the .686P, and .model flat. Make sure to add a language after flat. I'll be using the C langtype.
Next is where you could add option casemap:noneif you're using the masm sdk. Since I'm keeping this relatively simply, I'll skip over that, and do the definitions.
#define STD_INPUT_HANDLE ((DWORD)-10)
#define STD_OUTPUT_HANDLE ((DWORD)-11)
Can be translated in masm as:
STD_INPUT_HANDLE equ -10
STD_OUTPUT_HANDLE equ -11
Imports are easy, you use EXTRN keyword, then the symbol name from the .lib, followed by :PROC. Imports from the window's 32-bit libraries often begin with _imp__ , and end with with @ followed by the number of bytes of parameters.
I prefer adding the align 16 in at the begining of each segment. This will align the data/instruction, under it, on a 16 byte boundary.
The .cod is a bit confusing when it comes to read only data, it has a CONST ENDS but no CONST SEGMENT to begin the segment. The _g_szHelloWorld is the name. DB (Define Byte) is the data type. 'Hello Masm World', 00H is the data, a ascii string with a null terminator.
Next up is the code section which is also known as the .text section in the PE. Following the example from the .cod, each procedure (function) gets its own comdat.
_dwUnused$ = -4 Here _dwUnused$ is basically set as a alias for -4. So when we come across DWORD PTR _dwUnused$[esp+8], it really means, DWORD PTR [esp+8-4] or DWORD PTR [esp+4]. This is where dwUnused is stored on the stack.
It's much easier to copy the esp address into another register before pushing parameters. This way, you don't need a lea ecx, dwUnused[esp+4].
Every .asm file needs to end with END. This can cause funky errors if you put it anywhere else.