Guide GHB2 - Beginners Guide To Reverse Engineering

[XdarionX]

still more and more games and software are coded in object-oriented cpp (that means more inheriting and virt funcs) and reversing cpp has few hidden tweaks that if you know you make your life easier. This two part guide interested me (really good info, simple cpp code explained):
https://alschwalm.com/blog/static/2016/12/17/reversing-c-virtual-functions/
https://alschwalm.com/blog/static/2017/01/24/reversing-c-virtual-functions-part-2-2/
for those who like more books:
https://www.blackhat.com/presentations/bh-dc-07/Sabanal_Yason/Paper/bh-dc-07-Sabanal_Yason-WP.pdf

 

[h4nsbr1x]

These videos are useful and fun to watch. I've got some comments on function prologues and epilogues and some other asm stuff (from IDA tutorial 4)

Why are they pushing/popping all this stuff?

When you call a function you can expect some things to be left untouched (e.g. the stack, your wallet, your girlfriend) and other things are going to be used without your consent (e.g. memory south of the stack, Rake's mom). The convention for 32-bit apps is as follows:

EAX, ECX, EDX: assume these get overwritten after the call (EAX in particular is where the return value gets stored)
EBP, ESP, ESI, EDI: EBX: these must be preserved by the function

Argument Passing and Naming Conventions

Why these registers? Fun fact, in 16-bit days (pre-286) you could only dereference pointers in the BP, SP, SI, DI and BX registers (BX is the "general" register that can also be used for pointing to things), and it's expected that people would leave pointers lying around, whereas the others are fair game.

In 64-bit apps the convention is as follows:

RAX, RCX, RDX, R8, R9, R10, R11, XMM0-5, YMM0-5: assume these get overwritten (RAX is the return value, RCX/RDX/R8/R9 are your args)
RBP, RSP, RSI, RDI, RBX, R12, R13, R14, R15: these must be preserved by the function

x64 software conventions

So here's the typical prologue convention:

; prologue
push rbp ; save rbp (note this is more or less the same in 32 bit too)
mov rbp, rsp ; save rsp
sub rsp, 100 ; create space for local variables
and spl, 0xF0 ; 16-byte align
push rdi ; save if we use it
push rsi ; save if we use it
; ......   the compiler is only going to save registers if it wants to use them

And the epilogue matches up

pop rsi ; put it back
pop rdi ; tidy your room
mov rsp, rbp ; we did weird stuff to RSP so it's easier to copy from RBP
pop rbp ; put RBP back too

What is the "AND ESP, 0xFFFFFFF0 for"

In the tutorial there's this little line in the prologue that does an AND command on ESP:

and esp, 0xFFFFFFF0

In x64 it looks like this

and spl, 0xF0

These both do the same thing, they round the stack pointer down to the nearest 16 (RSP % 16 in C if you like). The reason they do this is because the processor can only read/write on aligned boundaries, whatever that means. In other words, if you want to read 4 bytes from 0x1230 that's fine, you can do that with a single MOV EAX, [0x1230], but if you try to read from 0x1231 it'll throw an exception when you do MOV EAX, [0x1231] (go and try it, I'll wait here). x32 has 8 byte FPU registers, and x64 has 16 byte XMM registers, so if your stack isn't aligned to the nearest 16 bytes there are some functions that will break when you call them and you'll lose hours trying to figure out what's wrong.

Why are ESI/RSI and EDI/RDI special?

Short answer is that it goes back to 8086 days. Long answer is that there are some optimised commands that use these. Here's some examples:

LODSB ; loads a byte from ESI/RSI into AL and increments it by one (char a = *(rsi++))
STOSB ; stores the byte from AL into ESI/RSI and increments it by one (*(rdi++) = a)
LODSW ; as above, but with words
LODSD
LOSDQ
STOSD
...
MOVSB ; this moves a byte from ESI/RSI to EDI/RDI and increments both by one
REP MOVSB ; this does a MOVSB, then decrements ECX/RCX, and if ECX/RCX isn't zero then it'll loop back on itself

You don't *have* to use these registers as you can dereference any pointer as a register, but the helper functions make these very useful. Here's a memcpy example:

lea rsi, input
lea rdi, output
mov rcx, input_length
rep movsb

As a result, you'll often find RSI/RDI pointing to strings or structures (or more likely, to the ends of them as they've probably recently been copied).

Other x64 craziness

Make yourself familiar with the x64 calling convention for windows: x64 calling convention

This confuses the shit out of me. In x32 you'd push all your arguments onto the stack in reverse order. In x64, you put your first 4 arguments in RCX,RDX,R8,R9, your other arguments then get pushed *in reverse* onto the stack. You also need to have already created 4 QWORDS of space on the stack because the called function is going to use that space and destroy your local variables otherwise. You also need to unwind the stack yourself (in x86 the stdcall convention does this for you).

So in win32 you'd do this:

push 0 ; arg 7
push FILE_ATTRIBUTE_NORMAL ; arg 6
push OPEN_EXISTING ; arg 5
push 0 ; arg 4
push 0 ; arg 3
push GENERIC_READ ; arg 2
lea eax, filename
push eax ; arg 1
call CreateFileA ; in MASM you can just do invoke CreateFileA, arg1, arg2 etc and that'll build this for you

Easy, no stack prep, no stack tidy. In x64 you have to do this:

; we need to create stack space for our first 4 args
; 4 * 0x8 = 0x20
; BUT we push 3 more args
; that's 0x38 which doesn't line up to 16 (0x10) bytes
; so we need to subtract 0x28 from RSP
; then when we push our last 3 args it'll round off to 0x40
; and halloween bill gates and steve ballmer won't eat us
lea rsp, [rsp-0x28] ; uses less bytes than sub rsp, 0x28
lea rcx, filename ; arg 1
mov rdx, GENERIC_READ ; arg 2
mov r8, 0 ; arg 3
mov r9, 0 ; arg 4
push 0 ; arg 7, stack is now 0x30 lower
push FILE_ATTRIBUTE_NORMAL ; arg 6, stack is 0x38 lower
push OPEN_EXISTING ; arg 5, stack is 0x40 lower
call CreateFileA
lea rsp, [rsp+0x40] ; reset stack

Doable, but annoying, and gives super weird bugs if you get something wrong.

Happy assembling kids!

 

[Rake]

@h4nsbr1x That's a great writeup, I've pushed your post to the first page in the second post so more people will see it when reading this guide, thanks

 

[timb3r]

@h4nsbr1x

 

[Rake]

If you want to learn reverse engineering, you need to understand it takes thousands of hours to get good at. And even when you're an expert, it can take hundreds of hours in a disassembler to actually achieve a task. People are incredibly naïve about how time consuming it is.