Tutorial Understanding Windows SysCalls - SysCall Dumper

Hexui Undetected CSGO Cheats Sinkicheat PUBG Cheat

timb3r

Semi-Retired
Dank Tier VIP
Jul 15, 2018
767
22,668
47
syscall-dumper.png


Coding my DOS COM Infector a few weeks ago made me remember the simplicity and elegance of writing code down at the lowest level using assembler. It also got me thinking about the structure of the modern Windows Operating System.

How does modern Windows switch from user mode to kernel mode? I knew Windows must have type of syscalls available to user mode applications but I wasn’t able to find any documentation on the subject. In-fact the sheer lack of information made me think that Microsoft had gone out of it’s way to hide these details from me. But that’s okay I am a reverse engineer at heart: so ripping into the undocumented parts of Windows doesn’t both me too much.

Understanding Syscalls:

Depending on your background you may already be aware of the security basics of a modern operating system. User mode memory is segmented from kernel mode this done through the use of privilege rings provided by the Intel x86 architecture. There are four rings available on this architecture starting at Ring 0 (most privileged) and ending at Ring 3 (least privileged). The kernel is typically runs in Ring 0 and user mode applications run in Ring 3.

As this is enforced at hardware level any attempts by Ring 3 code to access higher privilege levels will result in a general protection fault. However user mode code still needs to access functionality provided by the operating system (file IO, networking, etc) and to do this it needs to be able to run code in a different ring.

The way it does this is through the use of the SYSCALL instruction. The operating system can expose specific interfaces to user mode applications this way. On Operating Systems like Linux these syscalls are documented and well known; however on Windows these mysterious syscalls are no where to be found.

Locating the Syscalls:

We know for a fact these syscalls are hiding in user mode somewhere the question is where. I had my theories but the easiest way to locate them was to write a simple application and trace it’s execution from user mode to kernel mode.

Long story short the syscalls were hiding in ntdll.dll. Opening up the DLL shows precisely what we’re looking for:

ntopenprocess.png

Gotem: NtOpenProcess()

As you can see from the picture above the parameter is passed in via RCX (into R10). The syscall number is moved into the EAX register (meaning it’s a 32 bit value) then the SYSCALL is executed 0xF505 and a RET instruction follows. If you browse through NTDLL you’ll see this pattern repeat exactly through the only value that changes is the value passed to EAX.

Rip and Tear

Gee willikers Mister it sure would be nice if we could dump this information out. Well we can by observing that each syscall follows the exact same pattern we can easily parse NTDLL and extract this information.

The code provided below will do exactly that:
C++:
#include <stdio.h>
#include <windows.h>

int main(void)
{
    PIMAGE_DOS_HEADER pDosHeader = (PIMAGE_DOS_HEADER)GetModuleHandle("ntdll.dll");
    PIMAGE_NT_HEADERS pNtHeader = (PIMAGE_NT_HEADERS)((LPBYTE)pDosHeader + pDosHeader->e_lfanew);

    // Invalid file exit
    if (pDosHeader->e_magic != IMAGE_DOS_SIGNATURE || pNtHeader->Signature != IMAGE_NT_SIGNATURE)
        return -1;
 
    PIMAGE_EXPORT_DIRECTORY pExportDirectory = (PIMAGE_EXPORT_DIRECTORY)((LPBYTE)pDosHeader + pNtHeader->OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_EXPORT].VirtualAddress);
    if (!pExportDirectory)
        return -1;

    PDWORD dwAddress = (PDWORD)((LPBYTE)pDosHeader + pExportDirectory->AddressOfFunctions);
    PDWORD dwName    = (PDWORD)((LPBYTE)pDosHeader + pExportDirectory->AddressOfNames);
    PWORD dwOrdinal = (PWORD)((LPBYTE)pDosHeader + pExportDirectory->AddressOfNameOrdinals);

    unsigned char pBuf[32] = { 0 };
    const unsigned char pSig[4] = { 0x4C, 0x8B, 0xD1, 0xB8 };

    printf("SYSCALL    ADDRESS      FUNCTION\n");
    printf("-----------------------------------------\n");

    for (DWORD i = 0; i < pExportDirectory->NumberOfFunctions; i++)
    {
        memset(&pBuf, 0, 32);
        PVOID pAddr = (PVOID)((LPBYTE)pDosHeader + dwAddress[dwOrdinal[i]]);
        char *szName = (char*)pDosHeader + dwName[i];

        memcpy(&pBuf, pAddr, 32);

        if (!pAddr || !szName)
            break;

        for (int x = 0; x < sizeof(pSig); x++)
        {
            if (pBuf[x] != pSig[x])
                break;
         
            if (x == sizeof(pSig) - 1) {
                printf("0x%02X\t   %p\t%s\n", pBuf[4], pAddr, szName);
            }
        }
    }
}
This code will dump any function that NTDLL exports that begins with the following code:

4C:8B D1 mov r10, rcx
B8 000000000 mov eax, <number>
Our code will check for this pattern and extract the syscall number:
syscall-dumper.png

That’s cool and all but why?

There’s not too many reasons why you’d want to perform a direct syscall on Windows. Microsoft have decoupled their API from the syscall instruction for a reason: it allows them to change things. Imagine if you wrote your code for an earlier version of Windows and then following a patch they decided to swap the syscalls of OpenFile and DeleteFile! That would cause some serious chaos as programmers frantically tried to update their code.

However there are some perfectly valid reasons for using them; IF you’re prepared to do a lot of extra testing have safeguards in place in case something does change. Off the top of my head here’s some reasons why you might desire to use syscalls over conventional API:

  • To obfuscate or mask the functionality of your code.
  • To harden your code against hooking. Pretty hard to hook something if your application switches directly to kernel mode without an intermediary library like ntdll.
  • Exploit development. Syscalls seem like a perfect choice for exploitation development as exploits often target a specific version of Windows anyway.
  • Finally malware could leverage syscalls to make it less obvious the file is malicious. The code could be compiled with non malicious syscalls then modified in memory on run-time when certain conditions are met.
Finally here’s a small program in x64 Assembler that demonstrates how to use syscalls:

Code:
; Calling Syscall 7fh (NtGetTickCount) via x64 bit Assembly
; by Timb3r 2019 - https://gamephreakers.com
;
; nasm.exe -f win64 main.asm -o main.obj                                                                                  
; gcc.exe -o syscall.exe main.obj                            
;
global main                                                                                                                  
extern printf                                                                                                                

section .data                                                                                                                
        szMsg: db "NtGetTickCount: %lld",0                                                                                  

section .text                                                                                                                
    main:                                                                                                                    
        push    rbp                                                                                                  
        mov     rbp,rsp                                                                                                      
        sub     rsp,20h                                                                                              
                                                                                                                             
        mov     rcx, 0                                                                                                  
        mov     r10, rcx                                                                                                
        mov     eax, 7fh ; NtGetTickCount                                                                                
        syscall                                                                                                      

        mov     rcx, szMsg                                                                                                  
        mov     rdx, rdi                                                                                            

        call    printf                                                                                                      

        add     rsp,20h                                                                                                      
        mov     rsp,rbp                                                                                                      
        pop     rbp                                                                                                          
        ret
If this interests you, checkout this thread also
https://guidedhacking.com/threads/undocumented-windows-functions-structures.14438/
 
Last edited by a moderator:

0xDEC0DE

dead
Dank Tier VIP
Fleep Tier Donator
Dank Tier Donator
Oct 28, 2018
449
18,798
92
Great writeup, i made it sticky as i think its useful for others.
 

Rake

Cesspool Admin
Administrator
Jan 21, 2014
11,573
78,998
2,316
Very dope, thanks for sharing, gonna add it to the undocumented windows thread I made
 
  • Like
Reactions: timb3r
Attention! Before you post:

Read the How to Ask Questions Guide
99% of questions are answered in the Beginner's Guide, do it before asking a question.

No Hack Requests. Post in the correct section.  Search the forum first. Read the rules.

How to make a good post:

  • Fill out the form correctly
  • Tell us the game name & coding language
  • Post everything we need to know to help you
  • Ask specific questions, be descriptive
  • Post errors, line numbers & screenshots
  • Post code snippets using code tags
  • If it's a large project, zip it up and attach it

If you do not comply, your post may be deleted.  We want to help, please make a good post and we will do our best to help you.

Community Mods League of Legends Accounts