Solved Need help to call TraceLine correctly

Hexui Undetected CSGO Cheats Sinkicheat PUBG Cheat

Erarnitox

🐅
Meme Tier VIP
Trump Tier Donator
May 11, 2018
151
5,023
3
Anticheat
none
How long you been coding/hacking?
some years
Coding Language
C++


The solution was rather "easy" as always. It looks like the compiler optimized the passing of the first 2 arguments through registers, so to solve that you have to pass the first two arguments trhough registers as well, you can do that in an __asm block.
so for me the first two arguments got passed through edx and ecx, after setting these 2 registers, everything worked fine. :)



Hey, ive been at this function for some hours now and i can't seem to figure out how to call it correctly.
The function is the intersectclosest function ftom the game Cube2:Sauerbraten. Luckily this game is open source, so here is the source of the function:
intesectclosest:
 dynent *intersectclosest(const vec &from, const vec &to, fpsent *at, float &bestdist)
    {
        dynent *best = NULL;
        bestdist = 1e16f;
        loopi(numdynents())
        {
            dynent *o =
            (i);
            if(o==at || o->state!=CS_ALIVE) continue;
            float dist;
            if(!intersect(o, from, to, dist)) continue;
            if(dist<bestdist)
            {
                best = o;
                bestdist = dist;
            }
        }
        return best;
    }
You might say: easy _stdcall and make a typedef with the exact same arguments.
But that didnt work.

When i debug the function call in CheatEngine it heavily suggests a _cdecl call with 2 arguments:
86.png


As we can see 2 arguments get pushed to the stack and the caller cleans the arguments by 'add esp, 0x8' which removes 2 times a 4byte value from the stack.

However calling it like that doesnt seem to work either.
Cheat engine suggests this signature:

84.png


which honestly doesnt make sence to me.

I made a small program to test calling the function and it kinda works however im trying through trial and error how to use the parameters.
Because either i get hits only when very close to the enemy, none or always hit myself.

The only thing that is set is that the 2nd parameter has to be distance, even if the source would suggest otherwise.

if it helps here is the code i use to test calling the function:
Internal Traceline Caller:
#include <iostream>
#include <string>
#include <tchar.h>

#include "internal.h"
#include "Player.h"

struct Vector3{
    float x, y, z;
};

Player* player;

typedef Player* (_cdecl* tTraceLine)(void* null, float &distance, Player* from, Vector3 &to); //typedef of the function
tTraceLine TraceLine{ nullptr }; //create a function of that type

DWORD WINAPI InternalMain(HMODULE hMod) {
#ifdef _DEBUG
    AllocConsole();
    FILE* f;
    freopen_s(&f, "CONOUT$", "w", stdout);
#endif

    wchar_t gameName[]{ L"sauerbraten.exe" };
    size_t playerOffset{ 0x2074A4 }; //in Cheat Engine we found this static address of the player pointer (sauerbraten.exe + 0x2074A4)
    uintptr_t moduleBase = GetModuleBaseAddress(gameName); //get the address where the .exe got loaded inside our games process
    player = *reinterpret_cast<Player**>(moduleBase + playerOffset); //where the pointer to our player is located

    TraceLine = (tTraceLine) (moduleBase + 0x18E890); //assign the originals functions location to the created function

    Player* hitEnt = nullptr;
    float distance{ 1337.0f };
    Vector3 from{ player->hPosX, player->hPosY, player->hPosZ };
    Vector3 to{ 10000.0f, 0.0f, 0.0f };

    while (true){
        hitEnt = TraceLine(player, distance, player, from); //call the function
      
        if (GetAsyncKeyState(VK_END) & 1) {
            break;
        }

        if (hitEnt) {
            std::cout << hitEnt->posY << std::endl;
            std::cout << distance << std::endl;
        }
        else {
            std::cout << "Nothing was hit" << std::endl;
            std::cout << distance << std::endl;
        }
        Sleep(200);
    }


#ifdef _DEBUG //Close our console
    if(f != nullptr) fclose(f);
    FreeConsole();
#endif
    //eject our library
    FreeLibraryAndExitThread(hMod, 0);
    return 0;
}

BOOL APIENTRY DllMain( HMODULE hModule, DWORD  ul_reason_for_call, LPVOID lpReserved){
    switch (ul_reason_for_call){
    case DLL_PROCESS_ATTACH:
        //start a new thread for our cheat to run in so this thread can return to the injector
        HANDLE tHndl = CreateThread(nullptr, 0, (LPTHREAD_START_ROUTINE)InternalMain, hModule, 0, 0);
        if (tHndl) CloseHandle(tHndl); //if we where able to create a thread, close the handle to it
        else return FALSE; //if we where not able to create a thread, return false
        break;
    }
    return TRUE; //everything went fine
}
Would be glad if someone had an idea of what im doing wrong.
Maybe im still using the wrong calling convention or messed something else up?

Apprechiate any input, hints and ideas.
Thanks :)
 
Last edited:

Erarnitox

🐅
Meme Tier VIP
Trump Tier Donator
May 11, 2018
151
5,023
3
The solution was rather "easy" as always. It looks like the compiler optimized the passing of the first 2 arguments through registers, so to solve that you have to pass the first two arguments trhough registers as well, you can do that in an __asm block.
so for me the first two arguments got passed through edx and ecx, after setting these 2 registers, everything worked fine. :)
 

Petko123

Biggest paster
Dank Tier VIP
Trump Tier Donator
Feb 19, 2018
126
9,068
8
This looks like a __fastcall to me Msdn fastcall. I know that it looks like weird with add esp, 8 after the call, when it says that in __fastcall callee cleans the stack with poping the arguments and ret. Check inside that function if it pops the arguments and cleans it with ret. If it does that it's most likely a __fastcall.
 
  • Like
Reactions: Rake and Erarnitox

Rake

Cesspool Admin
Administrator
Jan 21, 2014
12,367
78,998
2,414
i tried to use __fastcall with 4 arguments, however that didnt seem to work. What im i doing wrong? However having it as __cdecl and passing the first 2 arguments through register works just fine.
Also when you use __fastcall the Callee has to clean the arguments which is not the case here. Im pretty certain the Caller cleans up.
Yeah you are correct, you have to use inline asm. It's wierd, it's args are identical to fastcall but the stack cleanup/setup isn't

ida gives:

C++:
game::monster *__usercall game__intersectclosest@<eax>(vec *from@<ecx>, vec *to@<edx>, game::monster *a3, float *a4)
which you think, you can change to:

C++:
game::monster *__fastcall game__intersectclosest(vec *from( ECX), vec *to (EDX), game::monster *a3, float *a4)
But then the stack isn't compatible with that function prototype

So IDA was correct in saying it's a __usercall, which typically means you need to use inline ASM
 
  • Like
Reactions: Erarnitox

Erarnitox

🐅
Meme Tier VIP
Trump Tier Donator
May 11, 2018
151
5,023
3
This looks like a __fastcall to me Msdn fastcall. I know that it looks like weird with add esp, 8 after the call, when it says that in __fastcall callee cleans the stack with poping the arguments and ret. Check inside that function if it pops the arguments and cleans it with ret. If it does that it's most likely a __fastcall.
it doesnt clean the arguments inside the function but i will try the __fastcall calling convention as i didnt try that one yet

Edit: its definetly not a __fastcall . Moving the player pointer into ECX before calling the function suggests a __thiscall, however since the arguments dont get cleaned up by the function and sauerbraten doesnt really use object oriented programming highly suggests a __cdecl calling cnvention and im 99% certain that is the case.

I am also able to call the function with the __cdecl calling convention, however the the order in which the arguments are pushed onto the stack and the return values i get back from the function dont make too much sence to me yet.

From my understanding the 2nd argument is the distance the ray traveled until it hit something.
The return value is the entity it hit.
It takes possibly these 4 arguments: a pointer to write the distance to as a float, 2 vector3 pointers (which are world positions to cast a ray from and to), and the player pointer or null.
 
Last edited:

Erarnitox

🐅
Meme Tier VIP
Trump Tier Donator
May 11, 2018
151
5,023
3
Check what ida says regarding the call and its args if you check the function inthere @Erarnitox
i did try ida, but the result was not helpful. I dont have ida pro only the free version. Also it might be helpful to note that the Visusal studio debugger detects the function pointer as a pointer to a function whith this signature:
void* func(const vec* a, const vec* b, void* maybePlayer, float* distance);
 

Petko123

Biggest paster
Dank Tier VIP
Trump Tier Donator
Feb 19, 2018
126
9,068
8
Maybe you can provide us screenshot of that function you're trying to call. Most important start and finish of it.
 

Erarnitox

🐅
Meme Tier VIP
Trump Tier Donator
May 11, 2018
151
5,023
3
Maybe you can provide us screenshot of that function you're trying to call. Most important start and finish of it.
sure:
Cheat Engine Disassembly of the function:
sauerbraten.game::intersectclosest - 83 EC 2C              - sub esp,2C { 44 }
sauerbraten.game::intersectclosest+3- 8B 44 24 34           - mov eax,[esp+34]
sauerbraten.game::intersectclosest+7- 53                    - push ebx
sauerbraten.game::intersectclosest+8- 55                    - push ebp
sauerbraten.game::intersectclosest+9- 56                    - push esi
sauerbraten.game::intersectclosest+A- C7 00 CA1B0E5A        - mov [eax],5A0E1BCA { 1510874058 }
sauerbraten.game::intersectclosest+10- 8B 2D B4CD8E00        - mov ebp,[sauerbraten.game::monsters+8] { (205) }
sauerbraten.game::intersectclosest+16- 8B 35 C0CD8E00        - mov esi,[sauerbraten.game::movables+8] { (0) }
sauerbraten.game::intersectclosest+1C- 8B 1D 3CCD8E00        - mov ebx,[sauerbraten.game::players+8] { (1) }
sauerbraten.game::intersectclosest+22- 57                    - push edi
sauerbraten.game::intersectclosest+23- 89 54 24 18           - mov [esp+18],edx
sauerbraten.game::intersectclosest+27- 33 FF                 - xor edi,edi
sauerbraten.game::intersectclosest+29- 8D 04 2E              - lea eax,[esi+ebp]
sauerbraten.game::intersectclosest+2C- 89 4C 24 10           - mov [esp+10],ecx
sauerbraten.game::intersectclosest+30- 33 D2                 - xor edx,edx
sauerbraten.game::intersectclosest+32- 03 C3                 - add eax,ebx
sauerbraten.game::intersectclosest+34- 89 54 24 14           - mov [esp+14],edx
sauerbraten.game::intersectclosest+38- 85 C0                 - test eax,eax
sauerbraten.game::intersectclosest+3A- 0F8E FC000000         - jng sauerbraten.game::intersectclosest+13C
sauerbraten.game::intersectclosest+40- 3B FB                 - cmp edi,ebx
sauerbraten.game::intersectclosest+42- 7D 0A                 - jnl sauerbraten.game::intersectclosest+4E
sauerbraten.game::intersectclosest+44- A1 34CD8E00           - mov eax,[sauerbraten.game::players] { (0102B108) }
sauerbraten.game::intersectclosest+49- 8B 34 B8              - mov esi,[eax+edi*4]
sauerbraten.game::intersectclosest+4C- EB 28                 - jmp sauerbraten.game::intersectclosest+76
sauerbraten.game::intersectclosest+4E- 8B CF                 - mov ecx,edi
sauerbraten.game::intersectclosest+50- 2B CB                 - sub ecx,ebx
sauerbraten.game::intersectclosest+52- 3B CD                 - cmp ecx,ebp
sauerbraten.game::intersectclosest+54- 7D 0A                 - jnl sauerbraten.game::intersectclosest+60
sauerbraten.game::intersectclosest+56- A1 ACCD8E00           - mov eax,[sauerbraten.game::monsters] { (0F82DD78) }
sauerbraten.game::intersectclosest+5B- 8B 34 88              - mov esi,[eax+ecx*4]
sauerbraten.game::intersectclosest+5E- EB 12                 - jmp sauerbraten.game::intersectclosest+72
sauerbraten.game::intersectclosest+60- 2B CD                 - sub ecx,ebp
sauerbraten.game::intersectclosest+62- 3B CE                 - cmp ecx,esi
sauerbraten.game::intersectclosest+64- 7D 0A                 - jnl sauerbraten.game::intersectclosest+70
sauerbraten.game::intersectclosest+66- A1 B8CD8E00           - mov eax,[sauerbraten.game::movables] { (0) }
sauerbraten.game::intersectclosest+6B- 8B 34 88              - mov esi,[eax+ecx*4]
sauerbraten.game::intersectclosest+6E- EB 02                 - jmp sauerbraten.game::intersectclosest+72
sauerbraten.game::intersectclosest+70- 33 F6                 - xor esi,esi
sauerbraten.game::intersectclosest+72- 8B 4C 24 10           - mov ecx,[esp+10]
sauerbraten.game::intersectclosest+76- 3B 74 24 40           - cmp esi,[esp+40]
sauerbraten.game::intersectclosest+7A- 0F84 A4000000         - je sauerbraten.game::intersectclosest+124
sauerbraten.game::intersectclosest+80- 80 7E 7C 00           - cmp byte ptr [esi+7C],00 { 0 }
sauerbraten.game::intersectclosest+84- 0F85 9A000000         - jne sauerbraten.game::intersectclosest+124
sauerbraten.game::intersectclosest+8A- F3 0F7E 06            - movq xmm0,[esi]
sauerbraten.game::intersectclosest+8E- 8B 46 08              - mov eax,[esi+08]
sauerbraten.game::intersectclosest+91- 66 0FD6 44 24 2C      - movq [esp+2C],xmm0
sauerbraten.game::intersectclosest+97- F3 0F7E C0            - movq xmm0,xmm0
sauerbraten.game::intersectclosest+9B- 8B 54 24 18           - mov edx,[esp+18]
sauerbraten.game::intersectclosest+9F- 89 44 24 34           - mov [esp+34],eax
sauerbraten.game::intersectclosest+A3- 66 0FD6 44 24 20      - movq [esp+20],xmm0
sauerbraten.game::intersectclosest+A9- F3 0F10 44 24 34      - movss xmm0,[esp+34]
sauerbraten.game::intersectclosest+AF- F3 0F5C 46 54         - subss xmm0,[esi+54]
sauerbraten.game::intersectclosest+B4- 89 44 24 28           - mov [esp+28],eax
sauerbraten.game::intersectclosest+B8- 8D 44 24 1C           - lea eax,[esp+1C]
sauerbraten.game::intersectclosest+BC- 50                    - push eax
sauerbraten.game::intersectclosest+BD- 51                    - push ecx
sauerbraten.game::intersectclosest+BE- 8D 44 24 28           - lea eax,[esp+28]
sauerbraten.game::intersectclosest+C2- F3 0F11 44 24 3C      - movss [esp+3C],xmm0
sauerbraten.game::intersectclosest+C8- F3 0F10 46 58         - movss xmm0,[esi+58]
sauerbraten.game::intersectclosest+CD- F3 0F58 44 24 30      - addss xmm0,[esp+30]
sauerbraten.game::intersectclosest+D3- F3 0F11 44 24 30      - movss [esp+30],xmm0
sauerbraten.game::intersectclosest+D9- F3 0F10 46 50         - movss xmm0,[esi+50]
sauerbraten.game::intersectclosest+DE- F3 0F11 04 24         - movss [esp],xmm0
sauerbraten.game::intersectclosest+E3- 50                    - push eax
sauerbraten.game::intersectclosest+E4- 8D 44 24 38           - lea eax,[esp+38]
sauerbraten.game::intersectclosest+E8- 50                    - push eax
sauerbraten.game::intersectclosest+E9- E8 2231FEFF           - call sauerbraten.linecylinderintersect
sauerbraten.game::intersectclosest+EE- 83 C4 10              - add esp,10 { 16 }
sauerbraten.game::intersectclosest+F1- 84 C0                 - test al,al
sauerbraten.game::intersectclosest+F3- 74 2B                 - je sauerbraten.game::intersectclosest+120
sauerbraten.game::intersectclosest+F5- 8B 44 24 44           - mov eax,[esp+44]
sauerbraten.game::intersectclosest+F9- F3 0F10 4C 24 1C      - movss xmm1,[esp+1C]
sauerbraten.game::intersectclosest+FF- F3 0F10 00            - movss xmm0,[eax]
sauerbraten.game::intersectclosest+103- 0F2F C1               - comiss xmm0,xmm1
sauerbraten.game::intersectclosest+106- 76 18                 - jna sauerbraten.game::intersectclosest+120
sauerbraten.game::intersectclosest+108- 8B D6                 - mov edx,esi
sauerbraten.game::intersectclosest+10A- F3 0F11 08            - movss [eax],xmm1
sauerbraten.game::intersectclosest+10E- 8B 1D 3CCD8E00        - mov ebx,[sauerbraten.game::players+8] { (1) }
sauerbraten.game::intersectclosest+114- 8B 2D B4CD8E00        - mov ebp,[sauerbraten.game::monsters+8] { (205) }
sauerbraten.game::intersectclosest+11A- 89 54 24 14           - mov [esp+14],edx
sauerbraten.game::intersectclosest+11E- EB 04                 - jmp sauerbraten.game::intersectclosest+124
sauerbraten.game::intersectclosest+120- 8B 54 24 14           - mov edx,[esp+14]
sauerbraten.game::intersectclosest+124- 8B 35 C0CD8E00        - mov esi,[sauerbraten.game::movables+8] { (0) }
sauerbraten.game::intersectclosest+12A- 47                    - inc edi
sauerbraten.game::intersectclosest+12B- 8D 0C 2E              - lea ecx,[esi+ebp]
sauerbraten.game::intersectclosest+12E- 03 CB                 - add ecx,ebx
sauerbraten.game::intersectclosest+130- 3B F9                 - cmp edi,ecx
sauerbraten.game::intersectclosest+132- 8B 4C 24 10           - mov ecx,[esp+10]
sauerbraten.game::intersectclosest+136- 0F8C 04FFFFFF         - jl sauerbraten.game::intersectclosest+40
sauerbraten.game::intersectclosest+13C- 5F                    - pop edi
sauerbraten.game::intersectclosest+13D- 5E                    - pop esi
sauerbraten.game::intersectclosest+13E- 5D                    - pop ebp
sauerbraten.game::intersectclosest+13F- 8B C2                 - mov eax,edx
sauerbraten.game::intersectclosest+141- 5B                    - pop ebx
sauerbraten.game::intersectclosest+142- 83 C4 2C              - add esp,2C { 44 }
sauerbraten.game::intersectclosest+145- C3                    - ret

You can see at the beginning it allocates space for local variables in the size of 0x2C and at the very end it frees that memory from the stack again.
 

Erarnitox

🐅
Meme Tier VIP
Trump Tier Donator
May 11, 2018
151
5,023
3
it's fastcall like petko said
i tried to use __fastcall with 4 arguments, however that didnt seem to work. What im i doing wrong? However having it as __cdecl and passing the first 2 arguments through register works just fine.
Also when you use __fastcall the Callee has to clean the arguments which is not the case here. Im pretty certain the Caller cleans up.
 
Attention! Before you post:

Read the How to Ask Questions Guide
99% of questions are answered in the Beginner's Guide, do it before asking a question.

No Hack Requests. Post in the correct section.  Search the forum first. Read the rules.

How to make a good post:

  • Fill out the form correctly
  • Tell us the game name & coding language
  • Post everything we need to know to help you
  • Ask specific questions, be descriptive
  • Post errors, line numbers & screenshots
  • Post code snippets using code tags
  • If it's a large project, zip it up and attach it

If you do not comply, your post may be deleted.  We want to help, please make a good post and we will do our best to help you.

Community Mods