Solved Help D3D Hooking in Rust

Hexui Undetected CSGO Cheats Sinkicheat PUBG Cheat

kookaburra

Full Member
May 2, 2020
11
114
0
Game Name
N/A
Anticheat
N/A
How long you been coding/hacking?
Couple months
Coding Language
Rust
So I've been trying to get a small d3d hook running with the Rust language, as I think it has a lot of potential for game hacking but I havent seen many people use it for that yet, and I'm having a bit of trouble getting the D3D9 device VTable, I was basically doing a translation of the code found in projects like Kiero and Indicium-Supra to get the device pointer, but every time I inject my DLL into something, i find that it returns some error that I'm unsure how to fix, the error value itself is 0x8876086C, but I cant find much info on it.

As far as I can tell, all my code looks correct, but I'd like any advice from people more experienced with D3D hooking stuff to see if theres something I'm missing here.

lib.rs:
use std::thread;
use std::ffi::CString;
use std::ptr;

use winapi::shared::{
    windef::*,
    minwindef::*,
    d3d9::*,
    d3d9types::*,
};
use winapi::um::{
    consoleapi::AllocConsole,
    winnt::DLL_PROCESS_ATTACH,
    winuser::*,
    processthreadsapi::GetCurrentProcessId,
    libloaderapi::{ DisableThreadLibraryCalls }
};

mod d3d9_hook;

fn init() {
    unsafe {
        d3d9_hook::get_d3d9_vtable();
    }
}

#[no_mangle]
pub extern "stdcall" fn DllMain(h_inst: HINSTANCE, fdw_reason: DWORD, _: LPVOID) -> BOOL {
    if fdw_reason == DLL_PROCESS_ATTACH {
        unsafe {
            AllocConsole();
            DisableThreadLibraryCalls(h_inst);
        };
        println!("INJECTED IN PROCESS");
        thread::spawn(|| init() );
    }
    return TRUE
}
d3d9_hook.rs:
use std::ffi::CString;
use std::mem;
use std::ptr;

use winapi::shared::{
    windef::*,
    minwindef::*,
    d3d9::*,
    d3d9types::*,
};
use winapi::um::{
    winuser::*,
    winnt::HRESULT,
    libloaderapi::{ GetModuleHandleA, GetProcAddress }
};

pub unsafe fn get_d3d9_vtable() {
    let d3d9_dllname = CString::new("d3d9.dll").unwrap();

    let (window_class, temp_window) = create_temp_window();

    println!("looking for d3d9.dll");
    let d3d9_handle = GetModuleHandleA(d3d9_dllname.as_ptr());
    if d3d9_handle as *mut _ == ptr::null_mut() {
        DestroyWindow(temp_window);
        UnregisterClassA(window_class.lpszClassName, window_class.hInstance);
        return
    }

    println!("creating d3d9 interface");

    type Direct3DCreate9Ex_t = unsafe extern "system" fn(UINT, *mut *mut IDirect3D9Ex) -> HRESULT;

    let d3d9_create_ex_name = CString::new("Direct3DCreate9Ex").unwrap();
    let create_ex_address = GetProcAddress(d3d9_handle, d3d9_create_ex_name.as_ptr());
    let Direct3DCreate9Ex_fn: Direct3DCreate9Ex_t = mem::transmute(create_ex_address);

    let mut d3d9_raw = ptr::null_mut();
    match Direct3DCreate9Ex_fn(D3D_SDK_VERSION, &mut d3d9_raw) {
        0 => {},
        _ => {return},
    };

    let d3d9_ex = match d3d9_raw.as_mut() {
        Some(interface) => interface,
        None => {
            println!("error creating interface");
            return
        }
    };

    println!("setting up display mode");
    let mut display_mode: D3DDISPLAYMODE = mem::zeroed();
    if d3d9_ex.GetAdapterDisplayMode(D3DADAPTER_DEFAULT, &mut display_mode) < 0 {
        DestroyWindow(temp_window);
        UnregisterClassA(window_class.lpszClassName, window_class.hInstance);
        return
    }

    let mut present_params: D3DPRESENT_PARAMETERS = mem::zeroed();
    present_params.Windowed = TRUE;
    present_params.SwapEffect = D3DSWAPEFFECT_DISCARD;
    present_params.BackBufferFormat = display_mode.Format;

    println!("creating d3d9 device");
    let mut d3d9_device_ex = ptr::null_mut();
    
    let result_device_err = d3d9_ex.CreateDeviceEx(
        D3DADAPTER_DEFAULT,
        D3DDEVTYPE_HAL,
        temp_window,
        D3DCREATE_SOFTWARE_VERTEXPROCESSING | D3DCREATE_DISABLE_DRIVER_MANAGEMENT,
        &mut present_params,
        ptr::null_mut(),
        &mut d3d9_device_ex
    );

    if result_device_err < 0 {
        println!("error creating device: {:#X}", result_device_err);
        DestroyWindow(temp_window);
        UnregisterClassA(window_class.lpszClassName, window_class.hInstance);
        return
    }

    match d3d9_device_ex.as_mut() {
        None => {
            println!("couldnt get device :(")
        },
        Some(device_ref) => {
            println!("Got Device!!!!!! {:#X}", d3d9_device_ex as usize);
        }
    }

}

unsafe fn create_temp_window() -> (WNDCLASSEXA, HWND) {
    println!("create_temp_window called");
    let mut window_class: WNDCLASSEXA = mem::zeroed();
    window_class.cbSize = std::mem::size_of::<WNDCLASSEXA>() as u32;
    window_class.style = CS_HREDRAW | CS_VREDRAW;
    window_class.hInstance = GetModuleHandleA(ptr::null());
    window_class.lpfnWndProc = Some(DefWindowProcA);
    window_class.lpszClassName = CString::new("d3d_hook").expect("Couldnt create CString for lpszClassName in create_temp_window").as_ptr();
    window_class.cbClsExtra = 0;
    window_class.cbWndExtra = 0;
    window_class.hIcon = ptr::null_mut();
    window_class.hIconSm = ptr::null_mut();
    window_class.hCursor = ptr::null_mut();
    window_class.hbrBackground = ptr::null_mut();
    window_class.lpszMenuName = ptr::null_mut();

    RegisterClassExA(&window_class);
    let window: HWND = CreateWindowExA(
        window_class.style,
        window_class.lpszClassName,
        CString::new("D3D Temp Window").expect("Couldn't create CString for window name").as_ptr(),
        WS_OVERLAPPEDWINDOW,
        0,
        0,
        100,
        100,
        ptr::null_mut(),
        ptr::null_mut(),
        window_class.hInstance,
        ptr::null_mut(),
    );

    return (window_class, window);
}
Cargo.toml:
[package]
name = "d3d9hooktest"
version = "0.1.0"
authors = [""]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lib]
crate-type = ["cdylib"]

[dependencies]
winapi = { version = "0.3.9", features = ["std", "ntdef", "windef", "minwindef", "winnt", "winuser", "consoleapi", "libloaderapi", "processthreadsapi", "d3d9"] }
 

kookaburra

Full Member
May 2, 2020
11
114
0
Just for some more info: the process I was injecting the DLL into was a 32-Bit build of the D3D9 Test Environment someone here made, to build this properly for that, you just need to run `cargo build --target i686-pc-windows-msvc` with all the proper toolchain stuff installed
 

mambda

headass
Escobar Tier VIP
Trump Tier Donator
Jun 25, 2014
2,298
37,938
269
Unfortunately I (and i think most others here) have never used Rust.

Are you able to debug it easily? Like via VS, or Windbg or something? Would help to pinpoint the issue
 

kookaburra

Full Member
May 2, 2020
11
114
0
I might be able to debug it, I'll probably try to do that soon, but it seems that the issue here is the usage of CreateDevice(), searching the error code gives very little information, but seems to point to the call somehow being invalid, I'm unsure
 

Pseudopourri42

Newbie
Full Member
May 29, 2016
22
658
3
Apparently, according to the internet the error you're getting seems to simply be just a driver / DX redistributable problem, I haven't looked in depth but here's solution on microsoft.com that looks interesting, and here are other results.

Personally what I do when I have problems with dx9 hooking, is that I use an asi loader and x32dbg to step trough my dll, also with x32dbg you can use this method to see if your code is getting the correct addresses, and I also use CE dissect data (basically reclass) to find where functions are in the Vtable in relation from your dx9 device (not relevant to the problem, but still useful to know), you can also use any injector to inject your dll, and CE to inject & debug too (which is what I did before x32dbg).

Also make you don't sure have anything that messes with d3d9.dll (overlays like fraps), apart from that I don't know rust and I'm not an expert but I haven't see any obvious errors, don't know if it will help but you can also take a look at the dummy device method code, or switch method and follow this tutorial to find the device manually.
 
Last edited:

kookaburra

Full Member
May 2, 2020
11
114
0
Update for anyone who might have been interested: I still have yet to get it working :(
It's been a while since I used the code here and I'm thinking I may attempt to write a D3D9 wrapper DLL instead just to see if that works, it might be a cleaner solution as it would let me modify the VTable before passing it to the program, unfortunately that entails a loooot of boilerplate code, so I probably won't write all of it, just whats needed for the D3D9 test environment. I don't know if I'll ever get this done however, I wish it were easier in Rust but it just seems like a bit too much work for me at the moment.

If anyone else here attempts a D3D hook in Rust I would love to see that though, I think this language could be incredibly promising for nice looking code in mods of games
 

Rake

Cesspool Admin
Administrator
Jan 21, 2014
12,204
78,998
2,400
0x8876086C = failed to create D3D device

So your error is in your call to CreateDevice, also why are you using CreateDeviceEx and not CreateDevice ? idk the difference but noone I know is using the Ex version

instead of creating your own window do:

C++:
BOOL CALLBACK EnumWindowsCallback(HWND handle, LPARAM lParam)
{
    DWORD wndProcId;
    GetWindowThreadProcessId(handle, &wndProcId);

    if (GetCurrentProcessId() != wndProcId)
        return TRUE; // skip to next window

    window = handle;
    return FALSE; // window found abort search
}

HWND GetProcessWindow()
{
    window = NULL;
    EnumWindows(EnumWindowsCallback, NULL);
    return window;
}

d3dpp.hDeviceWindow = GetProcessWindow();
and pass that as your third argument

try just D3DCREATE_SOFTWARE_VERTEXPROCESSING

Try a different d3d9 test environment, or try on CSGO -insecure

also, compare your code against this and make sure it's comparable:
Source Code - Get Direct3d9 and Direct3d11 Devices - Dummy Device Method
 

kookaburra

Full Member
May 2, 2020
11
114
0
I used to use EnumWindows to find it but that doesnt work if the program has either multiple windows or a CMD window (Which I am using to print debug info, and would like to keep if possible.) I found that there are other programs that create their own dummy window first, create a device with it, and then destroy the window purely to get access to the VTable, and that method seems to work consistently well in C++ projects I've seen
 

Rake

Cesspool Admin
Administrator
Jan 21, 2014
12,204
78,998
2,400
The EnumWindowsCallback works consistently well also, you can filter out your own window by title or classname or filter in the correct game window that way also, if you want to avoid the issue you mentioned. I'm telling you, your CreateDevice fails and I am providing you things you can try to get it to stop failing. It fails when there are problems with the arguments, so I'm providing you ways to change those args that might fix your problem.
 

Not2EXceL

The rust BC is trash...wait no im a shit dev
Dank Tier Donator
Nobleman
Jan 20, 2013
131
1,743
3
Took a few hours these past two nights to get a d3d9 hook working in rust. Got some cleanup todo, and I'll push it to a repo and post it here. Works, but the code could probably be improved.

 
Attention! Before you post:

Read the How to Ask Questions Guide
99% of questions are answered in the Beginner's Guide, do it before asking a question.

No Hack Requests. Post in the correct section.  Search the forum first. Read the rules.

How to make a good post:

  • Fill out the form correctly
  • Tell us the game name & coding language
  • Post everything we need to know to help you
  • Ask specific questions, be descriptive
  • Post errors, line numbers & screenshots
  • Post code snippets using code tags
  • If it's a large project, zip it up and attach it

If you do not comply, your post may be deleted.  We want to help, please make a good post and we will do our best to help you.

Similar threads

Community Mods