Hooking Non-Virtual Functions

While writing CS:S DM, I had to hook a non-virtual function in the HL2 engine. Unfortunately, SourceMM isn’t so magical that it can do this. I decided to try a first attempt on my own; cautiously, I’d never even done manual vtable hooking before.

The concept for virtual functions is easy and straight forward. Find the offset in the virtual function table and overwrite it with a “call gate”. The call gate calls a different function which decides whether to call the original. This is very easy since only one entry point needs to be modified.

However, non-virtual functions are a pain. They’re static blocks of code, and the only viable way to scan for their references is by walking the code section and calculating possible eip values. So, you have to edit the code.

The solution I came up with was quite hardcoded to both the calling convention and the actual function in question. But basically, it goes like this:

  1. Identify the address of the function you want to hook.
  2. Assembly a “call gate” in memory. This call gate should look something like:
    push ebp
    mov ebp, esp
    mov eax, [ebp+8]
    mov ecx, <address>
    call [ecx]
    pop ebp
    retn

    Since you assemble this code in memory, you can move the address of your call handler into the code at runtime, by calculating the offset of “mov, ecx [ASDF]“.
  3. Assembly a “stub gate” in memory, and save it. This would look something like:
    push esp ;push the stack pointer so we can get parameters
    call callgate
    ret
  4. Calculate the length of the stubgate. Save this many bytes of the original function’s code.
  5. Copy the stubgate over the original code. Don’t forget a call to page-aligned mprotect() or VirtualQuery()!
  6. Create another function called “force_original”. This function should unpatch the original function by copying the saved bytes, call it, then restore the stub gate. Obviously, this is not re-entrant at all and a serious flaw.
  7. Return, return, return.
  8. What has happened? The original function was edited to call a gateway function. This gateway function takes the original stack pointer and gives it to your handler. Your handler decides whether to force the original call (which requires two repatchings), and then returns to the caller.
  9. Using this you can make a hacky, hardcoded, and awkward callgate for a non-virtual, static function. I have an example for hooking a four parameter, void function on linux here: dropgate.asm

    I’ll make a “Part 2″ post later: how to do this with a much nicer reentrant jmp. Using this method, combined with a system of specifying stack width and calling convention, SourceMM might have non-virtual function hooking in the future. Mmmm!

2 Responses to “Hooking Non-Virtual Functions”

  1. PM says:

    Oh my god! You don’t want to integrate something into SourceMM that would not work on PaX!?!?!

  2. BAILOPAN says:

    PaX will like whatever the hell I give it! PaX sucks!

Leave a Reply

You must be logged in to post a comment.