x64hook is a hooking library for x64 code that aims to be able to patch everything!

The general problems with hooking general functions are: 1) The code to be overwritten with the jump to your code has relative jumps (forward) or calls. Those need to be fixed in the trampoline. As the trampoline is probably out of reach for 32bit jumps all the fixed jumps need to be x64 RIP relative jumps.

Conditional jumps are also handled - there are no jcc rel64 instructions though. That means that the condition is inverted as to jump, over the fixup code, further in the trampoline. The fixup code then is a unconditional 64bit jump.

2) The functions loops back into the code that has to be overwritten - that happens for functions with a loop (duh!) but also in case of tail-recursion optimization.

Currently hooking such functions is not supported and in fact this is not even detected. Detection: Search for the end of the function (which has its own problems), keep track of all jumps into the first few bytes.

3) Other threads might try to execute the code as it is overwritten - if that happens shit hits the fan and the normal working of the application will be disrupted. There are two methods of overcoming that problem: a) “The stop-the-world approach” stops all threads to hinder them from executing.

  That makes the hooking time intensive and easily detectable.

b) Writing all the problematic modifications atomically. That’s not doable for > 8bytes.

  How is that write reflected in the instruction cache?
  Instruction fetches follow a different memory model (no lock/unlock & µOp Caches)

c) The approach described in “Living on the edge: Rapid-toggline probes with cross modification on x86”

  Firstly writing a INT3 to the code which lets all other threads who execute that code jump into an newly
  installed execution handler with a spin loop until the hooking is done - i.e. execution can be safely
  resumed.
  Then the code is overwritten atomically with respect to cache lines.
  Lastly the exception handler is deactived.

As this is a x64 hooking library running on x64 the overwritten code is fairly long, making the occurence of the problems detailed above very likely.