In this section, we will explore different types of hooking mechanisms. In the following diagram, we can see various types of hooking techniques that rootkits use at different stages of the request processing flow:
Figure 6: The hooking mechanisms of rootkits
Rootkits can install hooks at different stages of this process flow:
- User-mode hooking/API hooking: These are the user-mode API hooking mechanisms that are used for hiding malware processes, files, registry keys, and more. We covered this in Chapter 4, Inspecting Process Injection and API Hooking.
- SYSENTER hooking: This is the first option that's available for the kernel-mode rootkits to hook. In this case, they change the address that sysenter will transfer the execution to, and intercept all requests from the user mode to the kernel mode.
- SSDT hooking: This technique works more closely with the functions that the rootkit wants to hook. This type of hooking modifies the SSDT to redirect requests to a...
