Hooking: Setup a hook is an action performed by a rootkit. We can summarize that action by placing a filter on top of a system API. For example,
NtOpenProcess is the API needed if we want to kill a process. If a rootkit hooks that API, it will be able to tell if the process termination is allowed or not, and protect its process.
Just to elaborate... There are many more ways in which a process can be targeted for termination, aside from NtOpenProcess.
NtOpenProcess (NTDLL) is called when OpenProcess (KERNEL32) is called. It is used to open a handle to a process and then this handle can be used to suspend the process, inject code into the process, or terminate the process - code injection could also be used for termination if NtTerminateProcess is hooked.
Examples.
1. Process threads. These can be targeted as well. You can use OpenThread (KERNEL32) which will call NtOpenThread (NTDLL) to open a handle to the threads within the process and then call TerminateThread (KERNEL32) which will call NtTerminateThread (NTDLL). If you can get a handle to one of the threads, you can also use it to attempt APC injection (however due to not having a handle to the process, you could just use shell-code for the APC injection instead of performing virtual memory operations to place code in its address space beforehand).
2. Handle hijacking. Some system processes will automatically have an open handle to the running processes on the system. On earlier versions of Windows, csrss.exe will have an open handle. On newer versions of Windows, lsass.exe and svchost.exe will have an open handle. You can inject code into such processes to hijack the handles they already have for usage.
3. User-Mode API hooks are usually set for specific processes, not all of them. A common example would be Task Manager (taskmgr.exe). You could try injecting into another process which is likely not to be targeted, and then attempt to terminate the process from within the unsuspected process.
I am certain creative security researchers can come up with more interesting examples.
There is one very easy solution though.
You can bypass the user-mode hooks by using a system call. A system call is when you call the same instructions NTDLL would have had executed to make code execution pass to the kernel to perform an operation, such as opening a handle to a process or terminating a process via the handle. In some scenarios, this can be blocked (e.g. WOW64 interception for a 32-bit process running on a 64-bit OS environment) but it is unlikely.
------------
Kernel-Mode rootkits are no longer prevalent due to the Extended Validation Code Signing Certificate requirement on Windows 10 by PatchGuard (Driver Signature Enforcement) and Kernel Patch Protection (KPP - also part of PatchGuard). Most people use 64-bit systems where this security feature is present, and malware authors tend to want compatibility for both 32-bit and 64-bit systems... PatchGuard has been around since Windows Vista and improves a lot for every new Windows version. Even prior to Windows 10, a normal code signing certificate was still required. Bypasses do exist via exploitation of the VirtualBox driver, but I do not recall ever seeing a real malware attack take place doing something like this.