How do Antivirus know which program called which API from the kernel space?

[ted114]

Actually, my main question is, how does runtime analysis really works on AVs perspectives. I cant really find an in-depth article online about how do they work, what are the common techniques that they use to approach a specific situation, etc.

So far, in my understanding of this runtime/behavioural analysis of AVs is that, for them to understand and analyze how a process behaves, they have to make API hooks (I hope you understand here, I cant explain much). And this gives them the ability to judge if a spcific program is malicious or not by analysing the APIs it called.

1) How do they know which process called an API that they hooked from kernel-space?

Say for example, an kernel mode driver of an AV placed a hook on an API, Program X->NtQuerySystemInformation->HookedZwQuerySystemInformation->ZwQuerySystemInformation. How would the kernel mode driver be able to identify who called the NtQuerySystemInformation (which in turn, leads to the execution of the hook-function)? Since we know, AVs cant simply rely on user-space API hooks for these scenarios, so they have to do it in kernel space.


2) And how do they know that a running process does some, can-be-considered, malicious things without even calling an API? like a decryption loop perhaps?

 

[Andy Ful]

...
1) How do they know which process called an API that they hooked from kernel-space?

Say for example, an kernel mode driver of an AV placed a hook on an API, Program X->NtQuerySystemInformation->HookedZwQuerySystemInformation->ZwQuerySystemInformation. How would the kernel mode driver be able to identify who called the NtQuerySystemInformation (which in turn, leads to the execution of the hook-function)? Since we know, AVs cant simply rely on user-space API hooks for these scenarios, so they have to do it in kernel space.
...

Because of PatchGuard, actually, most AVs do not rely on kernel-mode hooking, but rather on user-mode hooking.
https://breakdev.org/defeating-antivirus-real-time-protection-from-the-inside/

How to Make an antivirus engine | Tutorial • Adlice Software

Learn the theory with code snippets on how making your own robust and reliable Antivirus / Anti-malware engine. Step by step guide.

www.adlice.com

https://ired.team/offensive-security/bypassing-cylance-and-other-avs-edrs-by-unhooking-windows-apis
https://s7d2.scene7.com/is/content/...-library/white-papers/Universal_Unhooking.pdf
https://2016.zeronights.ru/wp-content/uploads/2016/12/You’re-Off-the-Hook.pdf

If I correctly remember, sometimes AVs can use hypervisors for kernel-mode hooking.
Furthermore, the mini-filter drivers can be used instead of SSDT hooking (to avoid PatchGuard).

Kaspersky Hooking Engine Analysis - UIC Academy

In this article we will talk about a few hooking techniques used by antivirus software. For the purpose of this analysis the antivirus chosen will be Kaspersky (http://www.kaspersky.com/it/trials PURE 3.0 Total Security), we will deal with various hooking techniques used both at user and kernel...

quequero.org

 

[ted114]

Because of PatchGuard, actually, most AVs do not rely on kernel-mode hooking, but rather on user-mode hooking.
https://breakdev.org/defeating-antivirus-real-time-protection-from-the-inside/

How to Make an antivirus engine | Tutorial • Adlice Software

Learn the theory with code snippets on how making your own robust and reliable Antivirus / Anti-malware engine. Step by step guide.

www.adlice.com

https://ired.team/offensive-security/bypassing-cylance-and-other-avs-edrs-by-unhooking-windows-apis
https://s7d2.scene7.com/is/content/...-library/white-papers/Universal_Unhooking.pdf
https://2016.zeronights.ru/wp-content/uploads/2016/12/You’re-Off-the-Hook.pdf

If I correctly remember, sometimes AVs can use hypervisors for kernel-mode hooking.
Furthermore, the mini-filter drivers can be used instead of SSDT hooking (to avoid PatchGuard).

Kaspersky Hooking Engine Analysis - UIC Academy

In this article we will talk about a few hooking techniques used by antivirus software. For the purpose of this analysis the antivirus chosen will be Kaspersky (http://www.kaspersky.com/it/trials PURE 3.0 Total Security), we will deal with various hooking techniques used both at user and kernel...

quequero.org

Thank you for that answer.

I actually cant understand something else. So does AVs does it this way?

Program X => Static Analysis => Dynamic Analysis (Emulation, something like AVASTs CyberCapture) => if malicious, stop, else execute on the real environment.

Is that how an AV operates? like, after the emulation, if not malicious, it executes it again on a separate procress that is not emulated anymore? (but ofcourse some functions is still hooked by the AVs to protect itself)

Or does the AV intercepts the real creation of creating the process (using the PsCreateProcessNotifyRoutine) and just run the program under an emulated environment.

So basically my main question is, does a program go thru an emulator (for dynamic analysis) and executed again to the real-unemulated environment? or does it run inside an emulator (emulated-environment) all the time (and not executed again) ?

thank you for your time.

 

[Andy Ful]

If you mean hooking, then AV stops for a moment the execution of real code of the executed program. The hooks are prepared for some popular APIs. Next, AV modules analyze that code, which is placed somewhere in the memory. After this, the code is blocked (if suspicious) or allowed. Here is an example:
"Security software will hook specific user space API functions that are commonly used by malware. For example, a code hook installed on winsock.connect can examine the IP and port of an outgoing network connection and decide whether the connection should be allowed or blocked. A combination of hooks installed on OpenProcess, VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread detect malicious process injection. "

https://s7d2.scene7.com/is/content/cylance/prod/cylance-web/en-us/resources/knowledge-center/resource-library/white-papers/Universal_Unhooking.pdf

AV can also run the program in the sandbox to analyze the effects of running it, but this is another feature.

The PsCreateProcessNotifyRoutine is a routine in the kernel, it can be installed by a driver. It can be also used by AV to monitor process creation and termination events in the Windows kernel.

 

[ted114]

If you mean hooking, then AV stops for a moment the execution of real code of the executed program. The hooks are prepared for some popular APIs. Next, AV modules analyze that code, which is placed somewhere in the memory. After this, the code is blocked (if suspicious) or allowed. Here is an example:
"Security software will hook specific user space API functions that are commonly used by malware. For example, a code hook installed on winsock.connect can examine the IP and port of an outgoing network connection and decide whether the connection should be allowed or blocked. A combination of hooks installed on OpenProcess, VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread detect malicious process injection. "

https://s7d2.scene7.com/is/content/cylance/prod/cylance-web/en-us/resources/knowledge-center/resource-library/white-papers/Universal_Unhooking.pdf

AV can also run the program in the sandbox to analyze the effects of running it, but this is another feature.

The PsCreateProcessNotifyRoutine is a routine in the kernel, it can be installed by a driver. It can be also used by AV to monitor process creation and termination events in the Windows kernel.

I actually meant CPU Emulation.

But about hooking, do you think kernel-modules that placed hooks on these APIs has the ability to know which specific user-mode program called that API?

 

[Andy Ful]

I actually meant CPU Emulation.

But about hooking, do you think kernel-modules that placed hooks on these APIs has the ability to know which specific user-mode program called that API?

You do not need the kernel modules that placed hooks on these APIs, to find out which specific user mode process called a specific API. You can simply use a combination of hooks from my previous post. The API function OpenProcess will give you the open handles of the processes. Next, you can take advantage of other hooked APIs to get additional information if any process is suspicious. Furthermore, If you stopped the execution flow of the process, then you can use your own modules to analyze the code (already executed and not-yet-executed) and changes made in the system.

 

[Zartarra]

Andy

Thank you for the expert explanation . It is very interesting stuff.

 

[Andy Ful]

Andy

Thank you for the expert explanation . It is very interesting stuff.

Thanks. But, I am not an expert.
I read the articles from my first post, which have helped me to recall what I had learned many years ago.

 

[ted114]

You do not need the kernel modules that placed hooks on these APIs, to find out which specific user mode process called a specific API. You can simply use a combination of hooks from my previous post. The API function OpenProcess will give you the open handles of the processes. Next, you can take advantage of other hooked APIs to get additional information if any process is suspicious. Furthermore, If you stopped the execution flow of the process, then you can use your own modules to analyze the code (already executed and not-yet-executed) and changes made in the system.

"You do not need kernel-modules" you mean, AVs(in general, if not all) just use user-mode (dll-injection) + hooking (to all processes) to know and analyze what APIs (and thus, analyze how the program interacts with the system) ??

I see, so AVs just use kernel modules to protect itself from being attacked by malwares or send some notification to some of it's user mode processes (like, when a process is created).

Thanks.

 

[Andy Ful]

"You do not need kernel-modules" you mean, AVs(in general, if not all) just use user-mode (dll-injection) + hooking (to all processes) to know and analyze what APIs (and thus, analyze how the program interacts with the system) ??

I see, so AVs just use kernel modules to protect itself from being attacked by malwares or send some notification to some of it's user mode processes (like, when a process is created).

Thanks.

The malc0ders usually use APIs in userland, so AVs hunt for those APIs to fight the malware.

 

[ted114]

The malc0ders usually use APIs in userland, so AVs hunt for those APIs to fight the malware.

what does that even mean? yes, malc0ders use APIin userland, but what does "AVs hunt for those APIs" mean? Anyway, thanks for the help.

 

[Andy Ful]

what does that even mean? yes, malc0ders use APIin userland, but what does "AVs hunt for those APIs" mean? Anyway, thanks for the help.

hunt for = actively search for
I am sorry. I cannot put it more clearly.
Be safe.

 

[oldschool]

what does that even mean? yes, malc0ders use APIin userland, but what does "AVs hunt for those APIs" mean? Anyway, thanks for the help.

I believe he means AV devs will look for the same APIs to use in their own way as a countermeasure.

 

[Andy Ful]

I believe he means AV devs will look for the same APIs to use in their own way as a countermeasure.

The way of using some APIs by malware usually differs from using them by most clean applications. So, AVs can hook on the APIs commonly used by malware and look for suspicious patterns. The legal applications may also use the hooked APIs, so the AV developer has to find the patterns which will be characteristic to malware files.

 

[436880927]

they have to make API hooks

They do not have to rely on API hooking all of the time - it depends on what the AV needs to do.

Kernel-mode callbacks can be used by AVs for intercepting process creation and termination (PsSetCreateProcessNotifyRoutine/Ex/Ex2, including support for Pico processes with the *Ex2 version on Windows 10), process thread creation (PsSetCreateThreadNotifyRoutine/Ex), process image loading (PsSetLoadImageNotifyRoutine/Ex), handle object creation and duplication for process/thread/desktop objects (ObRegisterCallbacks) and more.

1) How do they know which process called an API that they hooked from kernel-space?

IoGetCurrentProcess function (wdm.h) - Windows drivers

The IoGetCurrentProcess routine returns a pointer to the current process.

docs.microsoft.com

When the hook handler is called, you'll be executing under the context of the process which is responsible for the system call which led up to the official kernel API being called. However, you should note that if you're also intercepting API calls from trusted callers (e.g. code executing with a CPL of 0) then the responsible process will be ntoskrnl.exe, so beware of that.

 

[436880927]

Because of PatchGuard, actually, most AVs do not rely on kernel-mode hooking, but rather on user-mode hooking.

PatchGuard's Kernel Patch Protection isn't an issue for several vendors on Windows 64-bit systems because of hardware-assisted virtualization technology. As time goes on, more AV vendors will begin to adapt to the concept and start implementing support for it.

Avast currently has support for hardware-assisted virtualization. ESET uses it for code emulation features. Kaspersky uses it and I assume it's for their banking protection. Comodo use it. I'm sure there will be others.

Bear in mind, if you want to use a hyper-visor for MSR hooking on the IA32_LSTAR nowadays, you have to deal with Microsoft's Kernel Virtual Address Shadow (KVAS) feature which was implemented to mitigate the Meltdown CPU vulnerabilities. Originally, there was a simple work-around, but Microsoft changed things again starting with Windows 10 1809.

Furthermore, the mini-filter drivers can be used instead of SSDT hooking (to avoid PatchGuard).

In general, you can use kernel-mode callbacks instead of patching the Windows kernel but only if there's an appropriate kernel-mode callback available for you.

Mini-filter drivers are an official mechanism for interacting with the Filter Manager - named fltMgr.sys - and as such, intercept file system events in real-time. However, the other kernel-mode callbacks can be used within a mini-filter driver as well.

KeServiceDescriptorTable/Shadow hooking is completely different to using a mini-filter driver and mini-filter drivers in themselves are not an alternate to SSDT hooking.

You can simply use a combination of hooks from my previous post. The API function OpenProcess will give you the open handles of the processes.

That is a terrible idea.

Hooking in general is a terrible idea when you do not know what you are doing, but hooking Win32 APIs like OpenProcess (KERNEL32/KERNELBASE) is definitely a terrible idea when there's NtOpenProcess (NTDLL).

Instead of hooking NtOpenProcess - which is called by OpenProcess - you can use ObRegisterCallbacks (kernel-mode callback).

One kernel-mode callback and it relieves the need to hook NtOpenProcess, NtOpenThread, NtDuplicateObject and some others if used correctly.

 

[Andy Ful]

Opcode,
Welcome back and thanks for expanding/correcting my post.
It is true that AV vendors can use hypervisors for kernel-mode hooking (I mentioned this in my post), but for now it is used only for some vendors and only for some special tasks. Microsoft is not happy when software vendors use kernel-mode hooking, so there is always a risk that after Windows Update something will go wrong. That is why (probably) AV vendors are still very conservative about this possibility.

...
Hooking in general is a terrible idea when you do not know what you are doing, but hooking Win32 APIs like OpenProcess (KERNEL32/KERNELBASE) is definitely a terrible idea when there's NtOpenProcess (NTDLL).

Instead of hooking NtOpenProcess - which is called by OpenProcess - you can use ObRegisterCallbacks (kernel-mode callback).
...

I was referring to the example from Cylance article without much thinking if it is a good idea :
"Security software will hook specific user space API functions that are commonly used by malware. For example, a code hook installed on winsock.connect can examine the IP and port of an outgoing network connection and decide whether the connection should be allowed or blocked. A combination of hooks installed on OpenProcess, VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread detect malicious process injection."

https://s7d2.scene7.com/is/content/cylance/prod/cylance-web/en-us/resources/knowledge-center/resource-library/white-papers/Universal_Unhooking.pdf

 

[436880927]

It is true that AV vendors can use hypervisors for kernel-mode hooking (I mentioned this in my post), but for now it is used only for some vendors and only for some special tasks. Microsoft is not happy when software vendors use kernel-mode hooking, so there is always a risk that after Windows Update something will go wrong.

You can't say fairer than that. Agreed.

With my hyper-visor, the technique I was using to hook IA32_LSTAR is incompatible past Windows 10 1809 (for KVAS systems - Meltdown mitigation) so I had to change the design - thankfully, someone else had already shared a good alternative technique online which didn't involve page table hooking.

That's a prime example of running into incompatibility issues when using a hyper-visor for such things.

I was referring to the example from Cylance article without much thinking if it is a good idea :

Basically, I think what Cylance proposed is a terrible idea. The network monitoring part they added was even funnier... what a silly thing to even suggest nowadays when there's WFP available for use (with sample examples from Microsoft)!

 

[436880927]

PatchGuard's Kernel Patch Protection isn't an issue for several vendors on Windows 64-bit systems because of hardware-assisted virtualization technology. As time goes on, more AV vendors will begin to adapt to the concept and start implementing support for it.

There has been an undocumented technique since Windows Vista/7 which can be used to hook all system calls on Windows 64-bit without explicitly bypassing PatchGuard's Kernel Patch Protection feature. Windows 10 1903 is supported.

Recently it was disclosed publicly by an employee of Riot Games/ESEA.