Do you really understand AV test results?

[mlnevese]

I could make a long reply to this, but you have average Joes out there that do not even know that Windows Defender ships with Windows.

The amount of education effort required to get a decent handle on Windows, general security, and security softs is not that onerous.

You cannot rely upon a security soft alone. It takes more than that nowadays. If a person does not understand this fact or is ignorant of it, then they are at a great disadvantage in securing their systems. What you don't know can and will hurt you.

Don't take me wrong. I completely agree that people should take some time to learn at least the basics on how to secure their computers. I still think it's unrealistic to expect all of them to do that though.

Taking my car example a little further, if someone told me I had to take time away from playing with my kids to learn more about car mechanics or electronics, it would require a really good argument for me to do that. I already have less time than I would like to dedicate to my kids as things stand....

 

[Prorootect]

Detonating a bad rabbit: Windows Defender Antivirus and layered machine learning defenses

Figure 4. Sample detonation events used by the machine learning model

Many rundll32.exe ... and if it would be possible, as a security measure, put this rundll32.exe file in C:\Windows\System32 on 'read-only' mode, deny write access? Check in the file properties the Read-only box... Would that be a good measure against infections, if I'm not mistaken?
- so that viruses will no longer be able to write? Ransomware could no longer be able to make system changes...or could be able to still make them?
How would Windows react on this attribute change?

 

[Deleted member 65228]

Many rundll32.exe ... and if it would be possible, as a security measure, put this rundll32.exe file in C:\Windows\System32 on 'read-only' mode, deny write access? Check in the file properties the Read-only box... Would that be a good measure against infections, if I'm not mistaken?
- so that viruses will no longer be able to write? Ransomware could no longer be able to make system changes...or could be able to still make them?
How would Windows react on this attribute change?

Rundll32.exe doesn't get replaced 99% of the time, this can cause other components in the environment to stop functioning. You should only expect something like this if the malicious software has a purpose of destruction, which isn't very common anymore because even for home user attacks, nowadays it is more focused around generating income... This is simply the reason as to why bootkit/rootkit infections stopped being prevalent in the wild (e.g. for MBR hijacking - you can't put down a note about paying Bitcoin unless the author understands 16-bit Assembly and can write their own boot-loader to make use of - and less change of getting paid compared to standard ransomware).

The infpub.dat file deleted with the rundll32.exe event logged on that call stack (well it isn't really a call stack but more a list of API calls which has been properly formatted which was taken from interception of various APIs such as for process creation) is not really a *.dat file. It's a Portable Executable (Dynamic Link Library) but was named to have the *.dat extension - this is an anti-analysis technique because it may fool users into believing it isn't really a Portable Executable, it won't however fool any decent Anti-Virus product which will identify the MZ bytes (which all Portable Executable's will have - the first two bytes represents MZ).

For the record, that ransomware attack required elevation and it was obviously malicious based on the analysis. I've analysed it myself manually - it should have been flagged the minute it dropped to the Windows folder because the entire thing was suspicious. Dropping a file to the Windows folder shouldn't be happening in the first place, especially if it is being masked as *.dat when it is really a PE... From a program unknown to the cloud. I don't know how Anti-Virus companies operate but I think they should have auto-flagged that and then the entire attack would have been prevented from even being attempted. Another indicator would be use of scheduled tasks directly after a suspicious action like that, or other factors such as when a program attempts to call NtAdjustTokenPrivileges (NTDLL) recursively to try and enable as many privileges as possible.

Rundll32.exe in this occasion was used to execute code belonging to a DLL file with a fake extension so the initial launcher of the malware doesn't have to load the module itself (either through a static link which wouldn't be applicable in these circumstances or a dynamic link (which would have been applicable under these circumstances), which may avoid suspicion because Anti-Virus solutions will receive a callback when a process loads a module. Maybe the author thought that by using rundll32.exe, the module would never be scanned... Possibility, and maybe some products were simply evaded due to this (pretty stupid if they were though).

Things like this leads me to agree with Lockdown (simply cannot help it, what he says about Chromebooks... Cannot disagree at all and I didn't even think of that until I saw him say it), average users should just use a Chromebook. Even work employees should just use a Chromebook most of the time unless they need software which runs on Windows and there's no good alternatives for Chrome OS/web-based at-least when working from home on the business work. Behavioural analysis is good but when simple things which are obviously suspicious don't get internally flagged it just makes you wonder where dynamic analysis is even going.

Anyway, what Microsoft has done is great but they are late to the party when it comes to logging operations system-wide/for specific applications. Kaspersky has been using the hyper-visor to emulate KiSystemCall64 (or at least intercept it and then return back to the original) for years now on 64-bit systems (no KeBugCheckEx invocation from PatchGuard because they use the hyper-visor). Still great progress from Microsoft though.

 

[Deleted member 65228]

Check in the file properties the Read-only box... Would that be a good measure against infections, if I'm not mistaken?
- so that viruses will no longer be able to write? Ransomware could no longer be able to make system changes...or could be able to still make them?
How would Windows react on this attribute change?

I doubt it'd make much of a difference.

For starters, I recon that the read-only setting is enforced via the Win32 API and not the Native API. If this is really the case then using NtDeleteFile/NtWriteFile would bypass it; I can always check on this for you if you'd like.

Secondly, viruses aren't prevalent anymore so I wouldn't worry about them in general. Virus infections have the goal of mass destruction, and as I said before, it isn't really about that most of the time nowadays. More stealing information (e.g. banking malware/keylogging -> login credentials and personal information -> can be sold by the attacker to others because the attacker themselves probably don't want them especially because using login credentials can expose them further and allow them to be caught out easier), encrypting files (or pretending to) and demanding a ransom or adware.

Thirdly, setting read-only for rundll32.exe doesn't stop all ransomware, and never will. Ransomware won't be replacing rundll32.exe, it'll only be using it, and it can still execute rundll32.exe with the read-only flags because the Windows Loader would be reading the data, not overwriting it. Many ransomware doesn't use rundll32.exe as well, so there's that too.

 

[Deleted member 65228]

Don't take me wrong. I completely agree that people should take some time to learn at least the basics on how to secure their computers. I still think it's unrealistic to expect all of them to do that though.

I agree with this as well. I think Lockdown is right and you are also right... People should educate themselves if they really want to be secure, but a lot of people don't. I know people who work 9 - 5 and when they come home they just want to rest and chill out (plus many other responsibilities like cooking dinner for their children, cleaning/tidying up, moving the bins, handling work calls after-hours/e-mails, etc). They won't be interested in security and will hardly know anything about computers aside from opening the web browser (they will neither even understand that there are different browsers and the differences) with maybe a typical AV package installed like Norton, and attempting to educate them would bore them out and it'd go in through one ear and out the other...

Time flies... Even if you aren't having fun!

 

[Andy Ful]

Anyway, what Microsoft has done is great but they are late to the party when it comes to logging operations system-wide/for specific applications.

I thought (optimistically) that Defender would use (in the cloud) the telemetry about suspicious system events from the infected computer. But it seems, that it adopted the pretty known path of 'detonating' the malware in the cloud to gather system events from the sandbox. That has known cons, because many malware samples are created to recognize sandboxes. Also running the malware twice make the analysis longer.

 

[Deleted member 65228]

That has known cons, because many malware samples are created to recognize sandboxes.

Microsoft will take precautions to help prevent software from identifying a virtual environment.

1. Patch the Process Environment Block if they have debugging capabilities for software being monitored (e.g. they use the WinDbg engine in Visual Studio and it would be cool if they used it for disassembly in-memory of monitored programs and checks certain chunks of memory to identify interesting things from the disassembly (e.g. ASM instructions -> opcodes)).

2. Memory control. If they are patching memory within the address space of the monitored process to log operations via interception of various APIs which are user-mode based and don't pass down to a system service routine (KeServiceDescriptorTable) or too tricky to log it from kernel-mode then they can control memory read and spoof bytes so the sample never identifies the memory has been altered.

3. Patch the NtClose and NtCreateThreadEx exploitation (abuse) for identifying monitoring being occurred. There are many others as well. However all these common techniques, esp. user-mode ones, may be completely useless for the malware depending on the route they take. (see below notes)

Anyway Microsoft have their own virtual environment on the cloud I would hope so they can use the hyper-visor/enable debug mode (but still control device driver image loads -> either via SeRegisterImageVerificationCallback/MmLoadSystemImage interception) and intercept from kernel-mode. I'll use 64-bit environment as example:
- Intercept KiSystemCall64. Intercepting routines like KiSystemCall64 allows you to intercept all system service routine invocation system-wide which relies on the System Service Descriptor Table where the address is not already known and used directly. For example, when a system call is performed from user-mode, it lands at KiSystemCall64 which is pointed to by IA32_LSTAR, and then after this the address is taken from the SSDT and the real routine is called... Or when a Zw* routine is called in kernel-mode, it lands at KiSystemCall64 as well (for x64).

If I had my own cloud analysis environment:
- Enable debug mode at boot of the environment.
- Hyper-Visor implementation.
- Patch routines present within ntoskrnl.exe -> log activities.
- Use the logs to determine specific behavior having being occurred/scoring system.

That is just an example, but Microsoft having a sandbox mechanism in the cloud is a huge advantage, not disadvantage I think.

 

[Prorootect]

I doubt it'd make much of a difference.

For starters, I recon that the read-only setting is enforced via the Win32 API and not the Native API. If this is really the case then using NtDeleteFile/NtWriteFile would bypass it; I can always check on this for you if you'd like.

Secondly, viruses aren't prevalent anymore so I wouldn't worry about them in general. Virus infections have the goal of mass destruction, and as I said before, it isn't really about that most of the time nowadays. More stealing information (e.g. banking malware/keylogging -> login credentials and personal information -> can be sold by the attacker to others because the attacker themselves probably don't want them especially because using login credentials can expose them further and allow them to be caught out easier), encrypting files (or pretending to) and demanding a ransom or adware.

Thirdly, setting read-only for rundll32.exe doesn't stop all ransomware, and never will. Ransomware won't be replacing rundll32.exe, it'll only be using it, and it can still execute rundll32.exe with the read-only flags because the Windows Loader would be reading the data, not overwriting it. Many ransomware doesn't use rundll32.exe as well, so there's that too.

Thank you Opcode for your nice explanations, yes could you then check this with an ransomware using rundll32.exe read-only, result would be interesting, please

 

[Andy Ful]

...
That is just an example, but Microsoft having a sandbox mechanism in the cloud is a huge advantage, not disadvantage I think.

Of course, it is an advantage - many AVs use sandboxes to analyze the malware samples. But where is the lock, there is also the key. So, some malware, soon or later, will find a way to recognize the sandbox. This is a mouse and cat game.
Also, the malware can apply known tactics to delay their malicious actions for example by one day. If it will be 'detonated' in the sandbox, then analysis will last a day, too (not good).
The layered Defender protection is nothing special, as compared to popular AVs which use AI. It is more practical as compared to Avast (for example), because of using post-infection on the fly signatures ('guinea pig'), and sadly that is all.
But the progress is evident, so let's wait patiently.

 

[Deleted member 65228]

So, some malware, soon or later, will find a way to recognize the sandbox.

If Microsoft are monitoring entirely from kernel-mode then the user-mode malicious software cannot access the kernel-mode memory to detect the kernel patches without a zero-day exploit. At-least then being exposed by UM memory patches won't be a possibility. So detecting the interception potential with that one is out the window. But then there is still identifying the virtual environment itself as you mention.

They can hide identifiers for the virtual environment, make sure that all the common (at-least) techniques are prevented - they can even install general software, dump a load of random documents scattered everywhere, setup fake auto-sign in credentials for browser configurations, etc. At-least then it will be a lot more rare for a malware sample to successfully differentiate while under the environment. But it'll never be full-proof.

I agree with you

 

[Evjl's Rain]

You can greatly increase Comodo script protection by going into advanced settings/miscellaneous/heuristic command line, and turn on full protection for the whole vulnerable process list. You can also add to the list, if you wish. It's pretty flexible.

I forgot that too. I also included some of @AtlBo's tweaks in my setup but unfortunately, I almost never them working

 

[AtlBo]

I forgot that too. I also included some of @AtlBo's tweaks in my setup but unfortunately, I almost never them working

Hello @Evjl's Rain. Almost none of the settings I have been using are properly tested. Apologies if they don't function as I indicated. Are you mostly referring to adding to the heuristic command-line process monitoring? I wonder about whether there is any point in going so far as to add most of them. Those were originally created for Bouncer.

If there are others issues, please let me know, and I will take a look to verify what you have seen...

 

[Evjl's Rain]

Hello @Evjl's Rain. Almost none of the settings I have been using are properly tested. Apologies if they don't function as I indicated. Are you mostly referring to adding to the heuristic command-line process monitoring? I wonder about whether there is any point in going so far as to add most of them. Those were originally created for Bouncer.

If there are others issues, please let me know, and I will take a look to verify what you have seen...

It was heuristic command-line process monitoring. I think they are working correctly for everyone but I have never seen them in action because I blocked them from execution using autosandbox

 

[AtlBo]

It was heuristic command-line process monitoring. I think they are working correctly for everyone but I have never seen them working because I blocked them from execution using autosandbox

OK thx. I use a/s as with your tests, so I don't see them either except for a browser extension (only in Chrome this happens).

Not sure what exactly is in this protection in the first place. Does it rely on a request for command line being initiated from a certain place or places? How does trust affect this in the case of memory injection of a trusted process? I have no idea about this, and Comodo already monitors all the actual command line host executors as unrecognized since they aren't digitally signed (even though from MS). o/c user who needs a script run will automatically use unblock to create bypasses for these if there is a script they must run which makes all the standard Comodo protections useless in the present and future.

Just an idea but it might be an interesting test to see if, with all protections off except c-l heuristics, c-l reliant malware would be blocked at any point in its execution. I guess it could still cause a good deal of damage before requesting c-l, but this sounds like a good one for a vid. I believe @cruelsister is on sabbatical, but mb she would look into that when she returns.

 

[boredog]

OK thx. I use a/s as with your tests, so I don't see them either except for a browser extension (only in Chrome this happens).

Not sure what exactly is in this protection in the first place. Does it rely on a request for command line being initiated from a certain place or places? How does trust affect this in the case of memory injection of a trusted process? I have no idea about this, and Comodo already monitors all the actual command line host executors as unrecognized since they aren't digitally signed (even though from MS). o/c user who needs a script run will automatically use unblock to create bypasses for these if there is a script they must run which makes all the standard Comodo protections useless in the present and future.

Just an idea but it might be an interesting test to see if, with all protections off except c-l heuristics, c-l reliant malware would be blocked at any point in its execution. I guess it could still cause a good deal of damage before requesting c-l, but this sounds like a good one for a vid. I believe @cruelsister is on sabbatical, but mb she would look into that when she returns.

Is cruel sister marring the prince?

 

[Andy Ful]

Some clarification about AV-Comparatives real-world tests:
AV-Comparatives: Real-World Protection Test - November 2017
AV-Comparatives: Real-World Protection Tests for June 2017 & Feb – June 2017
Real-World Protection Test March 2017
.
"PUP is not included in the test; PUP detection is turned on (just to avoid false claims, as there is no PUA included anyway). The test is not done on old samples, but on fresh/current samples (some products may in fact reach high rates because the malware is so fresh, as they block files which they have not seen before)."
"malware is also executed." (not only URLs)
"Chrome" (is used in the test for browsing)

 

[509322]

The layered Defender protection is nothing special, as compared to popular AVs which use AI.

Better not say that, you will be accused of being anti-Microsoft and spreading "new" Windows Defender FUD. To some, Microsoft and the "new" Windows Defender walk on water.

If there is anything redeeming going for it, it that it costs nothing. But most people expect other publishers' (such as AVAST's) superior protection to cost nothing anyway - because that is the prevailing mentality - the vast attitude is that security ALL softs should cost some ridiculously low amount like $10, $5 or $0. Somehow security soft publishers are supposed to pay their employees and pay all their bills with play money.

 

[509322]

Remember what AV lab test results are - they are marketing materials - not some report grounded in scientific principles like double-blind studies.

The whole point of this thread was "Do you\people understand AV testing ?"

The answer is clearly - no they do not.

• What is being tested ?
• How is it being tested ?
• What can influence the testing?
• What does the report realistically and practically mean?
• What do the graphs mean?
• What do the statistics mean?
• What can influence the statistics?
• How can the test statistics be manipulated?
• When are differences in the test result statistics not relevant?
• What is the practical difference between tested AVs for grandma user ?
• What is the practical difference between tested AVs for average office user
• ?What is the practical difference between tested AVs for the "user that wants to use stuff" ?
• What is the practical difference between tested AVs for the high-risk user ?
• What is the practical difference between an AV that scored 100 %, 99.1 %, 97.3 % and 95.9 % ?
• What is the practical difference between 5 stars or 4 stars or 3 stars, etc ?
• What does it mean when a security solution's performance varies little over time ?
• What does it mean when a security solution's performance varies wildly over time ?
• What does it mean when a security solution has isolated tests where its performance is inconsistent with its prior test lab history ?
• Can the test results for the specific sample set used to produce the test published results be extrapolated to all current and future malware for the security solution ?
• Etc, etc, etc

99.999 % of people would have a difficult if not impossible time answering the above questions.

If a person cannot answer such basic questions, then AV lab test results amount to nothing more than baseless and empty AV rankings for the average, typical test report reader - pretty pictures and graphs were the reader kinda sorta struggles to make out "Which is best AV" gravitating to the ones with 100 % green bars or all 5 stars for each group.

In other words, "people do not understand AV testing and the test results."

 

[Andy Ful]

I would not be so cruel for AV scoring tests. Some AVs are scoring very well (and some poor) on many different tests over a couple of years. So, for those AVs is very improbable, that it was by chance.
But, I agree that from looking at the results of the one concrete test, nothing interesting can be concluded.
Personally, on my computers I use Defender + hardening, so it is crucial for me to learn its strong and weak points. I think that posting the paen of prise on Defender would be stupid.
Anyway, I like the fact that Defender security is improving.

 

[Behold Eck]

YouTube tests are not useful for AV scoring, but some of them are very interesting in some other aspects. I like especially some videos posted on MalwareTips (Wilderssecrity) forum, when they are not focused on AVs scoring, but rather on showing something interesting about AV security. Most of those videos cannot be called as homemade, because of their pedagogical quality.

Yes, YouTube testers definitely have their place as they`re a great way to checkout a programs GUI, system impact, compatibility etc and not just detection results.

Of course the quality of the testers vary from the likes of Cruel Sister, Leo, Malware Geek etc to the more "run of the mill"/awful but yeah I think they do a great job.

P.S. Does anyone know how Panda Free keeps getting such good results as I`ve personally never seen it`s BB in action and it can`t all be down to the excellent Web filter, can it ?

Regards Eck