AVLab.pl Learn more about Remediation Time – response time to security incidents (the results from protection test in January 2023)

[Andy Ful]

@Oerlink,

The particular AV results (missed samples) are not important, as you noticed. The problem is that the test shows an upside-down picture of how one of the tested AVs normally works (most detections should be Pre-Launch instead of Post-Launch and remediation time is probably incorrect too). That AV works so on about 5% of computers in the world. I am not sure if showing this was the intention of the testers. Furthermore, that AV is not tested on default settings (BAFS does not work).
The same problem would be with a giraffe mutant with a short neck, presented in the ZOO as the typical example of a giraffe species.
Of course, AVLab can still use Firefox and simply add the note, that Defender's BAFS does not work in the test. No problem.

 

[ForgottenSeer 98186]

@Oerlink,

The particular AV results (missed samples) are not important, as you noticed. The problem is that the test shows an upside-down picture of how one of the tested AVs works (most detections should be Pre-Launch instead of Post-Launch and remediation time is probably incorrect too). That AV works so on about 5% of computers in the world. I am not sure if showing this was the intention of the testers.
The same problem would be with a giraffe mutant with a short neck, presented in the ZOO as the typical example of giraffe kind.
Of course, AVLab can still use Firefox and simply add the note, that Defender's BAFS does not work in the test. No problem.

I understand your point of view and I am not dismissing it.

I look at the test results this way - with or without BAFS - Microsoft Defender provides exceptionally good protection. To me personally, it does not matter if it is pre- or post- or remediation time is short or long. All I care about is whether or not the system is protected.

It is disturbing that Microsoft is so anti-Firefox - to the extent that it altered its web-based app code years ago, so those webpages ran slower in Firefox. Anyone remember that scandal?

 

[Trident]

I look at the test results this way - with or without BAFS - Microsoft Defender provides exceptionally good protection.

Exceptionally good protection is highly exaggerated. Microsoft Defender on this test as well as on many others has been proven to offer protection equal to the one offered by many other products. Nothing more, nothing less. Nothing to wow about or write home and definitely nothing “exceptionally good”. It is merely doing its job.

In regards to @cruelsister eternal love for scripts, I personally love to use unusual vectors but such test is not really a must. Majority of threats for home users still come as executable files.

It is disturbing that Microsoft is so anti-Firefox - to the extent that it altered its web-based app code years ago, so those webpages ran slower in Firefox. Anyone remember that scandal?

I will have to agree here. I also don’t see any reason why the methodology of AVLab Poland should be modified to accommodate Microsoft’s ancient and never ending browser war.
Firefox is not a depreciated project, it is is still somewhat popular browser that any user can download. Microsoft does not issue any proper warnings or documentation to inform the user about certain features of questionable cruciality being unavailable whilst using Firefox.

If protection is really compromised by the usage of not-recommended software, this should either be communicated in a clear language so users can make choices, or effort should be put for the software in question to be supported.

Also, I can not agree with @Oerlink ‘s opinion that whether it is pre-launch or post-launch doesn’t matter. When it comes to info-stealers which make a large percentage of malware (probably 2/10 home users would consider paying ransom when the majority of their important information is on their mobile devices— not really profitable). Detecting an info-stealer/RAT just seconds later is already too late and no remediation can be performed. Data is already exfiltrated, all passwords must be changed.

I personally found many facts that I can question in this test apart from Microsoft Defender, but I will keep them to myself for now, in order not to cause an unnecessary debate here.

 

[Andy Ful]

I personally found many facts that I can question in this test apart from Microsoft Defender, but I will keep them to myself for now, in order not to cause an unnecessary debate here.

You can PM the author of this thread.

 

[Trident]

You can PM the author of this thread.

For now I will let all questions slip and I will wait to see few tests more. I will ask many questions if certain tendencies continue or don’t.

 

[cruelsister]

Therefore, adding copies of the same files, I don't think is a good recommendation.

Agreed. But it is far easier for a Pro testing site to just accumulate samples appearing in the Wild that are essentially duplicates (MAAS is, after all, increasingly popular) and others that may actually be malformed than to do analysis on each to verify uniqueness and maliciousness. Although this will obviously juice up the numbers to appease the more is better crowd, it will sadly also dilute the AM effectiveness conclusions.

My personal feeling is that the quantity of malware samples used in a test is inversely proportional to conclusions regarding AM product efficacy. However limiting (handpicking) malware samples can also be overdone and may result in the possibility of bias.

 

[Trident]

I see 3 types of products on this test.

-The so called “standard antivirus” which uses signatures, reputation, machine learning, behavioural analysis, behavioural blocking and others;
Examples: Bitdefender, Avast, Avira

-Mainly cloud-first products that rely on cloud detonation and hashing for majority of their detections;
Examples: Microsoft Defender, Immunet by Cisco;

-Prevention-first products which don’t care enough to identify malware via the means above, but rather attempt to stop any damage.
Examples:
Xcitium

Samples seem to have been pulled from honeypots and classified automatically based on over 100 rules. This is described on AVLab’s official page.

To compare the 3 categories of products accurately (and not apples with oranges), post-launch vs pre-launch ratio and remediation time are not important at all. More important would be protected vs compromised.

To establish the difference between compromised and protected, it would be vital that all samples are categorised/labelled first.
You can’t perform a proper test if you don’t know what you are dealing with and where to look.

The final payload will always be one or more of the following:

Infostealers:
Both pre-launch and post-launch can be considered protected, unless there was Credentials Access or other type of exfiltration. Access to certain folders as well as the network traffic will have to be inspected. If data could not be sent back “home”, e.g Intrusion Prevention/Web Filtering suspended the connection or behavioural analysis terminated the infection chain on time the solution protected the system. Otherwise it is compromised.

Ransomware:
Both pre-launch and post-launch could be counted as protected, unless there are files encrypted and this could not be reversed by the product.
In that case it should be counted as compromised.

PUAs:
Successfully installing and running the PUA in question should be considered an indicator of compromise.

Rarely the final payload may be a coinminer, another loader or C&C may be dead already. In this case just deleting the malware would be enough to consider the system protected.

 

[Andy Ful]

Eureka. Now I can see how simple is the malware landscape.

 

[Trident]

Eureka. Now I can see how simple is the malware landscape.

The “malware landscape” for years has been copy/paste, changing evasion and distribution tactics, but the main three threats since the fake AVs disappeared years ago have remained ransomware, infostealers/RATs and malware-as-a-service operation (loaders) which is frequently the entry point. There is hardly any innovation in the “malware landscape”. By pulling 10 random threat researches from different companies and different years what I’ve said can easily be proven.

Here is the newest, brightest and shiniest star of the malware landscape. Based on not so new and shiny groups, tactics and codebases we’ve been observing for years.

Stealc: a copycat of Vidar and Raccoon infostealers gaining in popularity - Part 1

Stealc infostealer is another fully featured infostealer sold as a MaaS which emerged on underground forums in early 2023.

blog.sekoia.io

 

[Andy Ful]

I could introduce a simpler classification: "ransomware" and "the rest". There is no need to multiply entities beyond necessity.

 

[Trident]

I could introduce a simpler classification: "ransomware" and "the rest". There is no need to multiply entities beyond necessity.

And maybe that’s an even better way to label them.
Simpler and shorter.

But the main point remains that this test is a mixed bag and the remediation time, and ratio of threats blocked instantly vs 200 s later, hardly allow any valuable conclusion to be drawn for any of the products tested. The compromised/user dependant and protected system has been implemented by some other testing labs for a reason.

 

[ForgottenSeer 98186]

Exceptionally good protection is highly exaggerated.

A 99+% detection rate is exceptionally good.

Microsoft Defender on this test as well as on many others has been proven to offer protection equal to the one offered by many other products. Nothing more, nothing less.

It does not matter how other AV perform. 99+% is an absolute measure. If they all consistently produce 99+% detection\protection over time at various AV labs, then they all provide "exceptionally good" protection - assuming - that the tests are well-designed and executed.

Detecting an info-stealer/RAT just seconds later is already too late and no remediation can be performed. Data is already exfiltrated, all passwords must be changed.

This is not factual.

But the main point remains that this test is a mixed bag and the remediation time, and ratio of threats blocked instantly vs 200 s later, hardly allow any valuable conclusion to be drawn for any of the products tested.

@Adrian Ścibor

Testing and reporting "remediation time" is a bit problematic. First and foremost, the results generate incorrect interpretations amongst those that do not know any better and subsequent false statements about the products. For example, mischaracterizations and inaccurate statements about "post-launch" and also that "a product that permits post-launch and a longer remediation time results in a compromised system."

I know you explained it here, but you know, this is a forum where people are apt to mis-interpret and draw incorrect conclusions because they think they know (definitely not the first time you have experienced this phenomena):

A security product can have a very long remediation time due to networking or routing issues, while at the same time keep the system 100% clean after the remediation. Some products hold a suspicious process(es) in a suspended state and permit them to make no system modifications during the "remediation" process, others reverse any changes that were made, etc. This is a crude analysis, but comparing remediation times is like comparing an apple, a banana, and a tomato. They are all fruits, but completely different.

 

[Oldie1950]

A 99+% detection rate is exceptionally good.

If a larger number of other AV programs also have a recognition rate of over 99%, then over 99% is conceptually no longer exceptional but common.

 

[ForgottenSeer 98186]

If a larger number of other AV programs also have a recognition rate of over 99%, then over 99% is conceptually no longer exceptional but common.

The word "exceptional" can be used as a qualitative descriptor - such as "outstanding." The word has multiple meanings and usage. If y'all want to argue semantics - that the word "exceptional" ONLY means "better than what is typically found," then go right ahead. It still does not change the fact that 99+% detection rate is exceptionally good if consistent over time and across different tests.

Protection is measured on an absolute scale, and not a relative one. It would be idiotic to call a software that provides 60% detection "exceptional" because all the others to which it is being compared is providing less than 20% detection.

 

[Trident]

@Oerlink ,

I’ve observed few products (one of them being included on that test) failing to remediate RATs detected via dynamic analysis just seconds after execution. Some have improved after my reporting, one of them did not and still has this problem to date (it is not on this test). They have not asked for a reboot and they’ve claimed that threats are “fixed” and “secured” when in fact inspection of processes (where it has been injected) as well as network inspection showed that the RAT is actively listening based on its configured period of refreshment.

It was on that same test that I had created and saved accounts in Chrome Auto-Fill for testing purposes and suspicious log-in activity was observed not long after threats were “secured” (2-3 days) on few of them: Discord, Outlook (still to this day activity continues and the research was 2 years ago, it is a very large botnet), PayPal, Gmail and even funnier - an adult website (very popular one). Although I had saved virtual cards there was no evidence that this particular group was interested in that.
Sadly these accounts were all fake and did not provide the crypto that they seek.

I later on discovered these accounts on a Telegram channel. It shined some light on some connections that we didn’t know about.

I trust that some of them terminate the infection chain early and correctly, but the only way to be sure is to inspect and not to assume.

As for the remediation time itself, it doesn’t mean anything, it may have been a long sleep even. Again, no conclusion can be drawn just based on that.

 

[ForgottenSeer 98186]

@Oerlink ,

I’ve observed few products (one of them being included on that test) failing to remediate RATs detected via dynamic analysis just seconds after execution. Some have improved after my reporting, one of them did not and still has this problem to date (it is not on this test).

Can you provide the malware sample hashes?

@Oerlink ,

They have not asked for a reboot and they’ve claimed that threats are “fixed” and “secured” when in fact inspection of processes (where it has been injected) as well as network inspection showed that the RAT is actively listening based on its configured period of refreshment.

It does happen, doesn't it? Unfortunately. And it is not applicable only to InfoStealers. But it is the exception, and not the rule.

I trust that some of them terminate the infection chain early, but the only way to be sure is to inspect and not to assume.

@Adrian Ścibor

After each remediation test, did you check for malicious processes running in-memory, start-up tasks, network traffic, and other active IOCs?

The vast majority of users will not "inspect." Heck, they won't even know what is going on.

 

[Trident]

It does happen, doesn't it? Unfortunately. But it is the exception, and not the rule.

I think that for one company that employs 5K people and secures some of the largest businesses, also publishes extremely high-quality research on their website it is not the exception… It’s the rule. It’s not applicable only to infostealers but I am mainly passionate about them. I am not so interested in ransomware and other threats (unless they include exfiltration abilities as well).

It was not one sample, it was a lot from 2 families: one of them notorious for its love for Explorer.exe and for being distributed as a signed executable, and the other family was script-based. At the time of discovering the opendir, there were hundreds of scripts there calling each other, and all of them had detection rate of maximum 2-3 on VT. After my daily submissions (the opendir was updated few times a day) things got a bit better, but in both cases I observed inability for the final payload to be terminated. Infection chain was MS Office Document -> JS -> VBS-> LOLBin. It would also check certain directories to identify installed AV and download a tailored “Bypass” as they called it on the Middle Eastern forum.
I trust this information will be enough to pinpoint which threats exactly I am talking about.

But I can also get samples and hashes as well.

If you wanna have a nice sweet talk about this and other malware, feel free to PM me. This is not the right place for it.

 

[ForgottenSeer 98186]

But I can also get samples and hashes as well.

I take your word for it. I appreciate the offer to get samples, but I was only interested in hashes so that I can look-up malware sandbox analysis reports. By this point in time the samples you tested are no longer connecting out to active C&Cs or data collection points.

 

[Trident]

I take your word for it. I appreciate the offer to get samples, but I was only interested in hashes so that I can look-up malware sandbox analysis reports. By this point in time the samples you tested are no longer connecting out to active C&Cs or data collection points.

I will provide you with few hashes, PM me.

 

[ForgottenSeer 98186]

I will provide you with few hashes, PM me.

Thank you. Please send me a PM because PM creation has been disabled for my account.