AVLab.pl Learn more about Remediation Time – response time to security incidents (the results from protection test in January 2023)

[Adrian Ścibor]

Testing and reporting "remediation time" is a bit problematic. First and foremost, the results generate incorrect interpretations amongst those that do not know any better and subsequent false statements about the products. For example, mischaracterizations and inaccurate statements about "post-launch" and also that "a product that permits post-launch and a longer remediation time results in a compromised system."

I know you explained it here, but you know, this is a forum where people are apt to mis-interpret and draw incorrect conclusions because they think they know (definitely not the first time you have experienced this phenomena):

View attachment 273176

A security product can have a very long remediation time due to networking or routing issues, while at the same time keep the system 100% clean after the remediation. Some products hold a suspicious process(es) in a suspended state and permit them to make no system modifications during the "remediation" process, others reverse any changes that were made, etc. This is a crude analysis, but comparing remediation times is like comparing an apple, a banana, and a tomato. They are all fruits, but completely different.

In order to qualify a result on a sample as positive for POST-Launch, each product tested has its own logs, which we monitor and which are analysed by machine after collecting the logs from the test on a given sample. It does not matter the type of technology that first reacted to the suspicious activity, as this is often difficult and complicated to determine and it is not possible for every vendor to show such differences.

Remediation Time simply means that the product has responded to the suspicious activity in any way. We don't check what technology it reacted with, because that would just complicate things. Each product is configured to automatically perform a 'remediation': move to quarantine, kill the process, perform another action if that's not possible.

That's why we don't rank on samples: blocked by 'user decision'. Instead, the AV product does to automatically, and we "capture" the result.

Perhaps Remediation Time needs to be clarified. A different presentation. Maybe this discussion will allow us to draw conclusions about what we can change.

 

[Adrian Ścibor]

We agree that Including morphed samples in the test is not a good idea for several reasons. So, there is a question: how these files (polymorphic, metamorphic, and oligomorphic) are managed in the automated scenario?
Did Bitdefender miss 13 samples due to morphed malware or there were 13 different malware?

I was mean polymorphism created manually by testers to increase the number of malware samples. In contrast, you find samples from a similar family with an altered counter sum normally in the wild. Malware authors or script kiddies think alike and have access to similar tools. So you may find similar samples at different URLs/IP addresses from the other side of the world.

The particular AV results (missed samples) are not important, as you noticed. The problem is that the test shows an upside-down picture of how one of the tested AVs normally works (most detections should be Pre-Launch instead of Post-Launch and remediation time is probably incorrect too). That AV works so on about 5% of computers in the world. I am not sure if showing this was the intention of the testers. Furthermore, that AV is not tested on default settings (BAFS does not work).
The same problem would be with a giraffe mutant with a short neck, presented in the ZOO as the typical example of a giraffe species.
Of course, AVLab can still use Firefox and simply add the note, that Defender's BAFS does not work in the test. No problem.

Ok, this is noted. FYI - Dear Community.
From the next edition, each vendor in our test will have a separate webpage, so we can provide you more marketing and technical data. Give us a chance to improve.

 

[Digmor Crusher]

Lots of credit to you guys Adrian for the willingness to discuss and possibly change your testing methods. Many others seem to be set in stone as to how they do it.

 

[Adrian Ścibor]

Lots of credit to you guys Adrian for the willingness to discuss and possibly change your testing methods. Many others seem to be set in stone as to how they do it.

Many thanks, much appreciated. I hope we won't let you down. By the way, I invite each one of you to get in touch with me on my LinkedIn and engage in discussions on various topics.

 

[Adrian Ścibor]

To compare the 3 categories of products accurately (and not apples with oranges), post-launch vs pre-launch ratio and remediation time are not important at all. More important would be protected vs compromised.

To establish the difference between compromised and protected, it would be vital that all samples are categorised/labelled first.
You can’t perform a proper test if you don’t know what you are dealing with and where to look.

The final payload will always be one or more of the following:

Infostealers:
Both pre-launch and post-launch can be considered protected, unless there was Credentials Access or other type of exfiltration. Access to certain folders as well as the network traffic will have to be inspected. If data could not be sent back “home”, e.g Intrusion Prevention/Web Filtering suspended the connection or behavioural analysis terminated the infection chain on time the solution protected the system. Otherwise it is compromised.

Ransomware:
Both pre-launch and post-launch could be counted as protected, unless there are files encrypted and this could not be reversed by the product.
In that case it should be counted as compromised.

Thanks @Trident. I will consider your opinion. THAT should make things a lot simpler, because in the end it's the result that matters anyway, right?

 

[Trident]

Thanks @Trident. I will consider your opinion. THAT should make things a lot simpler, because in the end it's the result that matters anyway, right?

Indeed. For majority of users it would matter whether or not their CV and wedding pictures got encrypted, and whether or not the passwords saved in Chrome got harvested.
The pre-launch and post-launch is a bit more detailed information (frequently YouTubers do it too) but it doesn’t really provide enough information on whether or not files and information were really kept safe.

Since you use monitoring and logging, I guess you should be able to conclude if the system was compromised before remediation or if the product quickly terminated the infection chain before anything could be done(that is not linked to remediation time as most of the malware employs long sleeps with the belief that it will evade emulation and virtualisation). In the event of a ransomware infection you should be able to tell if the product could reverse the encryption (if it was performed).
Behavioural blocking may have handled all related processes, services, files, folders, scheduled tasks, shortcuts, registry entries and even notes, but that doesn’t help a lot if it wasn’t done on time/files were left encrypted.

For products that offer implementation of controlled access to folders, you can consider monitoring these folders only - if a product offers certain protection feature, it is not there to beatify the UI. Users are expected to read the product guide and make use of all features properly.

Not sure who considers which products to submit, but it’s important to note that Bitdefender for example, offers AMSI integration (script scanning in settings), command line scanning and memory scanning (both new) only in its paid versions. Vendors may wish to reconsider what’s tested.

 

[ForgottenSeer 98186]

Many thanks, much appreciated. I hope we won't let you down. By the way, I invite each one of you to get in touch with me on my LinkedIn and engage in discussions on various topics.

Be careful. We'll add 2000 man-hours of extremely tedious work to your test methodology. You know... security geeks.

If you start testing the time-to-block, how it was blocked, was the process terminated, by what mechanism, are there inactive or active remnants, has data been exfiltrated?,etc - then you are performing a type of testing and investigation that no other AV lab has done before.

 

[Trident]

Be careful. We'll add 2000 man-hours of extremely tedious work to your test methodology. You know... security geeks.

If you start testing the time-to-block, how it was blocked, was the process terminated, by what mechanism, are there inactive or active remnants, has data been exfiltrated?,etc - then you are performing a type of testing and investigation that no other AV lab has done before.

You made me laugh!

 

[Adrian Ścibor]

Be careful. We'll add 2000 man-hours of extremely tedious work to your test methodology. You know... security geeks.

If you start testing the time-to-block, how it was blocked, was the process terminated, by what mechanism, are there inactive or active remnants, has data been exfiltrated?,etc - then you are performing a type of testing and investigation that no other AV lab has done before.

I do not think that 2000 h is enough for analysis and programming

 

[Adrian Ścibor]

The rest files (including the attack vectors via scripts, shortcuts, DLLS, documents, legal but vulnerable EXE files, LOLBins, etc).

Interesting... Might be used to build interesting data stats.

In fact, we use them for:
a. monitoring potentially harmful processes eg. PsExec.exe, CertUtil.exe, Reg.exe, wscript.exe (more in attach)
b. suspicious and potentially harmful folders often used by malware to extract themselves or save droppers e.g. %TEMP% or c\:windows\*.exe
c. potentially harmful keys in the windows registry, e.g. *\Autorun

We could create something like a TOP10-TOPN of the most used LOLbins in the test.

 

[ForgottenSeer 98186]

In fact, we use them for:
a. monitoring potentially harmful processes eg. PsExec.exe, CertUtil.exe, Reg.exe, wscript.exe (more in attach)
b. suspicious and potentially harmful folders often used by malware to extract themselves or save droppers e.g. %TEMP% or c\:windows\*.exe
c. potentially harmful keys in the windows registry, e.g. *\Autorun

I can provide you with a large curated list outside of MalwareTips. It also includes incidence ("rate of abuse") of Windows utilities and LOLBins. This is historical data. Send me a PM if it is of any interest.

 

[Adrian Ścibor]

I can provide you with a large curated list outside of MalwareTips. It also includes incidence ("rate of abuse") of Windows utilities and LOLBins. This is historical data. Send me a PM if it is of any interest.

Something different than from LOLBAS ?

 

[ForgottenSeer 98186]

Something different than from LOLBAS ?

Yes