Question Testing security software in a rush with some samples - what do you think?

[Nikos751]

I did some malware testing (VM) in a rush using a malicious document, some stealers and a precracked software (office 2010) with malware inside the installer. I tested (all defaults) Eset, Kaspersky, Norton, Avast, McAfee, BitDefender, while the malicious installer was tested two times. I have no evidence of the test results for you to see as did not take any video or screenshots, just observed and now I want to share my observations with you, so I posted this here just for gossip, talk and comments, not as a test having a high value.

• Norton really has some problems with Stealers, mainly using IPS for detecting them while it only showed some Data Protector warning & blocked activity while installing the cracked software.Kaspersky IS & Eset IS did not appear very consistent as they could not detect the malware inside the office installer when it was running, they both needed right click & scan. Even then, Eset IS did not always detect the threat (I tested 2 times, the first one ended up with a detection, second ended up with no detection - no idea why). Other threats were detected quickly with no infection apparent.
• McAfee IS is not very effective when right clicking & scanning but when malware is being run it detects it. It also detects most unknown threats, but it leaves files as it detects payloads & spawned files malware create, while it's already running. Also, I liked that in 2nd test with the malicious installed (some hours difference from the first), it stopped the installation and deleted the installer as an unknown threat - the first time it detected some vbs here and there.
• BitDefender IS was very good with it's signatures and BB but I was surprised but it did not detect the malicious installer, neither by scanning or by running it.
• Lastly Avast Free, produced the most clean results, no need to run anything, all detections were pre execution, every single time. well done, i was surprised.
  *Samples were not carefully chosen, there was no serious methodology involved, as my free time is limited these days, I just wanted to do some testing and see how it gones with all those vendors.

 

[Jonny Quest]

An on-the-fly rush test still is informative to me, as it could maybe be what a normal day-to-day user may experience with their AV, and it being set on default?

 

[Nikos751]

An on-the-fly rush test still is informative to me, as it could maybe be what a normal day-to-day user may experience with their AV, and it being set on default?

All settings were defaults, yes. Install -> Update -> Test, with Internet Connection On, the host was Windows 10 x64 fully updated.
You should have in mind though, that choosing samples that are really what an average user can come into, requires a complicated procedure; this was just what I thought would be interesting.

 

[oldschool]

Lastly Avast Free, produced the most clean results, no need to run anything, all detections were pre execution, every single time. well done, i was surprised.
*Samples were not carefully chosen, there was no serious methodology involved, as my free time is limited these days, I just wanted to do some testing and see how it gones with all those vendors.

Did you test AVG by any chance?

 

[SeriousHoax]

a precracked software (office 2010) with malware inside the installer.

Kaspersky IS & Eset IS did not appear very consistent as they could not detect the malware inside the office installer when it was running

BitDefender IS was very good with it's signatures and BB but I was surprised but it did not detect the malicious installer, neither by scanning or by running it.

Can you share more details on this pre-cracked office installer? What kind of malware it was? Was it just some kms tools or something else? Products like Kaspersky and Bitdefender may not detect some KMS tools unless they are infected with malware. KMS tools performing its intended function only are not malicious. Was ESET's PUA protection on? It's user dependant at the time of installing ESET so not sure if it is considered default settings or not.
Also Kaspersky by default doesn't notify or stop execution of items detected as "not-a-virus....." It only shows that in its UI and the Kaspersky tray icon turns yellow if I remember correctly.
Just curious about these misses.

 

[Trident]

they could not detect the malware inside the office installer

Because the installer (guessing cracked) is a self-extracting archive (compound file) scanning of archives will have to be enabled for real time protection to extract the file and scan it.

That goes for all antiviruses. I personally always enable archive scanning, but it creates a higher performance overhead. Threats in archives and installers are considered latent with the assumption that you or the archive will extract files and they will be scanned one by one. But frequently, additional threats can be embedded, such as injectors. The default setup where archives are not scanned is not great.

About Avast Free, many people are surprised, as they are not aware of the technology behind Avast. Avast is frequently the first and only one to somehow (be it with web shield, antivirus or something else) block a threat. The assumption that Avast is ineffective is wrong and is bias spread on various forums, including this one, by people who have 0 knowledge of threats, the antivirus industry and Avast as a company.

Today I tested few threats which were only picked up by Avast and the CheckPoint emulation.

 

[a090]

Because the installer (guessing cracked) is a self-extracting archive (compound file) scanning of archives will have to be enabled for real time protection to extract the file and scan it.

That goes for all antiviruses. I personally always enable archive scanning, but it creates a higher performance overhead. Threats in archives and installers are considered latent with the assumption that you or the archive will extract files and they will be scanned one by one. But frequently, additional threats can be embedded, such as injectors. The default setup where archives are not scanned is not great.

About Avast Free, many people are surprised, as they are not aware of the technology behind Avast. Avast is frequently the first and only one to somehow (be it with web shield, antivirus or something else) block a threat. The assumption that Avast is ineffective is wrong and is bias spread on various forums, including this one, by people who have 0 knowledge of threats, the antivirus industry and Avast as a company.

Today I tested few threats which were only picked up by Avast and the CheckPoint emulation.

Any reason why F-Secure doesn’t scan inside archives? I was surprised to read that, but decided it wasn’t a big enough deal to stop my purchase.

 

[Nikos751]

Because the installer (guessing cracked) is a self-extracting archive (compound file) scanning of archives will have to be enabled for real time protection to extract the file and scan it.

That goes for all antiviruses. I personally always enable archive scanning, but it creates a higher performance overhead. Threats in archives and installers are considered latent with the assumption that you or the archive will extract files and they will be scanned one by one. But frequently, additional threats can be embedded, such as injectors. The default setup where archives are not scanned is not great.

About Avast Free, many people are surprised, as they are not aware of the technology behind Avast. Avast is frequently the first and only one to somehow (be it with web shield, antivirus or something else) block a threat. The assumption that Avast is ineffective is wrong and is bias spread on various forums, including this one, by people who have 0 knowledge of threats, the antivirus industry and Avast as a company.

Today I tested few threats which were only picked up by Avast and the CheckPoint emulation.

I appreciate Avast very much, since they rolled out cloud scanning features , sandboxing & evogen several years ago, but I would place it amongst the 3 or 4 best ones. But I did not expect it to surpass every other top tier vendor so easily.
Avast had some problems with BSOD’s some years ago (that was the reason I uninstalled it several years ago), but people do not like it just because of the privacy fiasko and the ads - secondary products being promoted aggressively

The malicious object was a vbs file inside the.exe installer.

 

[Nikos751]

Did you test AVG by any chance?

No, for some reason I have forgot it exists

 

[Kongo]

Did you test AVG by any chance?

shouldn't the results be comparable to the ones from Avast?

 

[Nikos751]

Can you share more details on this pre-cracked office installer? What kind of malware it was? Was it just some kms tools or something else? Products like Kaspersky and Bitdefender may not detect some KMS tools unless they are infected with malware. KMS tools performing its intended function only are not malicious. Was ESET's PUA protection on? It's user dependant at the time of installing ESET so not sure if it is considered default settings or not.
Also Kaspersky by default doesn't notify or stop execution of items detected as "not-a-virus....." It only shows that in its UI and the Kaspersky tray icon turns yellow if I remember correctly.
Just curious about these misses.

Kaspersky actually detected it when scanned via context menu, as a trojan dropper (vbs).
Eset was tested also with PUA on (checked during installation)

I may be able to find that installer again and take a closer look at it and tell you when I find some free time again soon!

 

[Trident]

Any reason why F-Secure doesn’t scan inside archives? I was surprised to read that, but decided it wasn’t a big enough deal to stop my purchase.

By default many of them don't scan in archives (not to say all as there are over 200 antivirus, haven't tried all). Many have additional settings (Bitdefender, Eset, Norton, Trend Micro and others) that activates scanning in archives. Bitdefender warns that this reduces performance and is not needed as archives will be scanned on as-used basis. Some products have no setting to enable archive scanning (McAfee, ZoneAlarm, potentially F-Secure). These products rely on the assumption that once in use, all archive content will be scanned. Behavioural blocking and other components should stop the attacks.

shouldn't the results be comparable to the ones from Avast?

They are, the 2 products use the same codebase.

but people do not like it just because of the privacy fiasko.

No, there are many people that go around telling stories how ineffective it is. I've seen these stories here as well. Told by people who can't establish whether a pdf document contains malware or not. They know nothing about malware but they believe they can measure the effectiveness of an antivirus product.

 

[a090]

Great answer, as always. Thanks mate @Trident.