AVLab.pl Analysis of system protection against active online malware – July 2025

[Adrian Ścibor]

In the cases of TLSH (or SSDEEP potentially) usage, depending on bytes and distance allowance, it could be the same detection. Yes, it can be a new one as well.

That's just too much work and modifying samples by playing with bytes doesn't really create zero-days.

Exactly. A modified sample that does the same thing may not be a 0-day for Vendor 1, but it could be a 0-day for Vendor 2. In fact, a "0-day test" will not necessarily be valuable. Instead, it might be worth focusing on finding URLs/threats for testing, which is always a problem. Paradoxically, there isn't much that's unique about it.

 

[Andy Ful]

In-the-wild samples can be modified in many ways:

1. Adding/changing fake certificates. Evasive threats often use signed (usually fake) certificates.
2. Adding/removing/changing file resources from benign executables (in a passive way = unused in the runtime).
3. Changing the malware by adding another layer of obfuscation. It can be useful for scripts and scriptlets.
4. Changing hardcoded URLs.
5. Using popular Red Team tools (Metasploit, Empire, etc.) to create malware. Those tools are well known by AVs, so it is necessary to use points 1 - 3.
6. Using slightly modified GitHub offensive repositories.
7. etc.

All of this (except probably points 1 and 2) requires an experienced coder. It is not necessary to produce many samples. About 50 samples per month looks OK.

 

[Trident]

A lot can be done indeed.

Against business users, all these methods can void certain detections and provide insight what the AV will do in cases of targeted attacks.

But against home users, I don’t see a reason to spend time and effort modifying messy malware creations, when attackers spend typically 12-13 hours a way doing exactly that. There is no deficiency of in-the-wild malware.

Scripts can be modified as well by the addition of custom functions (which I’ve used for some tests).

Majority of script detections are based on AMSI (obfuscations are already peeled, regardless how many you will layer), as well as on calls and command lines.
Adding obfuscations may be the actual reason why malware is being detected, most AVs struggle not when typical encodings, replacements, concatenations and so are used, but on readable scripts that look like genuine IT admin dropped them.

 

[Andy Ful]

But against home users, I don’t see a reason to spend time and effort modifying messy malware creations, when attackers spend typically 12-13 hours a way doing exactly that. There is no deficiency of in-the-wild malware.

Tests with modified samples can roughly show how AVs can depend on post-factum detections (mainly signature detections). Those detections can significantly alter the in-the-wild protection. You must retest some detected samples after making modifications.
Due to a significant delay, AV testing labs create/test/report their own reality. When you try to interpret the results, it is like concluding that a person is nice because of a nice facial profile.

Post updated.

 

[New_Style_xd]

Sandbox failed once again, is anyone using it still safe?

Recent Results » AVLab Cybersecurity Foundation

The latest results of the Advanced In-The-Wild-Malware Test, in which we regularly evaluate software designed to protect devices.

avlab.pl

 

[Andy Ful]

If Comodo is unsafe, then what would you say about this?

It is risky to make conclusions on the example of one particular test.

 

[ForgottenSeer 123960]

These tests aren't a perfect measure, they are just a baseline to evaluate a product's general effectiveness.

 

[Trident]

If Comodo is unsafe, then what would you say about this?

View attachment 290490


It is risky to make conclusions on the example of one particular test.

Bitdefender free doesn’t include memory and command lines scanner. Not sure how and when command line scanner works but I am assuming this is on runtime… not sure. I did check the hashes back then, did not rescan the files, some files had detections by Bitdefender on VT (the scan dates were 20-30 days prior). Maybe the product could have glitched (bug/connection issue)?

Because Bitdefender usually does not miss 13 malware samples on any other test.

 

[New_Style_xd]

The mental gymnastics have begun here about the problem that is happening.
Someone comes and puts an image from January 2023... and it's really funny... lol

 

[New_Style_xd]

People must understand that Comodo's sandbox was not supposed to let anything run outside the box, with the lack of sandbox updates it is passing threats.

 

[Trident]

People must understand that Comodo's sandbox was not supposed to let anything run outside the box, with the lack of sandbox updates it is passing threats.

I’ve said that very very long time ago… ages ago. When you don’t update a component that comes in direct contact with malicious code, this component sooner or later will become prone to evasions, in the case of sandbox escapes, exploits, performance issues (as operating systems introduce new APIs) and so on.

But you must understand that this is a free product. For quality, users should get their wallet and pay up.
Comodo is offered as-is. Don’t like it? Don’t use it.

 

[cartaphilus]

Most classified systems are air-gapped. The route of infection would typically be external storage, assuming that rigorous standard security protocols are adhered to habitually. For the few that are isolated classified LANs there's sometimes connected backup and network storage, but this varies by a nation's security requirements for such machines and networks. (The security requirements can be extremely onerous and expensive - as in 5 million Euros for a small classified LAN) and require regular, routine inspections and audits. Other classified networks are logically separated by KVM switches or hardened switch-router-hardware I&A combos.

AV-Comparatives does occasional off-line testing and the results reveal the dependence upon cloud. Online detection rates plummet from the high 90%s down to the 80%s.

I get the infection route testing, but when it comes to these systems executable code virtually never comes directly from the internet. So the only effective test is execution from the desktop or executable storage.

As far as emanation attacks on air gapped systems they are more or less eavesdrop/electronic warfare types of attacks - as opposed to attempts to get malware onto the systems. There were POCs back in the day such as Tempest and Van Eck. Plus others.

The required proximity (effective distance) from the target system has grown substantially over the decades as sensor technology has been refined and made more effective.

Even if Kaspersky somehow gets installed on SIPR, JWICS or SIC it won't go anywhere. The networks don't communicate with NIPR so no K server will ever gain connection.

 

[Trident]

Even if Kaspersky somehow gets installed on SIPR, JWICS or SIC it won't go anywhere. The networks don't communicate with NIPR so no K server will ever gain connection.

Well if you read the system requirements, they (not sure about Kaspersky but most vendors) tell you that a high-speed (by high speed they mean 20-30 mbps) connection is required. It is your duty to ensure that the security software can connect to the internet.

That’s the same like complaining Netflix, or the Weather app or Spotify are unable to pull the latest info/content. You need to connect to a network and then they will.
This is not different.

 

[cartaphilus]

Well if you read the system requirements, they (not sure about Kaspersky but most vendors) tell you that a high-speed (by high speed they mean 20-30 mbps) connection is required. It is your duty to ensure that the security software can connect to the internet.

That’s the same like complaining Netflix, or the Weather app or Spotify are unable to pull the latest info/content. You need to connect to a network and then they will.
This is not different.

Indeed. Hence 3L agencies run Trilix or Symantec because they run their own database servers on those trunks.

Kaspersky wasn't/isn't an issue for US GOV agency networks. It's an issue for stupid employees bringing work "home" and working at their home computers which might be running KAV.

 

[New_Style_xd]

But those who pay for PRO are still left without protection due to this flaw.

 

[Trident]

Indeed. Hence 3L agencies run Trilix or Symantec because they run their own database servers on those trunks.

Kaspersky wasn't/isn't an issue for US GOV agency networks. It's an issue for stupid employees bringing work "home" and working at their home computers which might be running KAV.

Trend Micro as well. But Trellix, Symantec and TM all use cloud based machine learning. For businesses, TM added a very minimal client-side model. On home versions it’s not there. McAfee/Trellix can resort just to client side AI, sometimes it attempts to verify detections with the cloud (this can be switched off).
For Symantec, they use local models, but rely on network to verify the detection, sometimes the file can turn out to be trusted and the detection can be reversed by the cloud.

This all can be verified by checking the vendor documentations and guides.

They do not operate on full capacity when disconnected.

 

[Andy Ful]

The mental gymnastics have begun here about the problem that is happening.

Your problem is a pure illusion. One missed sample in one particular test is statistically insignificant. AVLab correctly counted as Excellent all AVs that missed 0-1 sample in the latest test.
The statistically insignificant result means that from this test is impossible to conclude that AVs with 0 missed samples were better in any way in the wild than AVs that missed 1 sample.

You could read the same in the OP:

In practice, 100% in a test sometimes does not mean 100% in reality – because a test always has a limited sample set. This may be a coincidence (luck in sample selection) rather than the actual superiority of the product. This is why some people may criticize option 1) because no one can test the entire population of malware or guarantee that there will be no errors in the tests.

https://malwaretips.com/threads/ana...nline-malware-–-july-2025.137258/post-1139745

 

[ForgottenSeer 114717]

Even if Kaspersky somehow gets installed on SIPR, JWICS or SIC it won't go anywhere. The networks don't communicate with NIPR so no K server will ever gain connection.

This is correct, but I'm unsure where Kaspersky comes into this discussion. Guess I'm not paying attention or reading every post in the thread.

Kaspersky wasn't/isn't an issue for US GOV agency networks. It's an issue for stupid employees bringing work "home" and working at their home computers which might be running KAV.

Kaspersky is an issue for US GOV agency networks due to the data collection and file harvesting. Not all US GOV networks are disconnected from the internet - and one of the reasons Kaspersky got banned was that it uploaded secret files from US Government and contractor systems to Kaspersky. Now, to his credit, Eugene Kaspersky did quickly inform the US Government and the contractors about the file uploads, but the prevailing view is that it does not matter much when the files are already out of your control - and therefore the secrecy is lost. Same happened in the UK and Lithuania. Same experience with Kaspersky being a huge data collector and file grabber. The overall concern being that while there is no active cyber-espionage being performed by Kaspersky using its software that has been proven to date, it certainly has the capabilities to do so. Combine that with the fact that there's a significant number of ex-KGB and FSB personnel at Kaspersky - or their family members are ex-KGB or FSB, it just comes down to a matter of being prudent not to use Kaspersky products within government systems.

Much of the issue centers around Eugene Kaspersky's very close relationship with the FSB. Kaspersky is naturally pro-Russian and will do what is required if requests are made by the Russian government. Eugene and his company always have worked against the US Government and NATO nations in subtle ways.

There's way more to it than that, but since no one is going to publicly release the intelligence there's a lot of misinformation circulating about the Kaspersky ban that is not accurate.

 

[lostpass]

Ivan Kwiatkowski:

I have resigned from my position in @Kaspersky's GReAT team. I'm very grateful for my time there and everything the team accomplished. I don't have any reason to believe anything I wrote about the company was untrue at the time I wrote it. I will now take a vacation away from infosec and announce future plans when they shape up!

"It's easy to guess why he changed his mind and left Kaspersky.

 

[ForgottenSeer 114717]

But you must understand that this is a free product. For quality, users should get their wallet and pay up.
Comodo is offered as-is. Don’t like it? Don’t use it.

The state of Comodo is not only determined by the fact that it is freeware with inadequate back end support.

The software was created for ideological purposes - and was never intended to be a polished, refined security software. Melih did try to keep it updated there for a while, but as time passed he just has not kept up for whatever reason(s). I suspect he expected much better reception from users - and that many would be willing to pay - but that's not how it turned out. The lesson is to never offer anything free because it will never obtain the financial support required.

As the software owner, Melih's real intent - the whole purpose of creating CFW/CIS - was/is to prove that the industry ideologies and models are deeply flawed. While that might be true, he just doesn't understand that producing a seriously under-supported product makes many not take him or his points seriously.