AV-Comparatives AV-Comparatives - Real World Protection Test Feb-May 2024

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Trident

Level 34
Verified
Top Poster
Well-known
Feb 7, 2023
2,349
Did you know that the signature engines on VirusTotal are not the ones that are in the consumer and enterprise products (this has always been the case)? Did you know that vendors copy each others' signatures from VirusTotal (Proven by Eugene Kaspersky and his team)?
The signatures of each vendor on VirusTotal don't necesarily have to be what's deployed on a wide-scale production environments.
VirusTotal is the ideal playground to test detection methods before they are pushed to customers.
This is not a secret to anyone and is made clear on VirusTotal.

Copying signatures is impossible as most (not all, for example I've managed to view the database of some Indian AVs like NetProtect) are encrypted and current computing methods will be unable to break the encryption, for a company to view the signatures. Whether or not memory dumping can help extract them, I've not tested and can't say. The tops names would definitely implement proper anti-debug logics.

It is possible for malware analysts and even automated systems to view detections on VirusTotal and copy the detections names, which are in no way protected by the law (not considered proprietary intelligence). Also, many vendors use third-party feeds, which is where similarities may have been derived from. But what they copy is the name, not the signature itself.
 
Last edited:

kailyn

Level 2
Jun 6, 2024
85
The signatures of each vendor on VirusTotal don't necesarily have to be what's deployed on a wide-scale production environments.
VirusTotal is the ideal playground to test detection methods before they are pushed to customers.
This is not a secret to anyone and is made clear on VirusTotal.
Theoretically the use of VirusTotal to "experiment" might be possible but unlikely. The primary reason that the signature engines are modified as they are is because of VirusTotal's integration requirements.

Copying signatures is impossible as most (not all, for example I've managed to view the database of some Indian AVs like NetProtect) are encrypted and current computing methods will be unable to break the encryption, for a company to view the signatures. Whether or not memory dumping can help extract them, I've not tested and can't say. The tops names would definitely implement proper anti-debug logics.

It is possible for malware analysts and even automated systems to view detections on VirusTotal and copy the detections names, which are in no way protected by the law (not considered proprietary intelligence). Also, many vendors use third-party feeds, which is where similarities may have been derived from. But what they copy is the name, not the signature itself.
Vendors do not literally copy one company's signature for a file and include it in their signature database. They copy each others' detections and then create their own signatures. It was proven by Eugene Kaspersky and his team.

Nobody said that they are literally copying each others' signatures. That is not what is meant within the context of the discussion. "Copying signatures" means they all copy each others detections. That is what Eugene Kaspersky meant when he said "copying signatures." There is no question that AV vendors do it. It is not illegal but it does have an element of theft of another company's work product. Eugene Kaspersky and other AV owners think it is underhanded and wrong because they invested heavily in created strong detection algorithms and methodologies, and other companies copy their detections. That seems entirely a legit, reasonable gripe.
 

Trident

Level 34
Verified
Top Poster
Well-known
Feb 7, 2023
2,349
Theoretically the use of VirusTotal to "experiment" might be possible but unlikely. The primary reason that the signature engines are modified as they are is because of VirusTotal's integration requirements.
The requirements of VirusTotal include usage of command line scanner which is what VT uses. The rest is all at the vendors discretion. A lot of them deploy experimental machine learning models, not yet tested for false positives and other heuristics. Some of them produce different names, for example the Symantec Heur.AdvML.A,B,C,D in products, is renamed Malware.AI.Low confidence/high confidence, etc.
Bitdefender for years has deployed an engine on VT that is nowhere to be found in official products.

Nobody said that they are literally copying each others' signatures. That is not what is meant within the context of the discussion. "Copying signatures" means they all copy each others detections. That is what Eugene Kaspersky meant when he said "copying signatures." There is no question that AV vendors do it. It is not illegal but it does have an element of theft of another company's work product. Eugene Kaspersky and other AV owners think it is underhanded and wrong because they invested heavily in created strong detection algorithms and methodologies, and other companies copy their detections. That seems entirely a legit, reasonable gripe.
Oh I understand now what you mean.

Well it may be possible for some action to be taken in this case, depending on how severe it is.
But Kaspersky is mostly based on heuristics where every single one will identify thousands of mutations. Vendors-thiefs will just create some hash-based detection that (if fuzzy hashing is used) may last few variants.
This won’t help them protect users.
 

nickstar1

Level 10
Verified
Well-known
Dec 10, 2022
452
let's also not forget Avast/AVG does offer password defense in the browser which is very good. Also i find Microsoft defender to not always detect threats post download avg nails it every time.
Screenshot 2024-06-17 025058.png
 

monkeylove

Level 13
Verified
Top Poster
Well-known
Mar 9, 2014
617
Just a single example:



There are discussions here at MT about vendors optimizing their product for the AV-Comaparatives tests. Fabian Wosar made posts here here about the problem and it does not just apply to Tencent and Qihoo.

Did you know that the signature engines on VirusTotal are not the ones that are in the consumer and enterprise products (this has always been the case)? Did you know that vendors copy each others' signatures from VirusTotal (Proven by Eugene Kaspersky and his team)?

You need to demonstrate that the ones who topped this recent real-world protection test cheated.

Compared to what was reported in the test, give us the correct results.
 

cartaphilus

Level 11
Verified
Top Poster
Well-known
Mar 17, 2023
536
I apologize if it's been already mentioned but it boggles my mind why are detections by AVAST, AVG, BullGuard, AVIRA different? They are all owned by NortonLifeLock i.e. GEN so shouldn't they all share the same signatures in order to bolster their defenses? Is the difference being made up by engine heuristic detection? So not definition detection? Thus, all use the same definitions but it's basically a test of the engines?

Since if they don't share the malware findings between each other and yet they are under the same umbrella then wow talk about silos of excellence!
 

cartaphilus

Level 11
Verified
Top Poster
Well-known
Mar 17, 2023
536
Just a single example:



There are discussions here at MT about vendors optimizing their product for the AV-Comaparatives tests. Fabian Wosar made posts here here about the problem and it does not just apply to Tencent and Qihoo.

Did you know that the signature engines on VirusTotal are not the ones that are in the consumer and enterprise products (this has always been the case)? Did you know that vendors copy each others' signatures from VirusTotal (Proven by Eugene Kaspersky and his team)?
Honestly I sure hope that vendors on VT share the detections of compared malware between each other. I thought that was the main idea behind VT? Sure it runs against all the engines but I thought the vendors were made aware of the malware they didn't detect and allowed to access that identifier? In the end same fight, same enemy.
 

kailyn

Level 2
Jun 6, 2024
85
You need to demonstrate that the ones who topped this recent real-world protection test cheated.

Compared to what was reported in the test, give us the correct results.
I never said that anybody cheated in the latest test. I said that some game the system by tuning their protections. Tuning is legal to the extent that it does not violate AV-Comparatives' rules. There is no doubt whatsoever that multiple vendors have tuned their products in every AV-Comparatives test and other lab tests in which they participate.

Please read carefully and do not make assumptions or make incorrect interpretations.

You can research it here at MT yourself. Search for the comments by Fabian Wosar about vendors gaming the system in the various AV lab tests. He is a leading, well respected security software industry expert. So you can rely upon what he says. It is a good place for you to start to be well informed.

It does not matter what you believe or what you want to hear, nobody in the know doubts that vendors game tests by tuning their protections. If your favorite product is performing well because you think the product is superior to others then good for you. You have the right to believe whatever you wish, regardless of the facts.

Most people want to know the facts so that they can make informed decisions. At least the ones who are interested in making informed decisions. If you need proof then you can research it. It only takes a few minutes.
 
  • Like
Reactions: comolokko

monkeylove

Level 13
Verified
Top Poster
Well-known
Mar 9, 2014
617
I never said that anybody cheated in the latest test. I said that some game the system by tuning their protections. Tuning is legal to the extent that it does not violate AV-Comparatives' rules. There is no doubt whatsoever that multiple vendors have tuned their products in every AV-Comparatives test and other lab tests in which they participate.

Please read carefully and do not make assumptions or make incorrect interpretations.

You can research it here at MT yourself. Search for the comments by Fabian Wosar about vendors gaming the system in the various AV lab tests. He is a leading, well respected security software industry expert. So you can rely upon what he says. It is a good place for you to start to be well informed.

It does not matter what you believe or what you want to hear, nobody in the know doubts that vendors game tests by tuning their protections. If your favorite product is performing well because you think the product is superior to others then good for you. You have the right to believe whatever you wish, regardless of the facts.

Most people want to know the facts so that they can make informed decisions. At least the ones who are interested in making informed decisions. If you need proof then you can research it. It only takes a few minutes.

Who are the "some" that you believe gamed the system in this recent test?
 
  • Like
Reactions: comolokko

kailyn

Level 2
Jun 6, 2024
85
Who are the "some" that you believe gamed the system in this recent test?
They all do while at the same time they tell consumers not to pay any attention to the tests that they participate in. See here to start:


It's rather hypocritical because every single one of the antivirus publishers who tells the consumer to "Do not pay attention to such and such test" are the ones that cry foul when they perform poorly, with some asking for a re-test after they modified their product. LOL.

From this point on do your own research. It's all here at MT. Please put some effort into researching it and learning about the things that are not revealed in published AV test lab reports. You can even ask @Adrian Ścibor about how some security software publishers game the tests, optimize the versions they submit for testing and so on.

Have a great day.
 
  • Like
Reactions: comolokko

monkeylove

Level 13
Verified
Top Poster
Well-known
Mar 9, 2014
617
They all do while at the same time they tell consumers not to pay any attention to the tests that they participate in. See here to start:


It's rather hypocritical because every single one of the antivirus publishers who tells the consumer to "Do not pay attention to such and such test" are the ones that cry foul when they perform poorly, with some asking for a re-test after they modified their product. LOL.

From this point on do your own research. It's all here at MT. Please put some effort into researching it and learning about the things that are not revealed in published AV test lab reports. You can even ask @Adrian Ścibor about how some security software publishers game the tests, optimize the versions they submit for testing and so on.

Have a great day.

From "some" it's now "all."

How did each of the participants in the current test game the system for this recent test? What evidence can you show to prove that?

How does that gaming affect the results, such that you can show to us the correct results?
 
  • Like
Reactions: comolokko

mlnevese

Level 28
Verified
Top Poster
Well-known
May 3, 2015
1,760
I apologize if it's been already mentioned but it boggles my mind why are detections by AVAST, AVG, BullGuard, AVIRA different? They are all owned by NortonLifeLock i.e. GEN so shouldn't they all share the same signatures in order to bolster their defenses? Is the difference being made up by engine heuristic detection? So not definition detection? Thus, all use the same definitions but it's basically a test of the engines?

Since if they don't share the malware findings between each other and yet they are under the same umbrella then wow talk about silos of excellence!
Avast and AVG use the same engine and have the same detection as expected. I have no idea if they are updated simultaneously but it wouldn't surprise me. The others still use their own engines so detection variations are expected even if they belong to the same company.
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top