AV-Comparatives AV-Comparatives - Real World Protection Test Feb-May 2024

[Trident]

Did you know that the signature engines on VirusTotal are not the ones that are in the consumer and enterprise products (this has always been the case)? Did you know that vendors copy each others' signatures from VirusTotal (Proven by Eugene Kaspersky and his team)?

The signatures of each vendor on VirusTotal don't necesarily have to be what's deployed on a wide-scale production environments.
VirusTotal is the ideal playground to test detection methods before they are pushed to customers.
This is not a secret to anyone and is made clear on VirusTotal.

Copying signatures is impossible as most (not all, for example I've managed to view the database of some Indian AVs like NetProtect) are encrypted and current computing methods will be unable to break the encryption, for a company to view the signatures. Whether or not memory dumping can help extract them, I've not tested and can't say. The tops names would definitely implement proper anti-debug logics.

It is possible for malware analysts and even automated systems to view detections on VirusTotal and copy the detections names, which are in no way protected by the law (not considered proprietary intelligence). Also, many vendors use third-party feeds, which is where similarities may have been derived from. But what they copy is the name, not the signature itself.

 

[kailyn]

The signatures of each vendor on VirusTotal don't necesarily have to be what's deployed on a wide-scale production environments.
VirusTotal is the ideal playground to test detection methods before they are pushed to customers.
This is not a secret to anyone and is made clear on VirusTotal.

Theoretically the use of VirusTotal to "experiment" might be possible but unlikely. The primary reason that the signature engines are modified as they are is because of VirusTotal's integration requirements.

Copying signatures is impossible as most (not all, for example I've managed to view the database of some Indian AVs like NetProtect) are encrypted and current computing methods will be unable to break the encryption, for a company to view the signatures. Whether or not memory dumping can help extract them, I've not tested and can't say. The tops names would definitely implement proper anti-debug logics.

It is possible for malware analysts and even automated systems to view detections on VirusTotal and copy the detections names, which are in no way protected by the law (not considered proprietary intelligence). Also, many vendors use third-party feeds, which is where similarities may have been derived from. But what they copy is the name, not the signature itself.

Vendors do not literally copy one company's signature for a file and include it in their signature database. They copy each others' detections and then create their own signatures. It was proven by Eugene Kaspersky and his team.

Nobody said that they are literally copying each others' signatures. That is not what is meant within the context of the discussion. "Copying signatures" means they all copy each others detections. That is what Eugene Kaspersky meant when he said "copying signatures." There is no question that AV vendors do it. It is not illegal but it does have an element of theft of another company's work product. Eugene Kaspersky and other AV owners think it is underhanded and wrong because they invested heavily in created strong detection algorithms and methodologies, and other companies copy their detections. That seems entirely a legit, reasonable gripe.

 

[Trident]

Theoretically the use of VirusTotal to "experiment" might be possible but unlikely. The primary reason that the signature engines are modified as they are is because of VirusTotal's integration requirements.

The requirements of VirusTotal include usage of command line scanner which is what VT uses. The rest is all at the vendors discretion. A lot of them deploy experimental machine learning models, not yet tested for false positives and other heuristics. Some of them produce different names, for example the Symantec Heur.AdvML.A,B,C,D in products, is renamed Malware.AI.Low confidence/high confidence, etc.
Bitdefender for years has deployed an engine on VT that is nowhere to be found in official products.

Nobody said that they are literally copying each others' signatures. That is not what is meant within the context of the discussion. "Copying signatures" means they all copy each others detections. That is what Eugene Kaspersky meant when he said "copying signatures." There is no question that AV vendors do it. It is not illegal but it does have an element of theft of another company's work product. Eugene Kaspersky and other AV owners think it is underhanded and wrong because they invested heavily in created strong detection algorithms and methodologies, and other companies copy their detections. That seems entirely a legit, reasonable gripe.

Oh I understand now what you mean.

Well it may be possible for some action to be taken in this case, depending on how severe it is.
But Kaspersky is mostly based on heuristics where every single one will identify thousands of mutations. Vendors-thiefs will just create some hash-based detection that (if fuzzy hashing is used) may last few variants.
This won’t help them protect users.

 

[nickstar1]

let's also not forget Avast/AVG does offer password defense in the browser which is very good. Also i find Microsoft defender to not always detect threats post download avg nails it every time.

 

[monkeylove]

Just a single example:

Anti-virus product caught cheating by independent test agency

AV-Comparatives, one of the world’s leading independent testers of anti-virus products, says that it has uncovered that at least one product isn’t playing by the rules.

grahamcluley.com

Tencent, Qihoo antimalware firms are accused of cheating, stripped of rankings in antivirus tests

Several independent testing firms claim the accused companies submitted programs specially tuned to ace the benchmarks.

www.pcworld.com

There are discussions here at MT about vendors optimizing their product for the AV-Comaparatives tests. Fabian Wosar made posts here here about the problem and it does not just apply to Tencent and Qihoo.

Did you know that the signature engines on VirusTotal are not the ones that are in the consumer and enterprise products (this has always been the case)? Did you know that vendors copy each others' signatures from VirusTotal (Proven by Eugene Kaspersky and his team)?

You need to demonstrate that the ones who topped this recent real-world protection test cheated.

Compared to what was reported in the test, give us the correct results.

 

[cartaphilus]

I apologize if it's been already mentioned but it boggles my mind why are detections by AVAST, AVG, BullGuard, AVIRA different? They are all owned by NortonLifeLock i.e. GEN so shouldn't they all share the same signatures in order to bolster their defenses? Is the difference being made up by engine heuristic detection? So not definition detection? Thus, all use the same definitions but it's basically a test of the engines?

Since if they don't share the malware findings between each other and yet they are under the same umbrella then wow talk about silos of excellence!

 

[cartaphilus]

Just a single example:

Anti-virus product caught cheating by independent test agency

AV-Comparatives, one of the world’s leading independent testers of anti-virus products, says that it has uncovered that at least one product isn’t playing by the rules.

grahamcluley.com

Tencent, Qihoo antimalware firms are accused of cheating, stripped of rankings in antivirus tests

Several independent testing firms claim the accused companies submitted programs specially tuned to ace the benchmarks.

www.pcworld.com

There are discussions here at MT about vendors optimizing their product for the AV-Comaparatives tests. Fabian Wosar made posts here here about the problem and it does not just apply to Tencent and Qihoo.

Did you know that the signature engines on VirusTotal are not the ones that are in the consumer and enterprise products (this has always been the case)? Did you know that vendors copy each others' signatures from VirusTotal (Proven by Eugene Kaspersky and his team)?

Honestly I sure hope that vendors on VT share the detections of compared malware between each other. I thought that was the main idea behind VT? Sure it runs against all the engines but I thought the vendors were made aware of the malware they didn't detect and allowed to access that identifier? In the end same fight, same enemy.

 

[kailyn]

You need to demonstrate that the ones who topped this recent real-world protection test cheated.

Compared to what was reported in the test, give us the correct results.

I never said that anybody cheated in the latest test. I said that some game the system by tuning their protections. Tuning is legal to the extent that it does not violate AV-Comparatives' rules. There is no doubt whatsoever that multiple vendors have tuned their products in every AV-Comparatives test and other lab tests in which they participate.

Please read carefully and do not make assumptions or make incorrect interpretations.

You can research it here at MT yourself. Search for the comments by Fabian Wosar about vendors gaming the system in the various AV lab tests. He is a leading, well respected security software industry expert. So you can rely upon what he says. It is a good place for you to start to be well informed.

It does not matter what you believe or what you want to hear, nobody in the know doubts that vendors game tests by tuning their protections. If your favorite product is performing well because you think the product is superior to others then good for you. You have the right to believe whatever you wish, regardless of the facts.

Most people want to know the facts so that they can make informed decisions. At least the ones who are interested in making informed decisions. If you need proof then you can research it. It only takes a few minutes.

 

[monkeylove]

I never said that anybody cheated in the latest test. I said that some game the system by tuning their protections. Tuning is legal to the extent that it does not violate AV-Comparatives' rules. There is no doubt whatsoever that multiple vendors have tuned their products in every AV-Comparatives test and other lab tests in which they participate.

Please read carefully and do not make assumptions or make incorrect interpretations.

You can research it here at MT yourself. Search for the comments by Fabian Wosar about vendors gaming the system in the various AV lab tests. He is a leading, well respected security software industry expert. So you can rely upon what he says. It is a good place for you to start to be well informed.

It does not matter what you believe or what you want to hear, nobody in the know doubts that vendors game tests by tuning their protections. If your favorite product is performing well because you think the product is superior to others then good for you. You have the right to believe whatever you wish, regardless of the facts.

Most people want to know the facts so that they can make informed decisions. At least the ones who are interested in making informed decisions. If you need proof then you can research it. It only takes a few minutes.

Who are the "some" that you believe gamed the system in this recent test?

 

[kailyn]

Who are the "some" that you believe gamed the system in this recent test?

They all do while at the same time they tell consumers not to pay any attention to the tests that they participate in. See here to start:

Antivirus companies opinions on testing labs

In the past few days, i have personally contacted popular security companies, faking to be a worried user of their product, claiming their software performed low on independent test labs. This was intended to force techinal support give their opinion on these kind of labs: should a regular user...

malwaretips.com

It's rather hypocritical because every single one of the antivirus publishers who tells the consumer to "Do not pay attention to such and such test" are the ones that cry foul when they perform poorly, with some asking for a re-test after they modified their product. LOL.

From this point on do your own research. It's all here at MT. Please put some effort into researching it and learning about the things that are not revealed in published AV test lab reports. You can even ask @Adrian Ścibor about how some security software publishers game the tests, optimize the versions they submit for testing and so on.

Have a great day.

 

[monkeylove]

They all do while at the same time they tell consumers not to pay any attention to the tests that they participate in. See here to start:

Antivirus companies opinions on testing labs

In the past few days, i have personally contacted popular security companies, faking to be a worried user of their product, claiming their software performed low on independent test labs. This was intended to force techinal support give their opinion on these kind of labs: should a regular user...

malwaretips.com

It's rather hypocritical because every single one of the antivirus publishers who tells the consumer to "Do not pay attention to such and such test" are the ones that cry foul when they perform poorly, with some asking for a re-test after they modified their product. LOL.

From this point on do your own research. It's all here at MT. Please put some effort into researching it and learning about the things that are not revealed in published AV test lab reports. You can even ask @Adrian Ścibor about how some security software publishers game the tests, optimize the versions they submit for testing and so on.

Have a great day.

From "some" it's now "all."

How did each of the participants in the current test game the system for this recent test? What evidence can you show to prove that?

How does that gaming affect the results, such that you can show to us the correct results?

 

[Behold Eck]

"Useless" Panda Free performed pretty well.

Was it Panda Free or one of the paid versions and if so what modules are giving them a good score ?

Regards Eck

 

[mlnevese]

I apologize if it's been already mentioned but it boggles my mind why are detections by AVAST, AVG, BullGuard, AVIRA different? They are all owned by NortonLifeLock i.e. GEN so shouldn't they all share the same signatures in order to bolster their defenses? Is the difference being made up by engine heuristic detection? So not definition detection? Thus, all use the same definitions but it's basically a test of the engines?

Since if they don't share the malware findings between each other and yet they are under the same umbrella then wow talk about silos of excellence!

Avast and AVG use the same engine and have the same detection as expected. I have no idea if they are updated simultaneously but it wouldn't surprise me. The others still use their own engines so detection variations are expected even if they belong to the same company.