AVLab.pl Security test on the example of 400 malicious samples in the wild (November 2022)

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
173
Hi Readers!

We have been following up on changes you have requested for the last two months, and now we have a summary of your proposals and our work. And it is! The November 2022 results.

First, please read the entire description of what is changed: Security Test On The Example Of 400 Malicious Samples In The Wild

Second, please LET US KNOW what do you think about a new statistic from the test: threat landscape: https://avlab.pl/en/recent-results/threat-landscape-in-november-2022/

Third, if you want to search for some SHA2 of sample, you can download CSV and start your work.

Fourth, the results website: Recent Results » AVLab Cybersecurity Foundation

I hope the new stats are useful from the marketing side. Starting from January 2023 we can extract more from Sysmon logs: Dwell Time, Remediation Time, Time to Detect and similar of interesting data. We'll see what we can improve for the marketing department. I'm waiting for your proposals!

Thanks a lot!

Adrian ;)
 

Attachments

  • security level test in numbers - November 2022.jpg
    security level test in numbers - November 2022.jpg
    1.8 MB · Views: 199

Like a Western!

Level 9
Verified
Well-known
Apr 6, 2016
440
I'm having trouble nowadays with tests published by companies/organization like AVLab AV-Test.
all i see is every product get a 6 for Protection by AV-Test and gets TOP Product badge. and o AVLab all i see is all products get 100%s
i mean it's good to see products get tested regularly and they keep the decent protection rate at least.
AV-C has been more helpful in my experience to see the actual difference between products.
now i'm not an expert and don't know if you can change this or not or even if this should change or not
this just how i see it, every product 100% in almost every test and every product 6 in protection and TOP Product badge for like most of them everytime. feels "not that helpful" to me.

Thanks for your work tho and for sharing Adrian
 

Azure

Level 28
Verified
Top Poster
Content Creator
Oct 23, 2014
1,712
I'm having trouble nowadays with tests published by companies/organization like AVLab AV-Test.
all i see is every product get a 6 for Protection by AV-Test and gets TOP Product badge. and o AVLab all i see is all products get 100%s
i mean it's good to see products get tested regularly and they keep the decent protection rate at least.
AV-C has been more helpful in my experience to see the actual difference between products.
now i'm not an expert and don't know if you can change this or not or even if this should change or not
this just how i see it, every product 100% in almost every test and every product 6 in protection and TOP Product badge for like most of them everytime. feels "not that helpful" to me.

Thanks for your work tho and for sharing Adrian
Completely understandable.

At the very least this kind of test could be very useful to those that want a security solution that focus more on pre-launch or post-launch
 

Andrezj

Level 6
Nov 21, 2022
248
I'm having trouble nowadays with tests published by companies/organization like AVLab AV-Test.
all i see is every product get a 6 for Protection by AV-Test and gets TOP Product badge. and o AVLab all i see is all products get 100%s
i mean it's good to see products get tested regularly and they keep the decent protection rate at least.
AV-C has been more helpful in my experience to see the actual difference between products.
now i'm not an expert and don't know if you can change this or not or even if this should change or not
this just how i see it, every product 100% in almost every test and every product 6 in protection and TOP Product badge for like most of them everytime. feels "not that helpful" to me.

Thanks for your work tho and for sharing Adrian
it is just basic av default configuration testing based upon amsto guidelines
none of the av are willing to pay for extended testing of their product with feature tweaks, hardened or maximum settings, even if they are willing then they would complain that the results are unfair because of feature differences in av products, that the tests are flawed
surely a test lab like avlab, which has done some more revealing tests in the past like script and banking trojans, has had challenge in getting the av makers to agree to those test methods or objections that test was not done properly
primarily because of lack of agreement on testing standards and methodology, this is why av tests are as they are, and there is amsto
certainly a capable lab like avlab is able and willing to do all the advanced testing that will reveal all the weaknesses and differences between av, if somebody is willing to pay for it
 

Shadowra

Level 33
Verified
Top Poster
Content Creator
Malware Tester
Well-known
Sep 2, 2021
2,245
Hello Adrian :)

I see a nice improvement in your tests, and it's nice to see a company listening to us!

But, are these fresh samples that were used? I can't find anything on them :/ (especially since I see that all the antivirus programs have blocked the 400 malwares)
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
173
I'm having trouble nowadays with tests published by companies/organization like AVLab AV-Test.
all i see is every product get a 6 for Protection by AV-Test and gets TOP Product badge. and o AVLab all i see is all products get 100%s
i mean it's good to see products get tested regularly and they keep the decent protection rate at least.
AV-C has been more helpful in my experience to see the actual difference between products.
now i'm not an expert and don't know if you can change this or not or even if this should change or not
this just how i see it, every product 100% in almost every test and every product 6 in protection and TOP Product badge for like most of them everytime. feels "not that helpful" to me.

Thanks for your work tho and for sharing Adrian
Hi @Like a Western! I understand your opinion, you don't even know how much!!
Consider the test scenario. Browser > link with malware -> User -> AV product reaction.

AV technologies in such a scenario are very effective. Believe me, malware authors don't make an effort. They go for ease over quantity, not quality.
I'll tell you more. One vendor demands malware very spread in the wild, and another fresh sample. This is contradictory.

It is not easy to satisfy the requirements of every vendor. And the user requirements are the hardest. Do you know why? Because there is a belief that there is no AV that will stop 100% of everything. Full agree! However, in tests they stop 100% of the malware set, and you have no control over it.

You need to know more that the test is followed by a consultation. The Vendor has the right to reject the malware if it proves that it was not a threat on the system or if it was a PUP/PUA.

it is just basic av default configuration testing based upon amsto guidelines
none of the av are willing to pay for extended testing of their product with feature tweaks, hardened or maximum settings, even if they are willing then they would complain that the results are unfair because of feature differences in av products, that the tests are flawed
surely a test lab like avlab, which has done some more revealing tests in the past like script and banking trojans, has had challenge in getting the av makers to agree to those test methods or objections that test was not done properly
primarily because of lack of agreement on testing standards and methodology, this is why av tests are as they are, and there is amsto
certainly a capable lab like avlab is able and willing to do all the advanced testing that will reveal all the weaknesses and differences between av, if somebody is willing to pay for it
In 2023, there will definitely be at least one test with manual attacks. It requires a lot of work, so it is not often done. In addition, it is exactly as you wrote. One Vendor is fine with it, and others are not, because these are not "in the wild" attacks. That's why we have different tests as service for vendors.

We will publish the results from the EDR-XDR test soon!

Hello Adrian :)

I see a nice improvement in your tests, and it's nice to see a company listening to us!

But, are these fresh samples that were used? I can't find anything on them :/ (especially since I see that all the antivirus programs have blocked the 400 malwares)
In the CSV file, you are given all the SHA256. You can compare them to VirusTotal, for example, but keep in mind that the engines on VT may not contain all the technology you have available in the product installed on your workstation.
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
173
Am I the only one thinking this? Why do the AV companies have any consideration in to how the testing is done or what samples are used? Is it because they pay you guys to do the tests? Its almost like they want to manipulate the results.;)
I disagree. Believe me, as a testing organization, you have contact with technical people at each vendor: CTO, CSO, Malware Analyst etc. They care about details, like MalwareTips Users, and want to learn as much as possible. I have yet to encounter a persuasion that we should change the result because they won't pay. Besides, you usually sign a contract for a whole year and both sides are the agreement. Regardless of the result, they still have to pay you.
You can agree not to publish the results. That's why sometimes in the tests the name is given: vendor A, Vendor B etc. They have feedback, but you don't show negative results.
They may suggest something, but never dictate how you should do something. That's why it suits some vendors and not others.
But this is normal. You work with people who want it. You don't talk to someone if they don't want it.
 

Andrezj

Level 6
Nov 21, 2022
248
Am I the only one thinking this? Why do the AV companies have any consideration in to how the testing is done or what samples are used? Is it because they pay you guys to do the tests? Its almost like they want to manipulate the results.;)
it is not because av company pays, it is because it is part of a fair agreement between the av company and the test lab, that agreement must be a mutual
av comparatives charges about 60,000 euros for its testing, the av companies have to agree to the test methodology - which provides a fairness plan to discuss challenges to results, mistakes or other problems that the av company states - or else the av company do not have to use av comparaitve services, av has no requirement to be tested
no av company will accept from test lab "you must agree and accept our tests and results without objection", such av lab will not get any business
there is no such thing as completely independent av testing and av company must accept results, test labs cannot force av company to accept test methods and results
av testing is directed by industry agreement conceptually, then on individual basis between av company directly with av test lab
part of industry standard is av company ability to challenge results, and usual solution is not to reveal test results
publish a test where av company has no input, av company disagrees with test lab results or considers it enough harm to their product reputation, they can file a legal case (they must prove significant financial damage), this is because lab like av comparatives is considered to carry influence on people opinions of the product and purchases of product, whereas some personal review or statement like youtube test carries little to no influence
just fyi, av companies have filed lots of lawsuits against reviews, statements and test results over the years, and none have won their case, such legal complaints are used primarily as an intimidation tactic
 
Last edited:
F

ForgottenSeer 97327

@Digmor Crusher and @Shadowra

I don't think that the AV-labs test with old samples. My guess is that meta-data is the issue. When you have programs on your PC and execute them they have a different meta and or telemetry data trail than when they are delivered through mail. downloaded from the internet or executed from an USB. When AV's become smarter, the tests have to incorporate this extra knowledge gained from meta data and telemetry data points.

Machine learning and Artificial Intelligence does not work well with wrong data points. So the same sample used by Shadowra and AV-labs could have different result simply because they were tested in a different context. All suspicious security forum members think the AV-labs tests are bogged. My guess is that they are not bogged or manipulated. The AV's tested just want to make sure their products are tested with the use-cases they are designed to protect for.
 

Andrezj

Level 6
Nov 21, 2022
248
@Digmor Crusher and @Shadowra

I don't think that the AV-labs test with old samples.
some av companies insist upon test include older malware
if it were even possible, make a true zero day malware test of all types of malware, where every sample of 10000 samples is only 1 hour old and av companies will not agree to participate in such a test because test results will eventually show many weaknesses
 

Digmor Crusher

Level 23
Verified
Top Poster
Well-known
Jan 27, 2018
1,236
I disagree. Believe me, as a testing organization, you have contact with technical people at each vendor: CTO, CSO, Malware Analyst etc. They care about details, like MalwareTips Users, and want to learn as much as possible. I have yet to encounter a persuasion that we should change the result because they won't pay. Besides, you usually sign a contract for a whole year and both sides are the agreement. Regardless of the result, they still have to pay you.
You can agree not to publish the results.That's why sometimes in the tests the name is given: vendor A, Vendor B etc. They have feedback, but you don't show negative results.
They may suggest something, but never dictate how you should do something. That's why it suits some vendors and not others.
But this is normal. You work with people who want it. You don't talk to someone if they don't want it.
Thanks Adrian. But shouldn't you show all results, negative or not? If all vendors score 100% what's the point?

it is not because av company pays, it is because it is part of a fair agreement between the av company and the test lab, that agreement must be a mutual
av comparatives charges about 60,000 euros for its testing, the av companies have to agree to the test methodology - which provides a fairness plan to discuss challenges to results, mistakes or other problems that the av company states - or else the av company do not have to use av comparaitve services, av has no requirement to be tested
no av company will accept from test lab "you must agree and accept our tests and results without objection", such av lab will not get any business
there is no such thing as completely independent av testing and av company must accept results, test labs cannot force av company to accept test methods and results
av testing is directed by industry agreement conceptually, then on individual basis between av company directly with av test lab
part of industry standard is av company ability to challenge results, and usual solution is not to reveal test results
publish a test where av company has no input, av company disagrees with test lab results or considers it enough harm to their product reputation, they can file a legal case (they must prove significant financial damage), this is because lab like av comparatives is considered to carry influence on people opinions of the product and purchases of product, whereas some personal review or statement like youtube test carries little to no influence
just fyi, av companies have filed lots of lawsuits against reviews, statements and test results over the years, and none have won their case, such legal complaints are used primarily as an intimidation tactic
There should be independent testing otherwise results could be skewed.

some av companies insist upon test include older malware
if it were even possible, make a true zero day malware test of all types of malware, where every sample of 10000 samples is only 1 hour old and av companies will not agree to participate in such a test because test results will eventually show many weaknesses
Isn't that the point of testing???
 
F

ForgottenSeer 97327

some av companies insist upon test include older malware
if it were even possible, make a true zero day malware test of all types of malware, where every sample of 10000 samples is only 1 hour old and av companies will not agree to participate in such a test because test results will eventually show many weaknesses
You wont find 10000 samples of 1 hour. It is hard to find around 250 new samples in a month.

I don't understand the logic of zero day samples floating around he internet which are known by unpaid/amateur security specialists and not know by the security specialist paid by AV-companies and the independent 'post intrusion' specialist (who support governments and companies after breeches for big money).

For me distrusting AV-labs and independant (post intrusion) specialist are as half-true and half wrong as the conspiracy theories of QAnon. umvolkung or great reset. They are based on small factual pieces of information, taken out of context resulting in logic like the moon is round and yellow and has holes in it, so it must be made of Gouda cheese.
 

Andrezj

Level 6
Nov 21, 2022
248
There should be independent testing otherwise results could be skewed.
who would compel such testing? there is no standard of security software performance to which any government, agency, or law making body can force an av company to comply
even if a government tried, they can only come up with a bare minimum set of guidelines - which is what the av industry has already agreed upon and participates in av testing
even with drugs and automobiles, things that are regulated for life and safety reasons, the manufacturers and industry organizations have lots of influence in shaping the regulations and testing they must comply with
Isn't that the point of testing???
from your point of view, but for the av company it is just to prove that their product meets a protection standard that is agreed upon by them, which is based upon industry standards of testing such as amsto
from the av company point of view, they particiapte in testing for marketing purposes, not to reveal all possible weaknesses
distrusting AV-labs
av test labs do their best efforts and work very hard to conduct reliable trustworthy testing, they try their best to detect av company that tries to optimize their product to perform will in tests (gaming the test), the labs refuse unreasonable demands by av companies, and some test labs just will not do business with certain av companies because of disagreements
lots of people believe or claim that av test labs alter results based upon av company paying them, that is not true for the reputable labs, every av test agreement that i have seen gives the av company the right to not publish results should they believe there is a problem with the test, this is a standard practice for most product testing
 

Digmor Crusher

Level 23
Verified
Top Poster
Well-known
Jan 27, 2018
1,236
There should be an independent group with independent funding and let them test any way they want. Then maybe we would see real results and not every AV testing within 2% of each other. If we already know the test results beforehand then what's the use of testing?
 

Andrezj

Level 6
Nov 21, 2022
248
There should be an independent group with independent funding and let them test any way they want.
government, academic and independent researchers who pentest, vuln hunt are independent, they can test however they want, they can report as they want
though few researchers pentest for av feature design weaknesses because those are already known or av bug bounty does not pay well
independent researchers do it for profit, seeking the remote code vulns and then selling them for six figures, few of those researchers care about av, they chase vulnerabilities in the software that is installed across the globe
the problem is the consumer must search for, study and understand the results
though not much help because the av company can decide not to fix and not publish that fact, ask @Adrian Ścibor av companies do not fix problems found in avlab tests
usa nsa, israeli unit 8200, uk gchq, and similar hack av to oblivion, they know all the weaknesses and will not reveal
 

simmerskool

Level 31
Verified
Top Poster
Well-known
Apr 16, 2017
2,094
There should be an independent group with independent funding and let them test any way they want. Then maybe we would see real results and not every AV testing within 2% of each other. If we already know the test results beforehand then what's the use of testing?
I'm guessing that we do not see truly independent testing because although it could be done, as soon as you publish a negative finding, you open up yourself to being sued. To get around the potential litigation problem, av vendors and testing orgs enter into the mutual testing agreements discussed above, and yes the av vendors participate for marketing.
 

Zero Knowledge

Level 20
Verified
Top Poster
Content Creator
Dec 2, 2016
841
All products score 100%? That doesn't sound correct. There is no way even Kaspersky, ESET or Emsisoft score 100% against true zero-day malware.
My guess is that they are not bogged or manipulated.
Agreed, I don't think the test results are fake or wrong, but they can be influenced and corrupted by a number of factors.
Its almost like they want to manipulate the results.;)
Bingo! That's the problem, where there is money aka fees for testing, companies will always try to influence the outcome to be positive. If you know you would fail a test, why would you participate? You wouldn't. It would be bad publicity and sink your business.
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
173
some av companies insist upon test include older malware
if it were even possible, make a true zero day malware test of all types of malware, where every sample of 10000 samples is only 1 hour old and av companies will not agree to participate in such a test because test results will eventually show many weaknesses
Point out such a source of malware.
I believe there is no such thing. What is zero-day from the user's point of view? Something that hit him the first time?

Consider, for example, APP.ANY.RUN. They have a great deal of files from users. How do you investigate whether they are zero-day? For some sample will be zero-day, for other vendors not.

You won't get a malware test base of many thousands in one month. Even paid malware services don't have such databases. I know, because I checked. I asked. I did the reconnaissance to even pay for it. They don't have such databases.

From our point of view, it is not the base that is important, but the REAL URLs that live a few minutes, a few hours max, and the malware is 404.

Another difficulty is that not every unknown file is malware. This is where Vendors may have the most doubt, as you have investigated it, show and proof that it is harmful!

Thanks Adrian. But shouldn't you show all results, negative or not? If all vendors score 100% what's the point?


There should be independent testing otherwise results could be skewed.


Isn't that the point of testing???
I meant in theory. You can find it, for example, in AV-C tests.
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top