Reliable AV Testing Labs according to AV-comp survey

Status
Not open for further replies.

Tiamati

Level 12
Thread author
Verified
Top Poster
Well-known
Nov 8, 2016
574
Last av comparatives survey showed what are the most trusted testing labs.
As i don't know most of them, I'd like to know your opinion and discuss about the results.




1580531700970.png

Source: https://www.av-comparatives.org/wp-content/uploads/2020/01/Security-Survey-2020_en.pdf
 
Last edited:

Outpost

Level 5
Verified
Well-known
Jan 11, 2020
220
I don't read and I don't trust testing labs. Everyone, no one excluded, even recently, has shown that he is making mistakes and that he is not impartial. They are not to be considered as the gospel.
I find MT Hub much more precise and above all impartial.
My vote goes to them, even if not listed. ;)
 
Last edited:

mlnevese

Level 26
Verified
Top Poster
Well-known
May 3, 2015
1,531
I don't read and I don't trust testing labs. Everyone, no one excluded, even recently, has shown that he is making mistakes and that he is not impartial. They are not to be considered as the gospel.
I find MT Hub much more precise and above all impartial.
My vote goes to them, even if not listed. ;)

Impartial, yes, more precise? I doubt that. Malwarehub uses samples already downloaded to the computer for tests bypassing many defense mechanism of the software tested. It's good to check heuristics and signature capabilities but it's not a precise test.
 

DDE_Server

Level 22
Verified
Top Poster
Well-known
Sep 5, 2017
1,168
Impartial, yes, more precise? I doubt that. Malwarehub uses samples already downloaded to the computer for tests bypassing many defense mechanism of the software tested. It's good to check heuristics and signature capabilities but it's not a precise test.
You may be right but the variety of tester for the same vendor product give you an indication about the strength and weakness of each one as the methodology of test may differ little between the tester for the same product (type of the sample ,environment of the test ,etc)
I find that quite useful for testing 😉😉
 
F

ForgottenSeer 823865

Impartial, yes, more precise? I doubt that. Malwarehub uses samples already downloaded to the computer for tests bypassing many defense mechanism of the software tested. It's good to check heuristics and signature capabilities but it's not a precise test.
+1

For the malware hub to be considered "accurate" , samples must be hosted in a server/crafted websites, then downloaded via the browser or received by email. Then if the malware manage to pass the browser security and SmartScreen, and the AV catch/miss it , then it will be a first step to be considered an accurate test.
If not, the malware hub is even less valid than test labs' "real world scenarios" , an being just a signature cross-checking not better than youtesters.
 

silversurfer

Level 85
Verified
Honorary Member
Top Poster
Content Creator
Malware Hunter
Well-known
Aug 17, 2014
10,055
For the malware hub to be considered "accurate" , samples must be hosted in a server/crafted websites, then downloaded via the browser or received by email. Then if the malware manage to pass the browser security and SmartScreen, and the AV catch/miss it , then it will be a first step to be considered an accurate test.
If not, the malware hub is even less valid than test labs' "real world scenarios" , an being just a signature cross-checking not better than youtesters.
Malware-Hub testing shows additionally how strong are dynamic modules like BB, HIPS, HEUR, it isn't only signature based testing, that must be said to avoid wrong things, but you probably don't like to agree as always for similar discussions about Malware-Hub...

Personally, I would say that all mentioned tests are NOT really accurate to reflect "real world" how people getting infected, so it isn't wrong at all!
 
F

ForgottenSeer 823865

Malware-Hub testing shows additionally how strong are dynamic modules like BB, HIPS, HEUR, it isn't only signature based testing, that must be said to avoid wrong things, but you probably don't like to agree as always for similar discussions about Malware-Hub...
Modules don't matter, if the malware magically appears on the desktop, which is the real flaw of malwarehub & youtester tests.
There is no such thing as a malware appearing in the desktop out of nowhere. You have to consider the attack vectors:

1- Downloaded/executed from the browser.
2- removable devices.
3- Emails attachments.
4- Networks shares.
5- legit installers compromised.
6- remote shells

For a test be accurate, the malware must be imported via one of those vectors while the product is active, and if so, it must be mentioned.
Not saying, i dont even consider a test made in a VM accurate, use a dedicated machine.

Anything else, whatever you say, can't be labelled as accurate (even if in some cases it can be close), it is just toying.

Personally, I would say that all mentioned tests are NOT really accurate to reflect "real world" how people getting infected, so it isn't wrong at all!
You know my point of view about test, especially those "independent & professional" labs...
 

silversurfer

Level 85
Verified
Honorary Member
Top Poster
Content Creator
Malware Hunter
Well-known
Aug 17, 2014
10,055
For a test be accurate, the malware must be imported via one of those vectors while the product is active, and if so, it must be mentioned.
Not saying, i dont even consider a test made in a VM accurate, use a dedicated machine.

Anything else, whatever you say, can't be labelled as accurate (even if in some cases it can be close), it is just toying.
Agree, I meant about Malware-Hub, it isn't accurate and can't be ever really accurate!
I wrote "all tests are NOT really accurate" and meant official tests like AV-Tests, AV-Comparatives, ect. and even more private tests by YouTube-Testers and malware testing in forums like here.

BUT Malware-Hub testing it's never useless to everyone, here are different opinions some like it and others just dislike, but why we need to discuss the same points again and again for ever, the truth is that discussing always the same topic is really useless...
 
F

ForgottenSeer 823865

Agree, I meant about Malware-Hub, it isn't accurate and can't be ever really accurate!
I wrote "all tests are NOT really accurate" and meant official tests like AV-Tests, AV-Comparatives, ect. and even more private tests by YouTube-Testers and malware testing in forums like here.

BUT Malware-Hub testing it's never useless to everyone, here are different opinions some like it and others just dislike, but why we need to discuss the same points again and again for ever, the truth is that discussing always the same topic is really useless...
First, i never said MH was BS or useless.
About the points you mentioned, you and me, since we are (were) involved with it and aware of its methodology, it is obvious; but look all at those members that consider MH almost as the Graal of testing,
I don't reject the hardwork MH's testers do, but things must be put at their right place. With MH limited resources and skills, you can't obviously attain the level of accuracy of some labs. MH tests are just better than the average Youtesters ones, no more. It is this aspect i am talking about.
The good points about MH is that you know what samples are used, which allow cross-checking from the audience and give some impartiality compared to Labs.
 

Outpost

Level 5
Verified
Well-known
Jan 11, 2020
220
Malware-Hub testing shows additionally how strong are dynamic modules like BB, HIPS, HEUR, it isn't only signature based testing, that must be said to avoid wrong things, but you probably don't like to agree as always for similar discussions about Malware-Hub...

Personally, I would say that all mentioned tests are NOT really accurate to reflect "real world" how people getting infected, so it isn't wrong at all!

I totally agree.
 

Burrito

Level 24
Verified
Top Poster
Well-known
May 16, 2018
1,363
SE Labs currently my favourite to look at.

Yeah.

SE Labs is willing to 'let losers be losers.'

They are willing to show the real ugly reality of some capabilities.

Some other labs try to have 'the losers' still finish above 90% --- as they are trying very hard not to offend any vendors --- i.e. paying customers.

For example... Letting losers be losers..

1580572198204.png



vs. 'Everybody gets a trophy' results --- Everybody finishes above 98% in this case.

1580572439137.png



Contrary to what some here seem to think.... the tests are generally fairly administered, not fixed, and generally represent products fairly.

My critique above simply indicates that sometimes the testers have intent to 'shame no product' that could be a future paying client. So the test is fair, it's just designed by the difficulty of the samples such that everybody does well. Because... for many of the tests that we see, the participants all paid to be there.
 
Last edited:

Outpost

Level 5
Verified
Well-known
Jan 11, 2020
220
Really ? you didn't read those "i dont trust labs, but MH" kind of stuff?

Ha ha LOL.

Can you quote exactly the point where I say MT Hub is the Grail?
You won't find him. And the rest is just your conclusion. Your interpretation. So don't put words and terms in my mouth that I haven't used.
 
Status
Not open for further replies.

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top