Status
Not open for further replies.

Outpost

Level 5
Verified
I don't read and I don't trust testing labs. Everyone, no one excluded, even recently, has shown that he is making mistakes and that he is not impartial. They are not to be considered as the gospel.
I find MT Hub much more precise and above all impartial.
My vote goes to them, even if not listed. ;)
 
Last edited:

mlnevese

Level 21
Verified
I don't read and I don't trust testing labs. Everyone, no one excluded, even recently, has shown that he is making mistakes and that he is not impartial. They are not to be considered as the gospel.
I find MT Hub much more precise and above all impartial.
My vote goes to them, even if not listed. ;)
Impartial, yes, more precise? I doubt that. Malwarehub uses samples already downloaded to the computer for tests bypassing many defense mechanism of the software tested. It's good to check heuristics and signature capabilities but it's not a precise test.
 

DDE_Server

Level 21
Verified
Impartial, yes, more precise? I doubt that. Malwarehub uses samples already downloaded to the computer for tests bypassing many defense mechanism of the software tested. It's good to check heuristics and signature capabilities but it's not a precise test.
You may be right but the variety of tester for the same vendor product give you an indication about the strength and weakness of each one as the methodology of test may differ little between the tester for the same product (type of the sample ,environment of the test ,etc)
I find that quite useful for testing 😉😉
 
F

ForgottenSeer 823865

Impartial, yes, more precise? I doubt that. Malwarehub uses samples already downloaded to the computer for tests bypassing many defense mechanism of the software tested. It's good to check heuristics and signature capabilities but it's not a precise test.
+1

For the malware hub to be considered "accurate" , samples must be hosted in a server/crafted websites, then downloaded via the browser or received by email. Then if the malware manage to pass the browser security and SmartScreen, and the AV catch/miss it , then it will be a first step to be considered an accurate test.
If not, the malware hub is even less valid than test labs' "real world scenarios" , an being just a signature cross-checking not better than youtesters.
 

silversurfer

Level 63
Verified
Trusted
Content Creator
Malware Hunter
For the malware hub to be considered "accurate" , samples must be hosted in a server/crafted websites, then downloaded via the browser or received by email. Then if the malware manage to pass the browser security and SmartScreen, and the AV catch/miss it , then it will be a first step to be considered an accurate test.
If not, the malware hub is even less valid than test labs' "real world scenarios" , an being just a signature cross-checking not better than youtesters.
Malware-Hub testing shows additionally how strong are dynamic modules like BB, HIPS, HEUR, it isn't only signature based testing, that must be said to avoid wrong things, but you probably don't like to agree as always for similar discussions about Malware-Hub...

Personally, I would say that all mentioned tests are NOT really accurate to reflect "real world" how people getting infected, so it isn't wrong at all!
 
F

ForgottenSeer 823865

Malware-Hub testing shows additionally how strong are dynamic modules like BB, HIPS, HEUR, it isn't only signature based testing, that must be said to avoid wrong things, but you probably don't like to agree as always for similar discussions about Malware-Hub...
Modules don't matter, if the malware magically appears on the desktop, which is the real flaw of malwarehub & youtester tests.
There is no such thing as a malware appearing in the desktop out of nowhere. You have to consider the attack vectors:

1- Downloaded/executed from the browser.
2- removable devices.
3- Emails attachments.
4- Networks shares.
5- legit installers compromised.
6- remote shells

For a test be accurate, the malware must be imported via one of those vectors while the product is active, and if so, it must be mentioned.
Not saying, i dont even consider a test made in a VM accurate, use a dedicated machine.

Anything else, whatever you say, can't be labelled as accurate (even if in some cases it can be close), it is just toying.

Personally, I would say that all mentioned tests are NOT really accurate to reflect "real world" how people getting infected, so it isn't wrong at all!
You know my point of view about test, especially those "independent & professional" labs...
 

silversurfer

Level 63
Verified
Trusted
Content Creator
Malware Hunter
For a test be accurate, the malware must be imported via one of those vectors while the product is active, and if so, it must be mentioned.
Not saying, i dont even consider a test made in a VM accurate, use a dedicated machine.

Anything else, whatever you say, can't be labelled as accurate (even if in some cases it can be close), it is just toying.
Agree, I meant about Malware-Hub, it isn't accurate and can't be ever really accurate!
I wrote "all tests are NOT really accurate" and meant official tests like AV-Tests, AV-Comparatives, ect. and even more private tests by YouTube-Testers and malware testing in forums like here.

BUT Malware-Hub testing it's never useless to everyone, here are different opinions some like it and others just dislike, but why we need to discuss the same points again and again for ever, the truth is that discussing always the same topic is really useless...
 
F

ForgottenSeer 823865

Agree, I meant about Malware-Hub, it isn't accurate and can't be ever really accurate!
I wrote "all tests are NOT really accurate" and meant official tests like AV-Tests, AV-Comparatives, ect. and even more private tests by YouTube-Testers and malware testing in forums like here.

BUT Malware-Hub testing it's never useless to everyone, here are different opinions some like it and others just dislike, but why we need to discuss the same points again and again for ever, the truth is that discussing always the same topic is really useless...
First, i never said MH was BS or useless.
About the points you mentioned, you and me, since we are (were) involved with it and aware of its methodology, it is obvious; but look all at those members that consider MH almost as the Graal of testing,
I don't reject the hardwork MH's testers do, but things must be put at their right place. With MH limited resources and skills, you can't obviously attain the level of accuracy of some labs. MH tests are just better than the average Youtesters ones, no more. It is this aspect i am talking about.
The good points about MH is that you know what samples are used, which allow cross-checking from the audience and give some impartiality compared to Labs.
 

Outpost

Level 5
Verified
Malware-Hub testing shows additionally how strong are dynamic modules like BB, HIPS, HEUR, it isn't only signature based testing, that must be said to avoid wrong things, but you probably don't like to agree as always for similar discussions about Malware-Hub...

Personally, I would say that all mentioned tests are NOT really accurate to reflect "real world" how people getting infected, so it isn't wrong at all!
I totally agree.
 

Burrito

Level 23
SE Labs currently my favourite to look at.
Yeah.

SE Labs is willing to 'let losers be losers.'

They are willing to show the real ugly reality of some capabilities.

Some other labs try to have 'the losers' still finish above 90% --- as they are trying very hard not to offend any vendors --- i.e. paying customers.

For example... Letting losers be losers..

1580572198204.png



vs. 'Everybody gets a trophy' results --- Everybody finishes above 98% in this case.

1580572439137.png



Contrary to what some here seem to think.... the tests are generally fairly administered, not fixed, and generally represent products fairly.

My critique above simply indicates that sometimes the testers have intent to 'shame no product' that could be a future paying client. So the test is fair, it's just designed by the difficulty of the samples such that everybody does well. Because... for many of the tests that we see, the participants all paid to be there.
 
Last edited:
Status
Not open for further replies.
Top