Orion

Level 2
And i have serious doubts about their methodology...
You are not alone.I had posted a link to the malware protection test topic which you can go back and read.I tried bringing the points close to real world as much as possible.Let me know your thoughts.
 
  • Like
Reactions: AtlBo
D

Deleted member 178

Quoting your post over there:

1)Are these samples the same ones used in real world test where you pasted the download link in the browser? If yes then why use a sample whose URL is already blocked.
Would be pointless indeed...

2)Has AV-C filtered out the samples because there is a very less amount that has the ability to spread by USB.You guys should be using malicious emails now because they are the biggest way of spreading ransom malware.
I think they just focus on "on-access" detection rather than "on-execution."

3)There is no clear statement of how they consider the sample as bypassed? Eg: If AV blocks and detects the dropped/downloaded binary the system is still protected even if the dropper runs or downloads this is usually applicable for ransomware which spreads from JS downloaders coming from a E-mail client.
Yes and i guess they focus on the droppers only.

4) Don't merge the results with real world graphs even if you plan to do so... the overlap is quite significant and it's more confusing.Making different graphs will allow us to see which AV is good at both or is inevitably getting good results at real world because of it's aggressive URL blocking but doesn't detect too much PE.
Agree

5)Were the samples downloaded on the system in presence of a disabled AV or what? How was it brought onto the system?
I raised the questions on Windows 8/10 because of smartscreen , most of them would be blocked by it , so i guess they disable all native protection, not even sure they use SUA.

6) If you want to perform this type of a test which is great in my view please do it with threats that come via email and USB specifically .Don't just ship a ton of binaries onto a stick or a system without knowing whether the samples really do spread via a USB or not.
Agree, but again what are their priority ? detection or execution?
 

Orion

Level 2
"The Malware Protection Test is an enhancement of the File Detection Test which we performed in previous years. It assesses a security program’s ability to protect a system against infection by malicious files; what is unique about this test is that in addition to checking detection in scans, it additionally assesses each program’s last line of defence. Any samples that have not been detected e.g. on-access are executed on the test system, with Internet/cloud access available, to allow features such as behavioural protection to come into play."

On access seems to be more signature oriented which is the wrong way if you have test the entire product.
 

Orion

Level 2
So the malware protection test is far more informative than the others.
Or so it would seem.Still there is little of no information of how the samples were bought onto the system and under which conditions.No information on the relation of the samples as well.Detected dropped binary is still a hit in my book.

there is little of no info on how the testbeds are created. All these 99.1% and such scores are not face value. The overlap of the product's detections is not as great as clementi/marx tests suggest.
 
  • Like
Reactions: AtlBo
D

Deleted member 178

And they don't even mention the malware; so their test can't be verified. As i said many times, they play the game in favor of vendors because 98.1% is still better than 0% by using real FUD malwares...
 

Orion

Level 2
And they don't even mention the malware; so their test can't be verified. As i said many times, they play the game in favor of vendors because 98.1% is still better than 0% by using real FUD malwares...
I think it's really tough for them to get brand new ones unless they are looking at live feeds from online sandboxes.

And what if the AV companies already got these samples and wrote signatures and AV-c added them later.We are back in testbed construction issues.
 

kamo_jisan

Level 1
Good morning All! And I'm very sorry for my reply, which is too delayed.
I had not be able to post reply to this theread for a few weeks.

By the way, I got many opinions at here.
Some people are saying that Antimalware solution tests are not fare for all solutions due to the business,
and other side people are saying that this awareness is not correct and some tests(e.g. AV-Comparatives...) are trusty.
I can not leach a conclusion that which awareness is correct,
but I got a conclusion that I should not refer antimalware solution tests too far, and some of these tests part are trusty.

I will make all answers to be useful for me to make secure PC. Thanks you!
 
  • Like
Reactions: AtlBo

mekelek

Level 28
Good morning All! And I'm very sorry for my reply, which is too delayed.
I had not be able to post reply to this theread for a few weeks.

By the way, I got many opinions at here.
Some people are saying that Antimalware solution tests are not fare for all solutions due to the business,
and other side people are saying that this awareness is not correct and some tests(e.g. AV-Comparatives...) are trusty.
I can not leach a conclusion that which awareness is correct,
but I got a conclusion that I should not refer antimalware solution tests too far, and some of these tests part are trusty.

I will make all answers to be useful for me to make secure PC. Thanks you!
the best reviews you can get are the ones coming straight from the users.
check the threads here, check the videos the users here made testing the products, check the malware samples section, etc.
 

ravi prakash saini

Level 13
Verified
@kamo_jisan to confuse you further l have subscription for Internet security from Kaspersky,emsisoft,avast and bitdefender ,however persently I am using comodo internet security and sometimes I do not use any antivirus from any company and still no infection
the first line of defence consists of smart screen+UAC+backup+control on mouse
for second line of defence you can choose any antivirus.
but remember antivirus can save you in one condition only and that is if you can keep your computer free of infections without using any security software
more confuse,you are free to curse me
 

Game Of Thrones

Level 5
Verified
the best reviews you can get are the ones coming straight from the users.
check the threads here, check the videos the users here made testing the products, check the malware samples section, etc.
Wrong. Sometimes forums are toward a specific product. From samples to some users, so i would suggest a trusty lab results not a user result
 

Fritz

Level 11
Well said @ravi prakash saini !

Also, adding to the unknown monetary influences, these test always just show a certain reaction to malware at a certain point in time. Add a week and it's a different game.

Throw a coin, getting 5 times heads doesn't mean the next 5 throws will yield 5 times tails. It's still 50/50 each throw.

That said, test are a nice indication where a solution is generally headed. After that, I check forums and test them myself. Some may be buggy for me, like Bitdefender, some may not show the reported behavior (e.g. high false positives with F-Secure, never happened to me).

So read the test results, add a nice whopping of Malwartips and put your personal cherry on top. Enjoy. :)
 
Wrong. Sometimes forums are toward a specific product. From samples to some users, so i would suggest a trusty lab results not a user result
The problem with trusting these so called professional testing facilities, is not only that they can be swayed by the top security companies quite easily, but their one size fits all methodology is not practical for testing many products that behave and react differently to threats. Not to mention that as users, we are not allowed open access to sample sets to even see if they are legit, in the wild, or even complete and non corrupted. There is no transparency, and you either have to blindly trust, or start testing yourself.
 

AtlBo

Level 26
Verified
Content Creator
To put it simply do not trust either of them Looks through the forums here would be a better idea to make an opinion on a AV.:)
I agree but I think there is an important purpose for independent (uhum...commercial) testing. In theory it can help users know how a program will perform at default settings. It also, and most importantly, puts a-v companies on an edge to make their defaults effective. This is good for PC owners in the general sense. It is a good idea to keep up with the latest PC security news too.

Just my opinion but these independent testing labs are not really aseptic but contaminated by marketing.
From our Hub we get real results.
(see above)

but remember antivirus can save you in one condition only and that is if you can keep your computer free of infections without using any security software
more confuse,you are free to curse me
OK, but the security software industry imo has a valuable role in keeping Microsoft and others with access to a PC honest. Who takes our side on issues about telemetry and so on and where would we be without this voice speaking for us?

The problem with trusting these so called professional testing facilities, is not only that they can be swayed by the top security companies quite easily, but their one size fits all methodology is not practical for testing many products that behave and react differently to threats. Not to mention that as users, we are not allowed open access to sample sets to even see if they are legit, in the wild, or even complete and non corrupted. There is no transparency, and you either have to blindly trust, or start testing yourself.
All of this is legit imo. Maybe it would help if testing organizations could be easily described and their methodologies known easily. I mean, they have credibility so maybe they should use that as Virus Total has done to better define who they are as a testing house (not that VT is a testing house, only that it has achieved credibility that the a-v testers perhaps have not yet achieved albeit in a different realm of the security grid). The methods regularly change which is admittedly made more complicated by the changing threat landscape. It is hard to know with great assurance what a test measures without a great deal of research, something most don't have time to do.

The scientific method always yields valid results, even if the testing methodologies of two tests are different. Yet, if the methods are different, the results mean something different in each case. It would help to know easily and without an hour of research in each case what a test is revealing, especially for the large and most trusted testers. Maybe they should work together to develop a way to create a common and simple presentation method for their testing methods, like a catchy single descriptive term for the test.
 
Last edited:

Game Of Thrones

Level 5
Verified
The problem with trusting these so called professional testing facilities, is not only that they can be swayed by the top security companies quite easily, but their one size fits all methodology is not practical for testing many products that behave and react differently to threats. Not to mention that as users, we are not allowed open access to sample sets to even see if they are legit, in the wild, or even complete and non corrupted. There is no transparency, and you either have to blindly trust, or start testing yourself.
Well about sharing the samples as far as i know about them(specially av comparative) they do share them with vendors, i don't think it's logical to share them for public! It's dangerous. Nearly all the names in av industry are participating in these tests and if they don't wanted to they can opt out. I did a research on the prizes that av comparative got, they are legit.
 

mekelek

Level 28
Well about sharing the samples as far as i know about them(specially av comparative) they do share them with vendors, i don't think it's logical to share them for public! It's dangerous. Nearly all the names in av industry are participating in these tests and if they don't wanted to they can opt out. I did a research on the prizes that av comparative got, they are legit.
you can't be biased in the malware samples section, that's what i would call the best place for "reviews".
 
  • Like
Reactions: AtlBo and Fritz