AV-Comparatives May 2017 test. Windows Defender did great!

[silversurfer]

The tests in the hub are good but NOT accurate as it too binary centric and not testing the real infection sources,chains and vectors. (URLs,Downloading the malware(some AV's like Avast have components to protect on download eg:cybercapture,filereputation(DRep) )

You have to find the right balance between being too binary centric and testing official entry points and chain.Also there has to be some relation with the binaries and their sources,eg: whats the point of testing something that's source is already dead or dies in a hour and also in my experience there tends to be lot of dead malware samples and some samples won't even do their dirty stuff on a VM so Behaviour blockers won't trigger

You don't have to like our testing here but you have to respect the work/efforts of other community members. You can't claim your own opinion as facts!
What are the real infection sources ? I'm assuming your thoughts we don't have to run the samples from the desktop...

I don't like such people like you they want to tell "I know everything about malware"

 

[Orion]

You don't have to like our testing here but you have to respect the work/efforts of other community members. You can't claim your own opinion as facts!
What are the real infection sources ? I'm assuming your thoughts we don't have to run the samples from the desktop...

I don't like such people like you they want to tell "I know everything about malware"

Who claimed I know everything about malware? Who said I don't respect the contributions? I myself analyze malware and hunt too.You simply assumed didn't you?.What is it with condescending tone? I have been a member of Antivirus and security related forums for 7 years and I have spent alot of time helping AV vendors and heck I even tweet when I find malware on twitter.I have spent time on this since teenage but you wouldn't respect that would you? I even analyze malware at times and I am learning it everyday.

Of course but you know everything about me and what I do online do you? or are you a expert to judge people's knowledge on here?

Who said I don't like such testing.Its interesting for all of us but it isn't perfect.See I said it's good but not supposed to be taken on face value.

Now coming to the "facts" I mean the real world scenario here (Even Umbra approves of this):

If you want to go to suspicious sites, just prepare to be infected anyway and make the precautions as backups and not storing anything even moderately sensitive on your machine. And I specifically said by "signatures". But there are also generic protections and layered protections.

See the typical chained scenario of today:
Porn site -> malicious js -> malicious pdf -> malicious downloader -> malicious binaries.

Test the whole infection chain and then talk about how legit you think your tests are.

Don't go to such porn site.
Don't use vulnerable apps.
Have antivirus with layered protection.

And then - who cares if av does not detect one of the downloaded malicious binaries, when the porn site is blocked and we detect the js and pdf?

It's very hard to evaluate the real-world performance of an AV solution when we don't (and I suspect we can't test the whole chain and prove if the user is protected. The tests on VT and such don't prove anything, but the ability of the engine to detect it by the signature.

The usual points about such static testing are:
a) the tests are carried long after the real infection took place, so it's kind of useless from today's point of view
b) the tests are carried without any context state information. Such information - if there is file named "document.doc .exe" in email, this is enough to ban the execution
c) the tests don't know anything about the relationship of the samples. If you detect the dropper, you don't have to detect the dropped binary.
d) the tests are too binary-centric and have only small amount of script/pdf/flash malware, althought these are one of the main vectors of getting thru to your computer.
e) there is little of no info on how the testbeds are created. All these 99.1% and such scores are complete nonsense from my point of view. The overlap of the product's detections is not as great as clementi/marx tests suggest.

This is not an excuse, that's an explanation what your really should read from the static tests. Yep, it's nice to be on the first places, but the world does not end if you're not there.

Is this enough for you yet? I love these tests as it is interesting for me to see how AV's perform against binaires especially AV's with strong web filters.

But again its about testing everything equally not just binaries!! Now I don't like people like you who love to come here and bark at other members without knowing anything about them or what they think and simply love to assume and bash them.This is really upsetting behaviour.

Also,why don't you go ahead and argue with the other members on here who too don't think your tests are perfect.You picked me out of all this looking at the amount of posts I have and thinking I am a newbie to the forums or the topic.Go ahead and argue with the others! I will continue to watch the results from the hub as its interesting to me to see some AVs with great web filters fall.

Best,
True Indian (Member of following forums: Wilders,emsisoft,malwarebytes,avast)

 

[silversurfer]

I don't have offending you personally but your own behavior is more rude for sure Do what you want it will no longer interesting me!

 

[Orion]

I don't have offending you personally but your own behavior is more rude for sure Do what you want it will no longer interesting me!

Who started it first mate? I even edited out the rude remark long before you responded but you were kind enough to quote it weren't You? You still haven't answered by direst questions sir! And Why not go argue with others and why you choose me?

And you just came out lashing saying "I don't like such people bla bla bla and assuming" you think that isn't rude? I have never been rude to anyone on here.

I see that you and some other people on here are very emotional about their tests and you can't stand 1 word against it especially by new members.You still want to argue Come on I am all ears mate! Trying to pick on newer members?

Also if my post offended you I apologize.

 

[Malware Person]

break it up guys, we shouldn't be arguing.

 

[ZeroDay]

break it up guys, we shouldn't be arguing.

I agree there's way too much arguing on here recently. It's a great forum lets try and keep it that way.

 

[Orion]

break it up guys, we shouldn't be arguing.

I am out of this honestly.I don't understand the emotions with these home grown tests.We on here have problems with well known testers too.How can we promote such stuff on here and let some 'home grown tester's with absolutely no idea of real world scenarios force it down our throats.We cannot let average users take these things on face value when we don't agree with well known testers itself (AV-C)

 

[mekelek]

Who claimed I know everything about malware? Who said I don't respect the contributions? I myself analyze malware and hunt too.You simply assumed didn't you?.What is it with condescending tone? I have been a member of Antivirus and security related forums for 7 years and I have spent alot of time helping AV vendors and heck I even tweet when I find malware on twitter.I have spent time on this since teenage but you wouldn't respect that would you? I even analyze malware at times and I am learning it everyday.

Of course but you know everything about me and what I do online do you? or are you a expert to judge people's knowledge on here?

Who said I don't like such testing.Its interesting for all of us but it isn't perfect.See I said it's good but not supposed to be taken on face value.

Now coming to the "facts" I mean the real world scenario here (Even Umbra approves of this):

If you want to go to suspicious sites, just prepare to be infected anyway and make the precautions as backups and not storing anything even moderately sensitive on your machine. And I specifically said by "signatures". But there are also generic protections and layered protections.

See the typical chained scenario of today:
Porn site -> malicious js -> malicious pdf -> malicious downloader -> malicious binaries.

Test the whole infection chain and then talk about how legit you think your tests are.

Don't go to such porn site.
Don't use vulnerable apps.
Have antivirus with layered protection.

And then - who cares if av does not detect one of the downloaded malicious binaries, when the porn site is blocked and we detect the js and pdf?

It's very hard to evaluate the real-world performance of an AV solution when we don't (and I suspect we can't test the whole chain and prove if the user is protected. The tests on VT and such don't prove anything, but the ability of the engine to detect it by the signature.

The usual points about such static testing are:
a) the tests are carried long after the real infection took place, so it's kind of useless from today's point of view
b) the tests are carried without any context state information. Such information - if there is file named "document.doc .exe" in email, this is enough to ban the execution
c) the tests don't know anything about the relationship of the samples. If you detect the dropper, you don't have to detect the dropped binary.
d) the tests are too binary-centric and have only small amount of script/pdf/flash malware, althought these are one of the main vectors of getting thru to your computer.
e) there is little of no info on how the testbeds are created. All these 99.1% and such scores are complete nonsense from my point of view. The overlap of the product's detections is not as great as clementi/marx tests suggest.

This is not an excuse, that's an explanation what your really should read from the static tests. Yep, it's nice to be on the first places, but the world does not end if you're not there.

Is this enough for you yet? I love these tests as it is interesting for me to see how AV's perform against binaires especially AV's with strong web filters.

But again its about testing everything equally not just binaries!! Now I don't like people like you who love to come here and bark at other members without knowing anything about them or what they think and simply love to assume and bash them.This is really upsetting behaviour.

Also,why don't you go ahead and argue with the other members on here who too don't think your tests are perfect.You picked me out of all this looking at the amount of posts I have and thinking I am a newbie to the forums or the topic.Go ahead and argue with the others! I will continue to watch the results from the hub as its interesting to me to see some AVs with great web filters fall.

Best,
True Indian (Member of following forums: Wilders,emsisoft,Malwarebytes,Avast)

didn't know visiting certain forums makes you a malware expert. hnggg

 

[VeeekTor]

break it up guys, we shouldn't be arguing.

I think it's relative, most people are so sheltered they think a healthy disagreement is a bad thing.

I can argue with a good friend, and still love them.

Let's not be so sheltered, we become weak, by seeking a non confrontational society.

I grew up in a very violent home, gun shots in the house ETC. My wife grew up like Ossie and Harriet, so when we argue, she thinks we are fighting, I think we are talking. She says "you just think we are not fighting, because no one is being thrown thru the window"...

So see it's all about perspective, I just have more tolerance for a disagreement than she does, which I think makes a person stronger.

 

[Winter Soldier]

BTW the topic of this thread is AV Comparatives May 2017 test, in case someone had forgotten that.

 

[ZeroDay]

I think it's relative, most people are so sheltered they think a healthy disagreement is a bad thing.

I can argue with a good friend, and still love them.

Let's not be so sheltered, we become weak, by seeking a non confrontational society.

I grew up in a very violent home, gun shots in the house ETC. My wife grew up like Ossie and Harriet, so when we argue, she thinks we are fighting, I think we are talking. She says "you just think we are not fighting, because no one is being thrown thru the window"...

So see it's all about perspective, I just have more tolerance for a disagreement than she does, which I think makes a person stronger.

I can assure you I'm far from sheltered and have lead a very colourful life, but the arguing on here is becoming petty, it's not even healthy debate in most cases.

 

[VeeekTor]

I can assure you I'm far from sheltered and have lead a very colourful life, but the arguing on here is becoming petty, it's not even healthy debate in most cases.

I guess I'm saying, it's OK to argue (discuss) IF you are saying "No the AV is not enough and here is why"... Another says "yes it is because it defeated the malware"

TO ME that is OK....

If I say your a stinking pig, and a slob then yes that is petty and is not needed... But healthy debate , sighting evidence for an opinion is educational, and should not be cast aside.

JMHO

 

[S3cur1ty 3nthu5145t]

Who claimed I know everything about malware? Who said I don't respect the contributions? I myself analyze malware and hunt too.You simply assumed didn't you?.What is it with condescending tone? I have been a member of Antivirus and security related forums for 7 years and I have spent alot of time helping AV vendors and heck I even tweet when I find malware on twitter.I have spent time on this since teenage but you wouldn't respect that would you? I even analyze malware at times and I am learning it everyday.

Of course but you know everything about me and what I do online do you? or are you a expert to judge people's knowledge on here?

Who said I don't like such testing.Its interesting for all of us but it isn't perfect.See I said it's good but not supposed to be taken on face value.

Now coming to the "facts" I mean the real world scenario here (Even Umbra approves of this):

If you want to go to suspicious sites, just prepare to be infected anyway and make the precautions as backups and not storing anything even moderately sensitive on your machine. And I specifically said by "signatures". But there are also generic protections and layered protections.

See the typical chained scenario of today:
Porn site -> malicious js -> malicious pdf -> malicious downloader -> malicious binaries.

Test the whole infection chain and then talk about how legit you think your tests are.

Don't go to such porn site.
Don't use vulnerable apps.
Have antivirus with layered protection.

And then - who cares if av does not detect one of the downloaded malicious binaries, when the porn site is blocked and we detect the js and pdf?

It's very hard to evaluate the real-world performance of an AV solution when we don't (and I suspect we can't test the whole chain and prove if the user is protected. The tests on VT and such don't prove anything, but the ability of the engine to detect it by the signature.

The usual points about such static testing are:
a) the tests are carried long after the real infection took place, so it's kind of useless from today's point of view
b) the tests are carried without any context state information. Such information - if there is file named "document.doc .exe" in email, this is enough to ban the execution
c) the tests don't know anything about the relationship of the samples. If you detect the dropper, you don't have to detect the dropped binary.
d) the tests are too binary-centric and have only small amount of script/pdf/flash malware, althought these are one of the main vectors of getting thru to your computer.
e) there is little of no info on how the testbeds are created. All these 99.1% and such scores are complete nonsense from my point of view. The overlap of the product's detections is not as great as clementi/marx tests suggest.

This is not an excuse, that's an explanation what your really should read from the static tests. Yep, it's nice to be on the first places, but the world does not end if you're not there.

Is this enough for you yet? I love these tests as it is interesting for me to see how AV's perform against binaires especially AV's with strong web filters.

But again its about testing everything equally not just binaries!! Now I don't like people like you who love to come here and bark at other members without knowing anything about them or what they think and simply love to assume and bash them.This is really upsetting behaviour.

Also,why don't you go ahead and argue with the other members on here who too don't think your tests are perfect.You picked me out of all this looking at the amount of posts I have and thinking I am a newbie to the forums or the topic.Go ahead and argue with the others! I will continue to watch the results from the hub as its interesting to me to see some AVs with great web filters fall.

Best,
True Indian (Member of following forums: Wilders,emsisoft,Malwarebytes,Avast)

Since it seems no one will step up and explain this that should, I will. As per your posted status on your profile, these are not just "STATIC" tests, although the HUB used to be just that, it used to be what I consider the "one right click wonder" where they ran a static test and submitted samples. Now someone came along and changed that, to both "STATIC & DYNAMIC" testing, so that all modules except the webfilter is tested at some point with each sample.

That said... You are correct there are other aspects, the problem is, those other aspects are not allowed here... They used to be, testing of URL's and other means used to be done here, but was removed by ADMIN and UPPER staff, as being too Dangerous. SO the testing done here, is as far as it will be allowed to go.

On that note, a little respect is all the guys ask for while being here and doing this on a daily basis, and NOT getting paid for it, but volunteering their personal time doing so.

As for your testing, you are not posting anything here to lead us to believe you do so, not in the hub, and not in the forums post. Now while I test independently myself, I had someone challenge that same fact about me and had no problem demonstrating it to him. If you would like to post a demonstration of testing correctly, and give points and tips and hell, even try to convince the Admin here to open up other testing aspects, then by all means, allow me to grab my popcorn and watch.

 

[Orion]

On that note, a little respect is all the guys ask for while being here and doing this on a daily basis, and NOT getting paid for it, but volunteering their personal time doing so.

1/3 made me respond.I can't post anything in the malware hub section till i have 100 posts on here.Also If you want some proof of whatever I do you can go check emsisoft forums and even my twitter @avman1995

So Do I.Why don't you tell silversurfer too.You won't because he is a level 36 or whatever right?What is wrong with pointing out flaws if you are talking about proper testing.Why do some of you like silversurfer come out rudely lashing on newer members and don't do that with the others who are also not taking your test seriously.Saying "I don't like people bla bla" without stating any proper contradiction to my post but instead just bash at me is both rude and childish.

Some of you seem to be bent upon or too emotional about your tests and you know I need to have 100 posts to post anything over here first.

Who said I don't respect your tests.Silversurfer pointlessly started that assumption and went on to post a taunting comment so I retaliated with what I thought made sense about real world usage and tests.Why do you home grown testers want to shove your "almighty methods" on us over here when we don't we agree with AV-C and others.Its our opinion and we can stick to it!

The very fact I watch the youtube tests on here and comment too on it shows my interest in that aspect.But again I think ganging up on me does the trick.

So now if we are finished with a pointless argument over home grown tests we can move on to bigger things.I think this forum has given a little too much of spot light to home grown tests so it allows people like silversurfer who doesn't even know the real world problems to assume and bark at someone who is at a lower forum rank thinking he won't get opposed.

A new thing is being seen now where people are producing YouTube videos ( YouTesters ) of their private tests of anti-virus or anti-malware programs.

Now a days, anyone can publish their videos to the world using YouTube. That's great, however, it also means that anyone can proclaim themselves an expert tester and have links to their YouTube tests or channel published all over the forums. In some regards, this can be considered click-thru spam, to drive traffic to up their video view counts. In other cases, they pit two products against each other in a video version of a This vs That debate.

First, a person can't post what is presented as a legitimate and valid test, but, when challenged as to the flaws in test methodology, fall back to saying they are not a professional tester and these were done for fun. The methodology flaws remain and that slight qualifier somewhere down in the thread won't stop the majority of readers from taking the results as some how meaningful when they are not.

Second, when you "collect lots of samples" and call them all malware, yet, have not verified the samples, your testbed is invalid. You can't just upload some samples to VT (or use a local product's scanner), to declare the samples legitimate malware for testing against the rest. Also, you can not collect samples for 10 days, test on the 10th day, and still call the samples 0-day.

Without verifying and/or testing the individual malware samples, you can't know how many of the samples are real malware versus harmless or broken files. With large numbers of collected samples, it is highly likely that the sample set has many duplicates, perhaps only named differently. Professional organizations always de-dupe their malware sets to prevent detection results from being skewed.

If a collection contains a wide mix of malware types, including those "potentially unwanted..." items, like riskware or similar, you need to separate them into groups for testing since listing total detections counts versus misses is skewed and misleading for the actual severity involved.

These are just a few of the reasons why we view these types of home grown tests as both meaningless and yes, even damaging since some people will start to blindly follow the results as an indicator of product effectiveness.

This is not a excuse.It is what you should be reading off such tests.In any case, I am not actually trying to define proper testing procedures here. I'll leave that to groups like AMTSO.

Conclusion: Its fun to watch AV's with great web filters fall against only binary centric test.But that isn't all.As stated before these tests need to taken with a grain of salt and more often than not most people do it for fun since they enjoy it and still want people to take their test seriously which cannot happen.It's good but can't be perfect.We appreciate what you do but its not something some of us can take at face value.

"didn't know visiting certain forums makes you a malware expert. hnggg"

Give me a break from this taunting posts and bullying! Isn't this the petty bickering on here? This argument isn't even healthy its just like what silversurfer said first.Taunting and very rude IMO.

Oh please! silversurfer thought i am newbie to the topic and tried to pick off me with a rude post.The entire goal of that was to show I am not a "don't know anything" newbie over here.Why don't you go and argue with silversurfer instead of ganging up on me here.What is it with the bullying and I will no more be responding to any childish or rude comments on this thread.If you have a valid argument then we can talk if not I would be obliged if some of you can stop ganging up a newbie.

 

[S3cur1ty 3nthu5145t]

1/3 made me respond.I can't post anything in the malware hub section till i have 100 posts on here.Also If you want some proof of whatever I do you can go check emsisoft forums and even my twitter @avman1995

So Do I.Why don't you tell silversurfer too.You won't because he is a level 36 or whatever right?What is wrong with pointing out flaws if you are talking about proper testing.Why do some of you like silversurfer come out rudely lashing on newer members and don't do that with the others who are also not taking your test seriously.Saying "I don't like people bla bla" without stating any proper contradiction to my post but instead just bash at me is both rude and childish.

Some of you seem to be bent upon or too emotional about your tests and you know I need to have 100 posts to post anything over here first.

Who said I don't respect your tests.Silversurfer pointlessly started that assumption and went on to post a taunting comment so I retaliated with what I thought made sense about real world usage and tests.Why do you home grown testers want to shove your "almighty methods" on us over here when we don't we agree with AV-C and others.Its our opinion and we can stick to it!

The very fact I watch the youtube tests on here and comment too on it shows my interest in that aspect.But again I think ganging up on me does the trick.

So now if we are finished with a pointless argument over home grown tests we can move on to bigger things.I think this forum has given a little too much of spot light to home grown tests so it allows people like silversurfer who doesn't even know the real world problems to assume and bark at someone who is at a lower forum rank thinking he won't get opposed.

A new thing is being seen now where people are producing YouTube videos ( YouTesters ) of their private tests of anti-virus or anti-malware programs.

Now a days, anyone can publish their videos to the world using YouTube. That's great, however, it also means that anyone can proclaim themselves an expert tester and have links to their YouTube tests or channel published all over the forums. In some regards, this can be considered click-thru spam, to drive traffic to up their video view counts. In other cases, they pit two products against each other in a video version of a This vs That debate.

First, a person can't post what is presented as a legitimate and valid test, but, when challenged as to the flaws in test methodology, fall back to saying they are not a professional tester and these were done for fun. The methodology flaws remain and that slight qualifier somewhere down in the thread won't stop the majority of readers from taking the results as some how meaningful when they are not.

Second, when you "collect lots of samples" and call them all malware, yet, have not verified the samples, your testbed is invalid. You can't just upload some samples to VT (or use a local product's scanner), to declare the samples legitimate malware for testing against the rest. Also, you can not collect samples for 10 days, test on the 10th day, and still call the samples 0-day.

Without verifying and/or testing the individual malware samples, you can't know how many of the samples are real malware versus harmless or broken files. With large numbers of collected samples, it is highly likely that the sample set has many duplicates, perhaps only named differently. Professional organizations always de-dupe their malware sets to prevent detection results from being skewed.

If a collection contains a wide mix of malware types, including those "potentially unwanted..." items, like riskware or similar, you need to separate them into groups for testing since listing total detections counts versus misses is skewed and misleading for the actual severity involved.

These are just a few of the reasons why we view these types of home grown tests as both meaningless and yes, even damaging since some people will start to blindly follow the results as an indicator of product effectiveness.

This is not a excuse.It is what you should be reading off such tests.In any case, I am not actually trying to define proper testing procedures here. I'll leave that to groups like AMTSO.

Conclusion: Its fun to watch AV's with great web filters fall against only binary centric test.But that isn't all.As stated before these tests need to taken with a grain of salt and more often than not most people do it for fun since they enjoy it and still want people to take their test seriously which cannot happen.It's good but can't be perfect.We appreciate what you do but its not something some of us can take at face value.

"didn't know visiting certain forums makes you a malware expert. hnggg"

Give me a break from this taunting posts and bullying! Isn't this the petty bickering on here? This argument isn't even healthy its just like what silversurfer said first.Taunting and very rude IMO.

Oh please! silversurfer thought i am newbie to the topic and tried to pick off me with a rude post.The entire goal of that was to show I am not a "don't know anything" newbie over here.Why don't you go and argue with silversurfer instead of ganging up on me here.What is it with the bullying and I will no more be responding to any childish or rude comments on this thread.If you have a valid argument then we can talk if not I would be obliged if some of you can stop ganging up a newbie.

Well it seems to me, they are stating what they are because obviously you are not paying attention. The samples here are vetted before posting for working non corrupted, they are also run through automated malware analysis sandboxes to verify working legit malware before posted as well as VT, just like I stated in the post I tagged you in. So you are spreading untrue statements by posting what you are, and you are trying to discredit them.

Maybe they should just shut the hub down and let you as some kind of wanna be expert school them instead, why should they waste their time volunteering if ever wanna be that walks in tries to discredit their test, some without fully looking and understanding what they are bashing, said it yourself, you do not have access to the samples, but yet claim to know all about them.

BTW, Levels and likes mean absolutely nothing here when it comes to knowledge.

 

[Orion]

Well it seems to me, they are stating what they are because obviously you are not paying attention. The samples here are vetted before posting for working non corrupted, they are also run through automated malware analysis sandboxes to verify working legit malware before posted as well as VT, just like I stated in the post I tagged you in. So you are spreading untrue statements by posting what you are, and you are trying to discredit them.

Maybe they should just shut the hub down and let you as some kind of wanna be expert school them instead, why should they waste their time volunteering if ever wanna be that walks in tries to discredit their test, some without fully looking and understanding what they are bashing, said it yourself, you do not have access to the samples, but yet claim to know all about them.

BTW, Levels and likes mean absolutely nothing here when it comes to knowledge.

I was talking about youtube testing if you read that properly!

First, a person can't post what is presented as a legitimate and valid test, but, when challenged as to the flaws in test methodology, fall back to saying they are not a professional tester and these were done for fun. The methodology flaws remain and that slight qualifier somewhere down in the thread won't stop the majority of readers from taking the results as some how meaningful when they are not.

And whatever you are assuming right here.When did I call myself that?? is wrong anyway and is straight down another unhealthy argument of a "self proclaimed genuine tester without even a certification" This is exactly what you are coming out to be with your assumptions.You don't want to agree I don't care but don't waste your time trying to discredit the argument and defend the flaws.

Maybe they should just shut the hub down and let you as some kind of wanna be expert school them instead, why should they waste their time volunteering if ever wanna be that walks in tries to discredit their test, some without fully looking and understanding what they are bashing, said it yourself, you do not have access to the samples, but yet claim to know all about them.

Maybe AMTSO should test your hub testing and certify it if you are that confident.

Why don't you accept that these tests cannot be taken on face value as we cannot test the entire chain instead of keeping on coming up with rude assumptions that I never made just what silversurfer did.Whats your problem?

If you want to go to suspicious sites, just prepare to be infected anyway and make the precautions as backups and not storing anything even moderately sensitive on your machine. And I specifically said by "signatures". But there are also generic protections and layered protections.

See the typical chained scenario of today:
Porn site -> malicious js -> malicious pdf -> malicious downloader -> malicious binaries.

Test the whole infection chain and then talk about how legit you think your tests are.

Don't go to such porn site.
Don't use vulnerable apps.
Have antivirus with layered protection.

And then - who cares if av does not detect one of the downloaded malicious binaries, when the porn site is blocked and we detect the js and pdf?

It's very hard to evaluate the real-world performance of an AV solution when we don't (and I suspect we can't test the whole chain and prove if the user is protected. The tests on VT and such don't prove anything, but the ability of the engine to detect it by the signature.

The usual points about such static testing are:
a) the tests are carried long after the real infection took place, so it's kind of useless from today's point of view
b) the tests are carried without any context state information. Such information - if there is file named "document.doc .exe" in email, this is enough to ban the execution
c) the tests don't know anything about the relationship of the samples. If you detect the dropper, you don't have to detect the dropped binary.
d) the tests are too binary-centric and have only small amount of script/pdf/flash malware, althought these are one of the main vectors of getting thru to your computer.
e) there is little of no info on how the testbeds are created. All these 99.1% and such scores are complete nonsense from my point of view. The overlap of the product's detections is not as great as clementi/marx tests suggest.

The very fact I watch the youtube tests on here and comment too on it shows my interest in that aspect.

You still don't understand or you haven't read that line and my post that started it all on the previous page.I said:

The tests in the hub are good but NOT accurate

Still some of you want to get all worked up? Come on! That's pathetic!
Even after I said that,you clearly still think I don't credit your tests or the testers.I have no problem with them These are just a few of the reasons why we should view these types of home grown tests as both meaningless and yes, even damaging since some people will start to blindly follow the results as an indicator of product effectiveness.You know I even comment on youtube tests despite all the flaws.Its interesting to me since i know what's going on but not to the avg. users who take everything on face value.

Stick and stones won't break my bones.You keep trying to showcase me as a "expert" or whatever your assumptions are it won't change the fact of the matter.Clearly,you and some people on here are misinterpreting "flaws and what the avg. user takes away from it" to "he doesn't credit us"

Why do you need a credit from me anyway? Ask AMTSO to evaluate and credit.We do what we do because we want to do it not for the credit! Even after mentioning it time and again that i am not taking any credit from them it looks like you people are bent upon on arrogant replies and taunts.I am rather sad to see people wanting to argue and not wanting to find ways of fixing the wrong conclusion taken by regular users.

PS. You can call me a expert or noob.I don't care for either label.For me what's important is the average user on this forum who watches these things.It will be like a slap on some people's face when I have 100 posts and join posting samples in the Hub.Of course then you will still think I am against it won't you? Go ahead.....You can keep calling me out as "no good self proclaimed expert" but that wouldn't fix the problem would it? How about we have a thread that warns the user not to take everything at face value with proper explaination.

So you still want to have a go at me? Good Luck.As I am done with this pointless argument and hopefully you won't take this on any further and keep calling me names which I don't appreciate from someone of a higher forums rank.Go ahead and tell the mods to ban me for trying to point out flaws and tell people to not take things on face value which for some of you = discrediting us!!

 

[ZeroDay]

1/3 made me respond.I can't post anything in the malware hub section till i have 100 posts on here.Also If you want some proof of whatever I do you can go check emsisoft forums and even my twitter @avman1995

So Do I.Why don't you tell silversurfer too.You won't because he is a level 36 or whatever right?What is wrong with pointing out flaws if you are talking about proper testing.Why do some of you like silversurfer come out rudely lashing on newer members and don't do that with the others who are also not taking your test seriously.Saying "I don't like people bla bla" without stating any proper contradiction to my post but instead just bash at me is both rude and childish.

Some of you seem to be bent upon or too emotional about your tests and you know I need to have 100 posts to post anything over here first.

Who said I don't respect your tests.Silversurfer pointlessly started that assumption and went on to post a taunting comment so I retaliated with what I thought made sense about real world usage and tests.Why do you home grown testers want to shove your "almighty methods" on us over here when we don't we agree with AV-C and others.Its our opinion and we can stick to it!

The very fact I watch the youtube tests on here and comment too on it shows my interest in that aspect.But again I think ganging up on me does the trick.

So now if we are finished with a pointless argument over home grown tests we can move on to bigger things.I think this forum has given a little too much of spot light to home grown tests so it allows people like silversurfer who doesn't even know the real world problems to assume and bark at someone who is at a lower forum rank thinking he won't get opposed.

A new thing is being seen now where people are producing YouTube videos ( YouTesters ) of their private tests of anti-virus or anti-malware programs.

Now a days, anyone can publish their videos to the world using YouTube. That's great, however, it also means that anyone can proclaim themselves an expert tester and have links to their YouTube tests or channel published all over the forums. In some regards, this can be considered click-thru spam, to drive traffic to up their video view counts. In other cases, they pit two products against each other in a video version of a This vs That debate.

First, a person can't post what is presented as a legitimate and valid test, but, when challenged as to the flaws in test methodology, fall back to saying they are not a professional tester and these were done for fun. The methodology flaws remain and that slight qualifier somewhere down in the thread won't stop the majority of readers from taking the results as some how meaningful when they are not.

Second, when you "collect lots of samples" and call them all malware, yet, have not verified the samples, your testbed is invalid. You can't just upload some samples to VT (or use a local product's scanner), to declare the samples legitimate malware for testing against the rest. Also, you can not collect samples for 10 days, test on the 10th day, and still call the samples 0-day.

Without verifying and/or testing the individual malware samples, you can't know how many of the samples are real malware versus harmless or broken files. With large numbers of collected samples, it is highly likely that the sample set has many duplicates, perhaps only named differently. Professional organizations always de-dupe their malware sets to prevent detection results from being skewed.

If a collection contains a wide mix of malware types, including those "potentially unwanted..." items, like riskware or similar, you need to separate them into groups for testing since listing total detections counts versus misses is skewed and misleading for the actual severity involved.

These are just a few of the reasons why we view these types of home grown tests as both meaningless and yes, even damaging since some people will start to blindly follow the results as an indicator of product effectiveness.

This is not a excuse.It is what you should be reading off such tests.In any case, I am not actually trying to define proper testing procedures here. I'll leave that to groups like AMTSO.

Conclusion: Its fun to watch AV's with great web filters fall against only binary centric test.But that isn't all.As stated before these tests need to taken with a grain of salt and more often than not most people do it for fun since they enjoy it and still want people to take their test seriously which cannot happen.It's good but can't be perfect.We appreciate what you do but its not something some of us can take at face value.

"didn't know visiting certain forums makes you a malware expert. hnggg"

Give me a break from this taunting posts and bullying! Isn't this the petty bickering on here? This argument isn't even healthy its just like what silversurfer said first.Taunting and very rude IMO.

Oh please! silversurfer thought i am newbie to the topic and tried to pick off me with a rude post.The entire goal of that was to show I am not a "don't know anything" newbie over here.Why don't you go and argue with silversurfer instead of ganging up on me here.What is it with the bullying and I will no more be responding to any childish or rude comments on this thread.If you have a valid argument then we can talk if not I would be obliged if some of you can stop ganging up a newbie.

All samples in the Hub are vetted. I'm not defending anyone and I don't take post counts, or the 'Staff' level as an indicator of the level of a user's knowledge because it absolutely isn't. I agree that there is, at times silly arguing and bickering, but that will happen anywhere you group a lot of people together.

With all that said, and I honestly do NOT mean to cause you offense but you are clearly very insecure and use the bullying card way too loosely. Like I say I couldn't care less how many posts a member has, or if they're staf or not, if I think they're wrong I'll tell them, but you just seem very insecure and angry. If you want people to show you respect and to be polite try doing the same first. I'll be putting you on my ignore list because you're clearly a very angry person who wants to argue.

 

[Lord Ami]

People, this is AVC-C thread Argue in private.

I just can not believe 100% detection stuff. Neither AV-C or AV-Test ones. It's just too common nowadays for many products to have 100% in detection tests while in real world it's not gonna happen. In my personal opinion it's just so AVs can market as "AV-C 100% detection rate!" or similar... They would really need to use better samples or techniques (they should have the resources). Until then, I don't choose product based on these test results. Again, my personal opinion.

 

[Orion]

With all that said, and I honestly do NOT mean to cause you offense but you are clearly very insecure and use the bullying card way too loosely. Like I say I couldn't care less how many posts a member has, or if they're staf or not, if I think they're wrong I'll tell them, but you just seem very insecure and angry. If you want people to show you respect and to be polite try doing the same first. I'll be putting you on my ignore list because you're clearly a very angry person who wants to argue.

I didn't even put you in between.I don't understand why you want to come in between of all the talk.You know it was SS who started it first.You can see it but you still want to put it on me? Cool go ahead and ignore me because my opinions offend you and because i responded to SS's rude response to me.This is just unwanted drama.

 

[silversurfer]

I didn't even put you in between.I don't understand why you want to come in between of all the talk.You know it was SS who started it first.You can see it but you still want to put it on me? Cool go ahead and ignore me because my opinions offend you and because i responded to SS's rude response to me.This is just unwanted drama.

You have to stop calling myself a rude person! You have offended myself for several times here