MT monthly antivirus test results
- Thread starter MikeV
- Start date
- Status
- Apr 25, 2013
- 5,355
This is only possible simultaneously
for example
Read the methodology AV-Comparatives -file detection test
- May 16, 2013
- 844
I'm in, only if someone can help me or give me some tips about malware testing in VM.
http://malwaretips.com/threads/2014-09-05-103.32934/page-2#post-256622
http://malwaretips.com/threads/2014-09-05-103.32934/page-2#post-256622
- Aug 25, 2014
- 580
- Sep 9, 2013
- 925
The Committee will decide the Rules and ''How to''
I just express my idea.
The ONLY ''rule'' that can be sctrict and not negotiable is :
People who like to participate in the project will NOT test in favor of any AV Vendor,and they post the raw truth about testing results.
/hunters/testers will decide the rest
I just express my idea.
The ONLY ''rule'' that can be sctrict and not negotiable is :
People who like to participate in the project will NOT test in favor of any AV Vendor,and they post the raw truth about testing results.
/hunters/testers will decide the rest
- Aug 25, 2014
- 580
But if a staff member gets contacted from an AV company to "increase" their "scores"? There'd be no difference between PCMag and us. Nobody knows what staff X will do if staff X gets evil.
- Sep 9, 2013
- 925
That's why the only rule that is not negotiable for this project is what i wrote above.
Only members trusted by the forum for their hard work, NOT fans of any AV Vendor.
- Aug 25, 2014
- 580
i am against the idea; in theory it is nice, in practice it is not credible, why?
1- who will guarantee the results?
2- who is "expert" enough to proceed the tests with a proper, flawless and consistent methodology ?
3- who will get the samples? from where? How? When?
too many questions, few benefits but potentials troubles for MT.
not saying i consider test labs as worthless, so i dont want MT be assimilated to them.
now it is just my opinion
1- who will guarantee the results?
2- who is "expert" enough to proceed the tests with a proper, flawless and consistent methodology ?
3- who will get the samples? from where? How? When?
too many questions, few benefits but potentials troubles for MT.
not saying i consider test labs as worthless, so i dont want MT be assimilated to them.
now it is just my opinion
- Jun 5, 2013
- 835
Hello, my 2 cents...
I think the idea is great, and that's exactly its problem. It's a great, big idea, one that would take a lot of coordinated work to get off the ground and flying properly. Of course, many users spend their time bringing individual test results up here, but to work coordinately is an entirely different matter. For one, testers would now have to stick to certain schedules, instead of doing the tests whenever they want/can, simply because there can't be too much time between tester #1 bringing AV#1 results and tester #2 bringing AV#2 results, if we want to compare the results with the same samples. And this is not to mention that the very nature of these tests would be much more complex and time-consuming than simple VM tests.
Then there is this issue that has been brought up about AV companies paying testers for better results, or even interfering in other ways with the test results in order to mask them and make themselves look better. This idea, as I said would require a lot of coordination, but it wouldn't require, nor would it be possible, to work as coordinately as a proper testing company. And this brings a serious problem, because if you are worried that a company might be accepting money to fake results, and this means the boss agreed, and all the workers said ok to that too; then how concerned can you be about individual testers that do not work in the same office, or do not even live in the same country, being approached by such companies? The lack of centralization could make such threats to the truthfulness of the tests even more severe, and much harder to control.
Maybe this could work without such fixed schedules, but then it wouldn't be different from the freedom we already have of posting test results, videos, etc.
If I can see this thing working, I believe it's with a group of trusted members working more or less together (I'll test AV#1, you test AV#2 kind of thing) releasing results of tests done with proper procedures, at their leisure and not with a fixed schedule. The community could vote for the products they want tested, plus doing a rotation to make sure a broad spectrum of products is tested.
As an additional note, I'd like to say I believe excluding clones is a bad idea in my opinion, because clones are not the same as their originals, and may lack features, or have extra features, plus they can be more or less buggy than the originals, not to mention sometimes the database is not the same as the original, so I see a lot of reasons to expect different results from the clones, which would make testing them useful.
I think the idea is great, and that's exactly its problem. It's a great, big idea, one that would take a lot of coordinated work to get off the ground and flying properly. Of course, many users spend their time bringing individual test results up here, but to work coordinately is an entirely different matter. For one, testers would now have to stick to certain schedules, instead of doing the tests whenever they want/can, simply because there can't be too much time between tester #1 bringing AV#1 results and tester #2 bringing AV#2 results, if we want to compare the results with the same samples. And this is not to mention that the very nature of these tests would be much more complex and time-consuming than simple VM tests.
Then there is this issue that has been brought up about AV companies paying testers for better results, or even interfering in other ways with the test results in order to mask them and make themselves look better. This idea, as I said would require a lot of coordination, but it wouldn't require, nor would it be possible, to work as coordinately as a proper testing company. And this brings a serious problem, because if you are worried that a company might be accepting money to fake results, and this means the boss agreed, and all the workers said ok to that too; then how concerned can you be about individual testers that do not work in the same office, or do not even live in the same country, being approached by such companies? The lack of centralization could make such threats to the truthfulness of the tests even more severe, and much harder to control.
Maybe this could work without such fixed schedules, but then it wouldn't be different from the freedom we already have of posting test results, videos, etc.
If I can see this thing working, I believe it's with a group of trusted members working more or less together (I'll test AV#1, you test AV#2 kind of thing) releasing results of tests done with proper procedures, at their leisure and not with a fixed schedule. The community could vote for the products they want tested, plus doing a rotation to make sure a broad spectrum of products is tested.
As an additional note, I'd like to say I believe excluding clones is a bad idea in my opinion, because clones are not the same as their originals, and may lack features, or have extra features, plus they can be more or less buggy than the originals, not to mention sometimes the database is not the same as the original, so I see a lot of reasons to expect different results from the clones, which would make testing them useful.
- Jul 23, 2013
- 2,115
You'd have the same problem as these AV labs have - how to determine what samples and in what percentage ratio to include so as to match their actual prevalence in the real world. And unless you have access to hundreds of millions of computers to see which malware is going around and actually infecting machines, all this is just a big guess.
- Apr 13, 2013
- 3,147
Although a very nice idea, a few problems that occur to my currently wine-addled mind may arise:
1). For just Real-time scans (no running of samples)- For accuracy all samples should be run at basically the same time for each AV tested, thereby negating any timeframe bias (ie Avast tested at 10AM, but by the time one gets to Zillya it is now 3PM, thereby giving Avast a headstart in detection). The only way to solve this issue is to have a number of members running samples at the exact same time on different AV’s, which is impractical.
2). There seems to be a compulsion to submit potentially malicious files to the vendor. If this is still done, would the results show the products's inherent strength of intrinsic Real-Time detection or instead the efficiency of the Emergency Response team?
3). For actually running samples- This presents a greater problem. To be accurate in determining if any malicious activity exists after running a particular sample, forensics must be done after letting the sample percolate around the system for some set time. To run any number of samples would be time consuming even if your setup doesn’t have what amounts to real-time forensics modules (FireEye). Without such a setup even a person with OCD on amphetamines will need at the VERY least 5 minutes per sample, so a small pack of 10 malware files will take about an hour to yield semi-meaningful results. To repeat this process on multiple AV’s would bring in time bias as noted above, and having multiple users doing malware analysis on different products at identical time to avoid the time bias issue is again unrealistic.
4). Mention has been made that malware samples should never be run in a VM- Although true, VM aware malware constitute a small percentage of samples found in the Wild. To say that since the results for 5% (VM aware) of all samples are invalid, the results for other 95% of non-VM aware samples are also invalid is incorrect and rather short-sighted. So for those testing in a VM, keep doing it.
To sum, although the idea of community testing is a noble one, the problems in proper implementation would make any results extremely suspect (kind of like the AV Comparative results).
1). For just Real-time scans (no running of samples)- For accuracy all samples should be run at basically the same time for each AV tested, thereby negating any timeframe bias (ie Avast tested at 10AM, but by the time one gets to Zillya it is now 3PM, thereby giving Avast a headstart in detection). The only way to solve this issue is to have a number of members running samples at the exact same time on different AV’s, which is impractical.
2). There seems to be a compulsion to submit potentially malicious files to the vendor. If this is still done, would the results show the products's inherent strength of intrinsic Real-Time detection or instead the efficiency of the Emergency Response team?
3). For actually running samples- This presents a greater problem. To be accurate in determining if any malicious activity exists after running a particular sample, forensics must be done after letting the sample percolate around the system for some set time. To run any number of samples would be time consuming even if your setup doesn’t have what amounts to real-time forensics modules (FireEye). Without such a setup even a person with OCD on amphetamines will need at the VERY least 5 minutes per sample, so a small pack of 10 malware files will take about an hour to yield semi-meaningful results. To repeat this process on multiple AV’s would bring in time bias as noted above, and having multiple users doing malware analysis on different products at identical time to avoid the time bias issue is again unrealistic.
4). Mention has been made that malware samples should never be run in a VM- Although true, VM aware malware constitute a small percentage of samples found in the Wild. To say that since the results for 5% (VM aware) of all samples are invalid, the results for other 95% of non-VM aware samples are also invalid is incorrect and rather short-sighted. So for those testing in a VM, keep doing it.
To sum, although the idea of community testing is a noble one, the problems in proper implementation would make any results extremely suspect (kind of like the AV Comparative results).
- Apr 25, 2013
- 5,355
on the contrary
90% of users have little understanding that they run)
Download from unknown sources, keygens, patches....etc.... Shutdown monitor in real time(AV), launch ->Result
In any case, we have amateur tests and they do not claim to rank - professionally
- Apr 13, 2013
- 3,147
i think 90% is an understatement. Looking at the recent Target and Home Depot breaches, both of which were initiated by someone double-clicking an email attachment (I guess that person would be in the 90%) resulting in a massive infection that wasn't caught by IT Professionals (the other 10%).
Last edited:
- Aug 25, 2014
- 580
- Status
Similar threads
