MT monthly antivirus test results

[MikeV]

Hello
I address to people who are and search/analyze malware in a daily basis.
I suggest a committee of who will make antivirus tests and they will post cumulative reports every end of each month. (something like AVComparatives, AVTest etc...)
People who will NOT test in favor of any AV Vendor,and they post the raw truth about testing results.

Test results needs to include:
Detection Rate
False Positives
Performance

Test results will be posted here every end of each month with a graphic board (like AVComparatives) if possible.
What do you think of my idea?

 

[thepierrezou]

It's a verry good idea

 

[NSG001]

I am sure (unless i dreamt it) in the past there was once a monthly chart

 

[Petrovic]

different samples(malware pack) - different AV can show different numbers of detections
Test for the detection-
This is only valid for a short period of time and not in long

 

[BoraMurdar]

Yes, it can be done but it will not be called "MalwareTips - Monthly Results"
Tests and results can be called *member* Monthly Results

 

[Exterminator]

Yes, it can be done but it will not be called "MalwareTips - Monthly Results"
Tests and results can be called *member* Monthly Results

I agree we already have Malware & Malware removal staff.

 

[NullPointerException]

I disagree. Do we want to sell out?

 

[Nico@FMA]

Requirements would have to be:

1: Windows 7 or 8 OS (Fully patched + Standard Settings & No Additions)
2: No virtualization but live & realtime testing.
3: Malware test methodology (Pre-infection testing, Real-time infection testing, Dormant and Remnant testing, Block & Removal & Repair testing, Zeroday & Injection & Hijack testing and finally information testing)

A: Pre-infection: See how the AV solution performs on a pre-infected system (Hostile environment test)
B: Real-time infection: See how AV solution usually performs on standard out of the box settings.
C: Dormant and Remnant: Sample pack testing and none active (require user actions) malicious files testing.
D: Block & Removal & Repair (Include vendor specialized tools), Zeroday & Injection & Hijack: Pretty much as it says...
E: Information: How does the AV solution inform the user and what recommendations it might show (Recommendations need to be followed so if AV solution says do not download then DON'T.

4: Installation self defend test + AV Recovery test (Use of vendor specialized tools to repair infection the main program could not)
5: Batch URL (No user intervention/actions!!) test (URLs not by hand but by automated process simulating a rogue program connecting to bad URL)
6: No hybrid engine testing (So no 1000 Bitdefender clones only "real" unique engines)

Many tests are bogus as testers execute files that in the real world no FOOL ever would double click.
So just follow the information provided by the system + security and follow it to the letter for optimum results.

There are much more idea's and options but these requirements would already be such a leap forward of any test shown on MT.
It would give our tests a huge credibility boost.
As mentioned BD and some other Vendors are selling their engine under license yet as VT shows many AV brands detect the same malware with pretty much the same name, so by testing for example the main engine of BD you pretty much can rule out any other BD clone.

That would be my recommendation.

 

[NullPointerException]

Requirements would have to be:

1: Windows 7 or 8 OS (Fully patched + Standard Settings & No Additions)
2: No virtualization but live & realtime testing.
3: Malware test methodology (Pre-infection testing, Real-time infection testing, Dormant and Remnant testing, Block & Removal & Repair testing, Zeroday & Injection & Hijack testing and finally information testing)

A: Pre-infection: See how the AV solution performs on a pre-infected system (Hostile environment test)
B: Real-time infection: See how AV solution usually performs on standard out of the box settings.
C: Dormant and Remnant: Sample pack testing and none active (require user actions) malicious files testing.
D: Block & Removal & Repair (Include vendor specialized tools), Zeroday & Injection & Hijack: Pretty much as it says...
E: Information: How does the AV solution inform the user and what recommendations it might show (Recommendations need to be followed so if AV solution says do not download then DON'T.

4: Installation self defend test + AV Recovery test (Use of vendor specialized tools to repair infection the main program could not)
5: Batch URL (No user intervention/actions!!) test (URLs not by hand but by automated process simulating a rogue program connecting to bad URL)
6: No hybrid engine testing (So no 1000 Bitdefender clones only "real" unique engines)

Many tests are bogus as testers execute files that in the real world no FOOL ever would double click.
So just follow the information provided by the system + security and follow it to the letter for optimum results.

There are much more idea's and options but these requirements would already be such a leap forward of any test shown on MT.
It would give our tests a huge credibility boost.
As mentioned BD and some other Vendors are selling their engine under license yet as VT shows many AV brands detect the same malware with pretty much the same name, so by testing for example the main engine of BD you pretty much can rule out any other BD clone.

That would be my recommendation.

This kinda test isn't bad, but tests like AVComparatives? Devil no.

 

[Nico@FMA]

This kinda test isn't bad, but tests like AVComparatives? Devil no.

AV Comparatives tests are on the surface similar yet their methodology is very different on top of that they test individual modules.
And the test i proposed is a full spectrum test. There is no point in testing only the AV engine as the program around it gives 9 out of 10 times the engine the balls needed to get something done. So only full spectrum testing will give adequate results.

 

[NullPointerException]

AV Comparatives tests are on the surface similar yet their methodology is very different on top of that they test individual modules.
And the test i proposed is a full spectrum test. There is no point in testing only the AV engine as the program around it gives 9 out of 10 times the engine the balls needed to get something done. So only full spectrum testing will give adequate results.

I do agree with you. But I don't want MT to sell out either.

 

[Malware1]

Who will make tests on a real machine ? Not me. I submit all my samples to AV vendors, it's a bad idea. These packs are not for AV testing.

 

[NullPointerException]

Real machines? The closet thing I've got to a "real machine for malware testing" is a VM.

 

[BoraMurdar]

Tests proposed by n.nvt are the real tests that every serious tester should conduct. At least I would try to do some if I had a proper testing machine.

 

[Nico@FMA]

Who will make tests on a real machine ? Not me. I submit all my samples to AV vendors, it's a bad idea. These packs are not for AV testing.

Well i personally have 3 machines here, that i can easy use for malware testing as they are imaged anyway, so if something goes wrong it takes less then 2 minutes to wipe and replace image = 100% clean system.
Its not that hard to make a test machine having a ready to go image. And for those that want to do somewhat meaning full testing this would be a huge plus.

@NullPointerException The tests are suppose to be for MT consumption and not for selling, so if people are going to test then this will be a unique feature to MT alone.

 

[NullPointerException]

Well i personally have 3 machines here, that i can easy use for malware testing as they are imaged anyway, so if something goes wrong it takes less then 2 minutes to wipe and replace image = 100% clean system.
Its not that hard to make a test machine having a ready to go image. And for those that want to do somewhat meaning full testing this would be a huge plus.

@NullPointerException The tests are suppose to be for MT consumption and not for selling, so if people are going to test then this will be a unique feature to MT alone.

Unless AV companies start contacting us and ask for "raise in scores".

 

[Atlas147]

Really like the idea of this, but I don't think every member here has a machine just lying around waiting to be used as a test machine.

 

[MikeV]

different samples(malware pack) - different AV can show different numbers of detections
Test for the detection-
This is only valid for a short period of time and not in long

Testing samples can be the same.
This project will be used ONLY to inform members of THIS forum.
And i repeat : The Committee need to be consistent by people who will NOT test in favor of any AV Vendor,and they post the raw truth about testing results

So here is my idea of how this project can work:

 

[Secondmineboy]

It would be good to know how many people could participate this and who can test which AV.

 

[MikeV]

It would be good to know how many people could participate this and who can test which AV.

OF COURSE my friend
I think everyone here knows who can participate in the project.