Malware Hub Methodolgy

[Lucent Warrior]

Lets open this up for suggestions, since there seems to be many opinions on what is best. A developer just walked in here and Admitted NO ONE knows how to properly Test all products.

This is the simple explanation of the Methodology that was administered by myself and @LabZero . I was known as the user illumination during that time.

It is a simple method, one that covers all products instead of being just tailored to a certain set of standards "like the type the AV testing organizations employ", and saying sorry about your products luck because it was not designed to respond like others. Because the method is so simple, it allows the tester to form the test around the product and not the other way around, which i think is very important and proper when testing.

Running the static scan first and correctly, so as to only test signatures then enabling the rest of the products modules then executing the samples one at a time and watching the changes done/made to the system and or how the product handles terminating them, such as Voodooshield suspending the process of the file in order to give the user time to react to the file ect. This style of testing while simplified, allows you to accurately watch the product work as designed.

Why is testing important: To not only make sure the product works as designed, but many times, this will also help rear the ugly heads of Bugs in the product itself. It also allows users to see what they are spending money on.

Analogy: What good is it, if you buy a car from a dealership, that has been built and tested at the facility, but the first time you drive it down the road it malfunctions and your wreck. You do not blame it on the driver, and or cover it up by stating you know better then anyone how the car was built.

Now, if any of you have a better method for testing all kinds of different approaches by many products, that can be placed into one set of rules/guidelines, im listening.

 

[hjlbx]

Running the static scan first and correctly, so as to only test signatures then enabling the rest of the products modules then executing the samples one at a time and watching the changes done/made to the system and or how the product handles terminating them, such as Voodooshield suspending the process of the file in order to give the user time to react to the file ect. This style of testing while simplified, allows you to accurately watch the product work as designed.

Why is testing important: To not only make sure the product works as designed, but many times, this will also help rear the ugly heads of Bugs in the product itself. It also allows users to see what they are spending money on.

1. Simple to follow test methodology

2. AV scan first to gauge the scan engine detection rate

3. Execute remaining undetected files to observe effectiveness of proactive protection(s)

When I participated in the MalwareTips Malware Hub testing this is the exact test procedure to which I adhered.

At the same time I did not assume that every sample was definitively malicious; some were no doubt false positives, others duds, and still others I could not tell what they were doing - nor did I try to find out as the simple test methodology and meeting the Hub posting requirements was time-intensive enough all by itself. That being said, most samples were deemed malicious by the security softs.

My criteria - if you want to call it that - was "Is the security soft behaving as I expect it to behave ?" If "No," then sometimes I earnestly did try to find out why.

There's many reasons why the soft didn't do what I expected it to do - things I would never have even considered. Now we all know most testers aren't going to take this extra step because it is even more time-intensive than the testing - not to mention that vendors rarely reply to these types of queries - so a lot of testers judge the extra effort as not worth the end result.

Bugs were most definitely revealed during MT Hub testing and those bugs were reported to the vendors with the samples attached. More often than not, those bug reports were accepted instead of rejected. So, there was a definite value to testing for some vendors.

* * * * *

If I had to perform Hasherazade-grade (anyone know who she is ?) analysis on every single sample file, then I would not have even bothered to participate.

* * * * *

If there is a better way to test - and one that does not place too onerous of a burden on the testers - then hopefully someone will provide the guidance by saying "Here, test this way, it is better and this is why..." At least to some bare-minimum standard to which all interested parties can agree that the testing will net some acceptable, meaningful results.

Until that guidance is provided and the test standard implemented, then 1, 2 and 3 above are what I would do within the context of testing here at MT.

Perhaps I am a simpleton and it's all amateur-night grade testing, but to judge the entirety of the testing performed and reported here as worthless is just a wee bit harsh.

 

[LabZero]

Thanks @Lucent Warrior for this necessary post and for having mentioned me: it is a pride for me to be part of the Malware Hub Testing Team!
And we are also proud of having improved Hub with the correct guidelines.

Sure, our tests are quite simple, but they follow the basic guidelines of the malware test.
You couldn't, of course, follow AMTSO guidelines in comprehensive and extended way, for obvious reasons, because not all of the tester guys are graduates in computer security or malware analysts.
Our guys are skilled, honest and determined to achieve a great goal for a fully independent result.
I am fully convinced of having reached this goal, even without the official approval of someone, we do not need approvals to continue this work because the big passion allows us to work hours and days on a test or analysis to be sure that our work is more professional and correct as possible.
When I was Hub Mod. sometime I was going to edit an incorrect or incomplete post, but now our guys are working in autonomy in "autofixing mode", that's the awesome milestone!
Of course, it is always possible to improve, mainly thanks to constructive criticism, no one has said to the contrary.
How many versions through an antivirus or a software before it will be complete, stable and functional ? Or are we all born masters?
I also agree, if someone suggest improvements, I'm listening too.

Thanks everyone.

 

[hjlbx]

Of course the typical MT Hub tester wants to do testing that provides accurate, meaningful results.

<- I say that with fervor - just look into my eyes.

Enlarge the avatar to get a good gander...

 

[Ana_Filiz]

Our guys are skilled, honest and determined to achieve a great goal for a fully independent result.

Very well said, thank you for your efforts! As an observer of your work, I would like to add a small and innocent suggestion: if it is possible to test, the way you do and with the rules you just have, more AVs: my humble suggestion: Dr. Web , Eset (I`m not so sure that you already work with this AV), Bitdefender, Avira and Trend Micro. Would it be that possible? At least with first option. Instead of changing the methodology better add some new AVs on the grill, it will be more interesting and joyfull so the time will be efficiently spent.

 

[hjlbx]

Very well said, thank you for your efforts! As an observer of your work, I would like to add a small and innocent suggestion: if it is possible to test, the way you do and with the rules you just have, more AVs: my humble suggestion: Dr. Web , Eset (I`m not so sure that you already work with this AV), Bitdefender, Avira and Trend Micro. Would it be that possible? At least with first option.

My observation is that very few people test Dr Web products for posting in the MT MH.