Hot Take Proper malware testing methodology

[Trident]

Did you ask AI how many people were misled? It's a very 'cheap' argument and it's just your opinion. Till now I haven't seen any proof that amateur testers cause much harm.

The tests are rather nerdy, majority of people do not care about them and as you see, and majority of test watchers reduce the results to “good/not good”.

It doesn’t cause anymore damage or harm then completely unarmed and ignorant statements like:
-Bitdefender is BugDefender
-McAfee is a Virus
-Norton is the heaviest and is sh*t
-Avira is good
-Comodo is the best

These statements have been circulating various platforms with very little evidence to support them for years. People read such statements from an anonymous user and if the style of writing looks slightly intelligent and masculine, they rush to uninstall the first 3, installing one of the latter.

No one is obliged to save users from their own bias.

 

[Parkinsond]

The tests are rather nerdy, majority of people do not care about them and as you see, and majority of test watchers reduce the results to “good/not good”.

It doesn’t cause anymore damage or harm then completely unarmed and ignorant statements like:
-Bitdefender is BugDefender
-McAfee is a Virus
-Norton is the heaviest and is sh*t
-Avira is good
-Comodo is the best

These statements have been circulating various platforms with very little evidence to support them for years. People read such statements from an anonymous user and if the style of writing looks slightly intelligent and masculine, they rush to uninstall the first 3, installing one of the latter.

No one is obliged to save users from their own bias.

+ MD is trash

 

[BSONE]

I think that all folks should do a hardware reset, i.e. press that magic button in their head for 10 seconds and breathe in and out slowly for awhile.

 

[Divine_Barakah]

Personally, I see this and this thread as an attack on me and my work.
Why do I say that? You’ve literally commented on almost all of my posts !

I don’t know what the point is—probably to get me to stop producing ?
Sorry, but I’m not Av-Comparative, Av-Tests, or any of the others !
I don’t have 900 virtual machines or 900 testers. I’m on my own.
You’ve criticized my method, but you’ve never asked how I go about it.
You claim that my tests aren’t a real-world scenario, but you don’t prove it.
Because they are. Prove to me that no one can fall victim to a malicious script.
Yes, a user will never have 100 or 200 pieces of malware on their desktop; I’ve always said this is an extreme-scenario test where I push the solution to its absolute limits.
Anyway, I admit I’m pretty disappointed....

genuinely sorry that my comments came across as a personal attack, Shadowra. That was absolutely not my intention, and I sincerely apologise if it felt that way. As I told you earlier, I deeply respect the immense time, effort, and care you put into creating content for this community. Nobody wants you to stop producing these threads.

My critique is purely directed at the scientific methodology, not at you as a person or a creator. Testing security architecture is incredibly complex, and my goal was simply to advocate for testing standards that protect everyday users from drawing the wrong concluaiona.

Implementing proper state isolation (reverting to a clean snapshot after a sample) or testing the web layer doesn't require 900 VMs or a corporate lab. It can be done on a single machine.

I have shared links to how professional labs conduct real-world tests and how they choose the samples and how they process and record the results.

 

[Divine_Barakah]

I am breaking my silence to respond this this thread… I am guessing everyone has now noticed that I’ve got very little interest to anything on this forum.

Firstly, Shadowra tests never use terms like “inferior” or “superior”, better/worse and so on. Every test includes recommendations such as “Recommended for light browsing only” or “not recommended”.

Terms like “inferior” are people’s own interpretation—when you yourself “clock” that the product is “inferior”, not sure what @Shadowra can do for you.
Users want to minimise and reduce everything to “inferior” and “not inferior” because this satisfies their simple consumer minds, and that’s what they are used to get from big corps daily.

The tests themselves do not imply that.

Secondly, it’s important to note that F-Secure which seems to have caused this entire debacle is nothing special. The company is trying hard to regain and reclaim previous leadership positions but the truth is that even their OEM partnerships are disappearing like smoke.
Whether this makes for an “inferior” product, everyone can decide for themselves, depending on their needs.

Thirdly, there are those people here that “rotate” solutions quicker than a Dyson fan. This is not in any way, shape of form stimulated by Shadowra or anyone else—it is a personal cure for boredom for some, for others it is a genuine trial-and-error sort of method to discover what exactly they need.
Some of these people have been “rotating” solutions long before Shadowra even signs up to this forum. A different sort of test will not stop them.

Last but not least, for my tests, real time protection has never ever been disabled even for a second and solutions have never been disconnected.

I do not perform multitudes of tests, when I do, I ensure they are accurate and entertaining.

Hi Trident. Semantics semantics and the result is one. It does not make a difference for a user to see "0", " Not recommended" Or "mediocre".


Regarding F-Secure, a company's commercial business partnerships have absolutely zero mathematical relevance to whether a specific sandbox test was valid. Whether F-Secure is winning or losing corporate contracts doesn't change the fact that running un-vetted, un-flushed malware packs sequentially on a single operating system session violates basic testing hygiene. It's a classic tactic to shift the focus from the testing process to the brand's market reputation. I criticised the flawed methodologu, provided links from professional labs and I offered suggestions. I am not defending X product, I criticised the process which includes every single prpduct tested.

While a small hardcore group of forum hobbyists (I am one of them) do rotate software for entertainment, it ignores the vast majority of silent readers. Regular users visit security forums seeking objective advice. When they see a video showing their current security suite succumbing to an artificial system collapse, it causes real anxiety and drives unnecessary software hopping.

Thank you for your "entertaining" Tests. My critique also applies for your tests for the very same reasons I mentioned. But I appreciate the entertainment tbh.

 

[Divine_Barakah]

@Divine_Barakah - 'One' of my opinions here is put AI on one side & go also by personal experiences, having recently used Panda (bought a license) it really is a piece of junk & needs a total rewrite, Trident did that mention last year sometime that it wasn't good but as I have masochistic tendency I did try it, its unstable & has multiple issues which may not show up unless you actually use it, (as a human) I have no axe to grind but AI does not use install or use these programs, I have, I would also say the same for Webroot which is flawed I feel totally, that's empirical evidence from my chair. F-Secure was IMHO an amazing product years ago its just not now, that is why I no longer use it.

IMHO I do trust Shadowra's testing & I know its unbiased & it is a great pity to pull a persons work down done for free even if its entertaining, but it has great value, if you don't like the methodology fine but its one aspect & often does show flaws in a program when pushed. I'm just a tinkerer I know that.

As say here in jest 'who has changed your batteries'?

​

Thanks for sharing this, my friend, but your reply completely misses the point of this thread. I am NOT defending X product (I use what I want when I want). It is not about the product tested, but rather how it is tested (the process).

 

[Trident]

Thank you for your "entertaining" Tests. My critique also applies for your tests for the very same reasons I mentioned. But I appreciate the entertainment tbh.

Wow, that’s a change of style and heart over there.

“Succumbing to an artificial system collapse”.
What a language.

To which tests btw this critic applies, because it has been circa 2 years since I’ve published any…?

I have shared links to how professional labs conduct real-world tests and how they choose the samples and how they process and record the results.

Well, when I go the official Tesla repair shop, they offer me to sit down, have a coffee, they even offer me a ride back home.
This doesn’t mean that if I bring my car down the road to the Kwik Fit, they won’t be able to service it.
It’s 2 different transport methods that lead the same destination.
One is your local bus, the other one is a Uber Pool taxi.

The official tests are not any less flawed.

 

[Divine_Barakah]

Did you ask AI how many people were misled? It's a very 'cheap' argument and it's just your opinion. Till now I haven't seen any proof that amateur testers cause much harm.

You're entitled to your opinion, my friend. You're defending the flawed, unscientific test methodologu.

I wrote a disclaimer and explicitly said what I asked Gemini. Using that to insult me is a sign of technical defeat. I provided data, links and supported my argument.

Again feel free to disregard my thread, that's your constitutional right

 

[Divine_Barakah]

,Wow, that’s a change of style and heart over there.

“Succumbing to an artificial system collapse”.
What a language.

To which tests btw this critic applies, because it has been circa 2 years since I’ve published any…?

Nice defense mechanism. You said semantics does not matter, right?

Whether you published a test yesterday, or two years ago does not matter. The methodology is the same, the process is the same.

Before we continue this discussion, you're a respected developer and I do respect you and your work, so I hope you do not take any of this personally.

You literally stepped into this thread to defend the flawed methodology despite me posting links from professional lab and how they conduct their testinf. By defending this metho, my critique then applies to your stance.

 

[Trident]

Nice defense mechanism. You said semantics does not matter, right?

Whether you published a test yesterday, or two years ago does not matter. The methodology is the same, the process is the same.

Before we continue this discussion, you're a respected developer and I do respect you and your work, so I hope you do not take any of this personally.

You literally stepped into this thread to defend the flawed methodology despite me posting links from professional lab and how they conduct their testinf. By defending this metho, my critique then applies to your stance.

I am not defending any methodology, my views on malware tests have been made clear many times on many threads.

The “official” tests are no rainy daisy either.
The AMTSO hasn’t done anything above the bare minimum for over 10 years now.
It is exactly these “official” tests that cause a install/uninstall wave.

My methodology as seen on the last McAfee test was anything but flawed.

I am all about respecting the work that people do, as well as the very same people doing it.

If you don’t like the dates in the food hall you are going through, you can just walk. You don’t need to stop and tell the seller that you don’t like them.

 

[Divine_Barakah]

I am not defending any methodology, my views on malware tests have been made clear many times on many threads.

The “official” tests are no rainy daisy either.
The AMTSO hasn’t done anything above the bare minimum for over 10 years now.

My methodology as seen on the last McAfee test was anything but flawed.

I am all about respecting the work that people do, as well as the very same people doing it.

If you don’t like the dates in the food hall you are going through, you can just walk. You don’t need to stop and tell the seller that you don’t like them.

It seems the technical debate has ended?

You're not defending any methodology but your "McAffee test was anything but flawed"?

And regarding ALSO, when your tests do not comply with the standards it ja easier for you to declare the frameworks themselves are irrelevant.

Finally, your example about dates completely misrepresents how public forums, like MT, work. A test published in a public forums is not private stall in a marketplace where an independent vendor sells their producta. When tests and data are published, it affects the perception of and drives user anxiety.

 

[Jan Willy]

Again feel free to disregard my thread

No sir, I don't disregard your thread. The only thing is we don't share an opinion. I don't doubt your good intentions.

 

[Divine_Barakah]

Perhaps people should do their due diligence before trusting content like this…?

Thank you for fully conceding my entire argument.

 

[Sorrento]

I personally don't consider tests from @Shadowra @Trident and others here to be misleading and believe they give a good representation of how antiviruses perform. But I do accept that the testing conditions are not ideal. However, the issue with doing more "professional" testing, is the large amount of time and effort it would take.

Reading post# the testing done in actuality would give little or no info as to how the AV's perform in real life situations, it is purely analytical, I feel users comments on this thread give their personal experiences using these products in real life, with other programs running, experiences with support or lack of, licensing issues, cost of renewal, tests done by Shadowra for example show the AV's on a PC, we get to see the GUI, whether extreme conditions cause the program to lock the PC, memory, etc. etc. etc. you do not get that from labs, using PC's since the 286 tells me to if anything take the labs reports as a small part a AV's capability, most give very good results anyway.

As a hi-fi nut all my life as my father was, 40 years ago or more testers looked for THD (harmonic distortion) & RMS power into a 8 Ohm resister, little if any time was spent actually listening to music which is really the point, played by the aforesaid gear, however in time some realised for example a loudspeaker is not a resister its also a motor & & in real life situations amps did very sound different, (this took many years) low powered amps can sound great, now the power output, frequency response etc can be somewhat irrelevant, how it sounds in your home is what matters & maybe how your gear looks, whether my wife likes the looks, of course spec matters but that is not the be & end all as in all things in life - Not the best analogy but more to testing than post one though relevant.

 

[Divine_Barakah]

Reading post# the testing done in actuality would give little or no info as to how the AV's perform in real life situations, it is purely analytical, I feel users comments on this thread give their personal experiences using these products in real life, with other programs running, experiences with support or lack of, licensing issues, cost of renewal, tests done by Shadowra for example show the AV's on a PC, we get to see the GUI, whether extreme conditions cause the program to lock the PC, memory, etc. etc. etc. you do not get that from labs, using PC's since the 286 tells me to if anything take the labs reports as a small part a AV's capability, most give very good results anyway.

As a hi-fi nut all my life as my father was, 40 years ago or more testers looked for THD (harmonic distortion) & RMS power into a 8 Ohm resister, little if any time was spent actually listening to music, in time some realised a loudspeaker is not a resister its also a motor & & in real life situations amps did very sound different, low powered amps can sound great, now the power output, frequency response etc can be somewhat irrelevant, how it sounds in your home is what matters & maybe how your gear looks, spec matters but that is not the be & end all - Not the best analogy but more to testing than post one.

Modern endpoint security doesn't operate in a vacuum. It relies on global telemetry loops, behavioral heuristics, and cloud reputation layers that fire in milliseconds. Evaluating a modern cloud-first engine using an isolated 286-era testing philosophy (dumping raw files on a disk to see if a local scanner catches them) completely misses the paradigm shift in modern threat execution.

 

[Sorrento]

I'm not arguing with AI, had enough of that a while back, if you feel that all there is to the situation I exit the thread & similar

 

[Divine_Barakah]

I'm not arguing with AI, had enough of that a while back, if you feel that all there is to the situation I exit the thread & similar

AI didn't invent the concept of behavioral heuristics or the other layers of protection. The science is the science. By refusing to engage with the actual computer science and instead blaming the tool used to format the argument, you are openly admitting that you have absolutely zero technical rebuttal.

 

[Trident]

I am not sure what’s going on @Divine_Barakah, you are one of the few users I respect on here since your account was The Cog in The Wheel and we spoke about Trend Micro.

Today your posts make me think if your account actually got hijacked or something?

You sound awfully like another user with whom you had a conflict, then again, if you are both using Gemini then no wonder.

I am genuinely bamboozled and concerned.

 

[Jonny Quest]

+ MD is trash

LOL, yes, as how many millions of users worldwide use that as their default AV, and we don't hear about mass outbreaks of malware on Windows systems, do we.

 

[Parkinsond]

LOL, yes, as how many millions of users worldwide use that as their default AV, and we don't hear about mass outbreaks of malware on Windows systems, do we.

All has pros and cons; no one tool fits all the users; every setup fits better with one AV than the remaining.
It has to be individualized.