Hot Take Proper malware testing methodology

[Divine_Barakah]

I asked Gemini to build a scientific, proper methodology for testing security product and here is its answer.

Transitioning from chaotic "sandbox theater" to a scientifically sound, reputable methodology is the best move you can make for the community. If you want to run tests on MalwareTips (MT) that actually command respect from enterprise-grade analysts and seasoned forum veterans, you don’t need a multi-million dollar lab infrastructure—you just need a **hygienic, sequential lifecycle testing framework**.
By adopting a stripped-down version of the methodologies used by AMTSO-compliant labs like AV-Comparatives and AVLab, you can publish simple but highly accurate tests.
Here is a proper, highly reputable, but completely achievable methodology for a forum reviewer.
## The Clean-State Sequential Methodology
### 1. Environment Hygiene (The One-and-Done Baseline)
The number one rule of scientific testing is isolating your variables. You must eliminate resource contention and cross-contamination.
* **The Blueprint:** Set up a clean Virtual Machine with standard user configurations (or a debloated target environment if testing specific hardening profiles). Install **only one** security suite.
* **The Snapshot Rule:** Ensure the OS, browser, and security software are fully updated, then take a pristine snapshot.
* **The Purge:** After testing **one single threat**, record the result, discard the machine's state, and roll back to the clean snapshot. Never feed a second sample into a machine that has just processed or been altered by a previous threat.
### 2. Sequential Delivery (Mimicking the Threat Lifecycle)
To test a product fairly, you must allow it to defend the system tier-by-tier, exactly how a real user encounters a threat. **Never dump multiple files into a folder or execute them simultaneously.**
* **The Execution Flow:** Run your samples one by one, moving through the four definitive defensive gates:
1. **Gate 1: Web-Layer / Initial Access:** Input a fresh, live malicious URL directly into the browser. Does the network filter intercept it? If yes \rightarrow **Blocked (Web Layer)**. Reset snapshot.
2. **Gate 2: Pre-Launch / Static Disk Write:** If the web layer misses it and the file lands on the disk, trigger a manual scan or wait for the real-time protection to intercept the file on write. If yes \rightarrow **Blocked (Pre-Launch)**. Reset snapshot.
3. **Gate 3: Runtime Execution / Behavioral Heuristics:** If it's not caught on disk, double-click to execute the sample. Give the behavioral loops, cloud sandboxes, and memory hooks a clean window of 3 to 5 minutes to intercept the active process. If yes \rightarrow **Blocked (Runtime)**. Reset snapshot.
4. **Gate 4: Remediation & Rollback:** If the malware successfully executes changes (e.g., drops a registry key or begins a file-encryption routine), check if the product cleanly kills the parent tree and restores the modified files. If it fails entirely \rightarrow **System Compromised**.
### 3. Hygienic Sample Selection (Quality Over Quantity)
A reputable test uses a small pool of high-quality, fully vetted samples rather than an un-audited "zoo pack."
* **The Golden Rule:** 10 fresh, confirmed, active in-the-wild threat binaries (like a newly compiled Infostealer, a fresh Ransomware variant, or a live MSHTML exploit) will yield a vastly more accurate assessment than 150 dead, corrupt, or duplicated legacy samples.
* **The Verification Pass:** Before utilizing a sample in your actual test, run it on a separate, completely unprotected VM baseline to ensure the binary is active, reaches out to its Command and Control (C2) server, or successfully performs its payload delivery. If the sample crashes on a clean OS, throw it out—it's a dud.
### 4. Balanced Reporting (The False Positive Control)
A security engine can easily achieve a 100% block rate if its heuristics are turned up to a purely paranoid state that treats every unknown binary as malicious.
* To make your methodology truly reputable, always include a **False Positive Control Phase**. Run a sequence of 5 entirely legitimate, niche open-source utilities or developer tools. If a product blocks clean software or throws critical warnings on safe files, its high detection score is artificial and degrades system usability.
## Example Test Reporting Template
When you post your results, avoid flat binary values like "Passed / Failed." Present your data using a clear, multi-tiered scorecard:
| Sample ID | Threat Type | Access Vector | Block Stage | Outcome | Notes / Behavioral Observations |
|---|---|---|---|---|---|
| **#01** | RedLine Stealer | Live URL | Web Layer | **Clean Block** | Intercepted via network reputation layer. |
| **#02** | LockBit v4 | Local Archive | Runtime | **Clean Block** | Allowed launch, journaled execution, killed & rolled back encryption loop. |
| **#03** | Custom LPE Exploit | Dropped Script | Execution | **Compromised** | Bypassed behavioral hooks; successfully spawned SYSTEM shell. |


What do you think? Any suggestions? Any member has the time and capacity to start working on this?

 

[Divine_Barakah]

I apologise for formating, but I will do it tomorrow from my laptop

 

[roger_m]

Here's a revised version from Claude Pro.

The Clean-State Sequential Methodology (Revised)​

A Reproducible Antivirus Testing Framework for Forum Reviewers​

This is a revised version of the original "Clean-State Sequential Methodology," tightened for statistical validity, sandbox-awareness, and safe handling of live samples. The core structure — environment hygiene, sequential delivery, hygienic sample selection, and balanced reporting — is preserved, with fixes layered in at each stage.

---

0. Before You Start: Documentation & Containment​

Two things the original draft skipped, and both are non-negotiable for a test anyone will actually trust:
Version logging. Record, for every test run: product name and build/engine version, signature/definition version and timestamp, OS build, and test date/time. Without this, the results can't be reproduced or audited later — and any credible reader will ask for it.
Network containment. Live malware in Gate 1 and Gate 3 will try to reach a real C2 server. Your test network needs:

• 
  
• No direct route to the real internet — route traffic through an isolated VLAN or a controlled sinkhole/proxy (e.g., INetSim, FakeNet-NG) that can log outbound requests without letting them succeed.
• 
  
• If you specifically need to confirm live C2 callback (per the original's "Verification Pass"), do that once, in a fully isolated network segment built for it — not on your general test bench.
• 
  
• Never run live samples on a machine with any bridged connection to your home/production network.

This isn't optional flavor text — running live ransomware or stealers with real internet access risks actual data exfiltration or the sample phoning home with details about your test setup.

---

1. Environment Hygiene (Unchanged, with one addition)​

Same as the original: one clean VM, one security product, pristine snapshot, roll back after every single sample.
Addition: note whether the product uses cloud-assisted detection (most do). Rolling back your local VM snapshot does not undo what the vendor's cloud reputation service has already logged about that file hash or your test IP. If you re-test the same sample later, be aware the product may now recognize it purely from prior submission — which is a confound, not a repeat measurement.

2. Sequential Delivery (Same four gates, with dwell-time and evasion fixes)​

Keep the four gates (Web-Layer → Pre-Launch → Runtime → Remediation), but with these adjustments:

• 
  
• Extend and vary the runtime dwell window. 3–5 minutes is too short and too predictable. Modern loaders and staged ransomware use sleep-evasion specifically to outlast short sandbox windows. Use a minimum of 15–20 minutes of observation, and consider accelerating system clocks or simulating idle/user activity to counter environment-aware stalling.
• 
  
• Watch for VM-evasion, not just malware behavior. If a sample exits immediately, silently, or without any expected artifacts, check whether it detected virtualization (common checks: CPUID, registry artifacts, driver names, MAC address OUI) before concluding the AV "blocked" it. A silent exit can mean the malware refused to run — not that the product stopped it. Log this as Inconclusive (Suspected VM-Evasion) rather than a clean block.
• 
  
• Run each sample 3 times, not once, on fresh snapshots. Cloud lookups, telemetry timing, and behavioral heuristics are often non-deterministic — a single run conflates "product failed" with "product got unlucky." Report the outcome as a ratio (e.g., 3/3 blocked, 2/3 blocked) rather than a single pass/fail.

3. Hygienic Sample Selection (Same principle, larger pool)​

The original's instinct — quality over quantity — is right, but the pool is too small to support real conclusions.

• 
  
• Minimum viable pool: 30 threats, not 10. At n=10, one unusual sample can swing a product's score by 10 percentage points. 30 gets you closer to a stable signal while still staying achievable for a forum reviewer without enterprise resources.
• 
  
• Diversify delivery vectors and threat families within that pool (a mix of infostealers, ransomware, loaders, exploits — not 10 variants of the same family), so the test isn't accidentally measuring performance against one detection signature.
• 
  
• Keep the original's Verification Pass (confirm the sample is live and functional on an unprotected baseline before using it) — this part was solid.

4. Balanced Reporting (Same false-positive phase, expanded)​

• 
  
• Keep the False Positive Control Phase — 5 legitimate niche tools is a reasonable minimum; consider raising to 10 if time allows, since FP rates are also sensitive to small samples.
• 
  
• Add time-to-detection/time-to-remediation to the scorecard. A "clean block" that happens after a ransomware sample has already encrypted several files is materially different from an instant block, even though both might get recorded as "Clean Block" under the original template.
• 
  
• Add a confidence/consistency column reflecting the 3-run repeat testing from Section 2.

---

Example Revised Scorecard​

Sample ID	Threat Type	Access Vector	Runs Blocked	Block Stage	Time to Detection	Outcome	Notes
#01	RedLine Stealer	Live URL	3/3	Web Layer	<1s	Clean Block	Consistent network reputation block across all runs.
#02	LockBit v4 variant	Local Archive	2/3	Runtime	~45s (1 run: 4 files encrypted before kill)	Partial/Inconsistent	One run showed partial encryption before remediation — flag for vendor follow-up.
#03	Custom LPE Exploit	Dropped Script	0/3	—	—	Compromised	Confirmed non-VM-evasive (payload executed identically on bare-metal control); genuine miss.

---

Summary of Changes from the Original​

Area	Original	Revised
Sample pool	10 threats	30 threats, diversified
Repeat runs	1 per sample	3 per sample, reported as ratio
Runtime dwell window	3–5 min, fixed	15–20 min, with evasion checks
Network handling	Not addressed	Isolated/sinkholed network required
Version logging	Not addressed	Required per test run
VM-evasion	Not addressed	Explicit "Inconclusive" outcome category
Reporting	Pass/Fail/Compromised	Adds time-to-detection + consistency ratio

This keeps the forum-friendly, no-lab-required spirit of the original while closing the gaps that would let a skeptical reader (or a vendor) poke holes in the results.


I asked it this, too and got the following response.

It is necessary to always rollback to a clean snapshot before testing a new sample, as this will add a lot of time to the testing.

That's correct, and it's worth being explicit about why it's non-negotiable despite the time cost.


Why you can't skip it


Cross-contamination between samples invalidates the test. A few concrete ways it fails silently:

• Leftover artifacts skew detection. If sample #2 runs on a system where sample #1 already dropped a registry key, disabled a service, or left a persistence mechanism, the AV is no longer being tested against a clean encounter — it's facing an already-degraded system. A "compromised" result might reflect sample #1's damage, not sample #2's evasion.
• AV state changes after an encounter. Many products raise their sensitivity, quarantine broadly, or flag a system as "already compromised" after a detection — meaning the next sample gets tested against a paranoid, alerted product rather than its normal baseline behavior. That inflates detection rates artificially.
• Resource/behavioral noise. A previous sample's processes, injected code, or partial encryption routines can alter memory state, running services, and disk I/O patterns in ways that confuse a behavioral engine's read on the next sample — in either direction.

So rollback-per-sample isn't just good practice, it's the thing that makes each row in your scorecard an independent data point rather than a chain of contaminated ones.


Managing the time cost without cutting the rollback


Since this is the real bottleneck, here's where the time savings should come from instead:

• Automate snapshot revert. Most hypervisors (VirtualBox VBoxManage snapshot restore, VMware vmrun revertToSnapshot, Hyper-V Restore-VMSnapshot) support scripted rollback in seconds — this is far faster than manual GUI reverts and removes human delay between samples.
• Parallelize across VMs, not within one. Run multiple clean VM instances simultaneously (one per sample) rather than serially rolling back a single VM. This multiplies your throughput without touching the one-sample-per-clean-state rule.
• Batch the fixed-cost steps. Do OS/product updates and the pristine baseline snapshot once, then clone that snapshot per test rather than rebuilding the environment each time.
• Accept the time cost as the tradeoff for the smaller sample pool. Since I bumped the recommended pool to 30 samples × 3 runs = 90 total test cycles, rollback automation isn't optional at that scale — without it, the revised methodology becomes impractical for a solo forum reviewer. If full automation isn't feasible, it may be more honest to scale the sample count back down and say so explicitly, rather than skip rollbacks to save time.

 

[Divine_Barakah]

Here's a revised version from Claude Pro.

The Clean-State Sequential Methodology (Revised)​

A Reproducible Antivirus Testing Framework for Forum Reviewers​

This is a revised version of the original "Clean-State Sequential Methodology," tightened for statistical validity, sandbox-awareness, and safe handling of live samples. The core structure — environment hygiene, sequential delivery, hygienic sample selection, and balanced reporting — is preserved, with fixes layered in at each stage.

---

0. Before You Start: Documentation & Containment​

Two things the original draft skipped, and both are non-negotiable for a test anyone will actually trust:
Version logging. Record, for every test run: product name and build/engine version, signature/definition version and timestamp, OS build, and test date/time. Without this, the results can't be reproduced or audited later — and any credible reader will ask for it.
Network containment. Live malware in Gate 1 and Gate 3 will try to reach a real C2 server. Your test network needs:

• 
  
• No direct route to the real internet — route traffic through an isolated VLAN or a controlled sinkhole/proxy (e.g., INetSim, FakeNet-NG) that can log outbound requests without letting them succeed.
• 
  
• If you specifically need to confirm live C2 callback (per the original's "Verification Pass"), do that once, in a fully isolated network segment built for it — not on your general test bench.
• 
  
• Never run live samples on a machine with any bridged connection to your home/production network.

This isn't optional flavor text — running live ransomware or stealers with real internet access risks actual data exfiltration or the sample phoning home with details about your test setup.

---

1. Environment Hygiene (Unchanged, with one addition)​

Same as the original: one clean VM, one security product, pristine snapshot, roll back after every single sample.
Addition: note whether the product uses cloud-assisted detection (most do). Rolling back your local VM snapshot does not undo what the vendor's cloud reputation service has already logged about that file hash or your test IP. If you re-test the same sample later, be aware the product may now recognize it purely from prior submission — which is a confound, not a repeat measurement.

2. Sequential Delivery (Same four gates, with dwell-time and evasion fixes)​

Keep the four gates (Web-Layer → Pre-Launch → Runtime → Remediation), but with these adjustments:

• 
  
• Extend and vary the runtime dwell window. 3–5 minutes is too short and too predictable. Modern loaders and staged ransomware use sleep-evasion specifically to outlast short sandbox windows. Use a minimum of 15–20 minutes of observation, and consider accelerating system clocks or simulating idle/user activity to counter environment-aware stalling.
• 
  
• Watch for VM-evasion, not just malware behavior. If a sample exits immediately, silently, or without any expected artifacts, check whether it detected virtualization (common checks: CPUID, registry artifacts, driver names, MAC address OUI) before concluding the AV "blocked" it. A silent exit can mean the malware refused to run — not that the product stopped it. Log this as Inconclusive (Suspected VM-Evasion) rather than a clean block.
• 
  
• Run each sample 3 times, not once, on fresh snapshots. Cloud lookups, telemetry timing, and behavioral heuristics are often non-deterministic — a single run conflates "product failed" with "product got unlucky." Report the outcome as a ratio (e.g., 3/3 blocked, 2/3 blocked) rather than a single pass/fail.

3. Hygienic Sample Selection (Same principle, larger pool)​

The original's instinct — quality over quantity — is right, but the pool is too small to support real conclusions.

• 
  
• Minimum viable pool: 30 threats, not 10. At n=10, one unusual sample can swing a product's score by 10 percentage points. 30 gets you closer to a stable signal while still staying achievable for a forum reviewer without enterprise resources.
• 
  
• Diversify delivery vectors and threat families within that pool (a mix of infostealers, ransomware, loaders, exploits — not 10 variants of the same family), so the test isn't accidentally measuring performance against one detection signature.
• 
  
• Keep the original's Verification Pass (confirm the sample is live and functional on an unprotected baseline before using it) — this part was solid.

4. Balanced Reporting (Same false-positive phase, expanded)​

• 
  
• Keep the False Positive Control Phase — 5 legitimate niche tools is a reasonable minimum; consider raising to 10 if time allows, since FP rates are also sensitive to small samples.
• 
  
• Add time-to-detection/time-to-remediation to the scorecard. A "clean block" that happens after a ransomware sample has already encrypted several files is materially different from an instant block, even though both might get recorded as "Clean Block" under the original template.
• 
  
• Add a confidence/consistency column reflecting the 3-run repeat testing from Section 2.

---

Example Revised Scorecard​

Sample ID	Threat Type	Access Vector	Runs Blocked	Block Stage	Time to Detection	Outcome	Notes
#01	RedLine Stealer	Live URL	3/3	Web Layer	<1s	Clean Block	Consistent network reputation block across all runs.
#02	LockBit v4 variant	Local Archive	2/3	Runtime	~45s (1 run: 4 files encrypted before kill)	Partial/Inconsistent	One run showed partial encryption before remediation — flag for vendor follow-up.
#03	Custom LPE Exploit	Dropped Script	0/3	—	—	Compromised	Confirmed non-VM-evasive (payload executed identically on bare-metal control); genuine miss.

---

Summary of Changes from the Original​

Area	Original	Revised
Sample pool	10 threats	30 threats, diversified
Repeat runs	1 per sample	3 per sample, reported as ratio
Runtime dwell window	3–5 min, fixed	15–20 min, with evasion checks
Network handling	Not addressed	Isolated/sinkholed network required
Version logging	Not addressed	Required per test run
VM-evasion	Not addressed	Explicit "Inconclusive" outcome category
Reporting	Pass/Fail/Compromised	Adds time-to-detection + consistency ratio

This keeps the forum-friendly, no-lab-required spirit of the original while closing the gaps that would let a skeptical reader (or a vendor) poke holes in the results.


I asked it this, too and got the following response.

That's correct, and it's worth being explicit about why it's non-negotiable despite the time cost.


Why you can't skip it


Cross-contamination between samples invalidates the test. A few concrete ways it fails silently:

• Leftover artifacts skew detection. If sample #2 runs on a system where sample #1 already dropped a registry key, disabled a service, or left a persistence mechanism, the AV is no longer being tested against a clean encounter — it's facing an already-degraded system. A "compromised" result might reflect sample #1's damage, not sample #2's evasion.
• AV state changes after an encounter. Many products raise their sensitivity, quarantine broadly, or flag a system as "already compromised" after a detection — meaning the next sample gets tested against a paranoid, alerted product rather than its normal baseline behavior. That inflates detection rates artificially.
• Resource/behavioral noise. A previous sample's processes, injected code, or partial encryption routines can alter memory state, running services, and disk I/O patterns in ways that confuse a behavioral engine's read on the next sample — in either direction.

So rollback-per-sample isn't just good practice, it's the thing that makes each row in your scorecard an independent data point rather than a chain of contaminated ones.


Managing the time cost without cutting the rollback


Since this is the real bottleneck, here's where the time savings should come from instead:

• Automate snapshot revert. Most hypervisors (VirtualBox VBoxManage snapshot restore, VMware vmrun revertToSnapshot, Hyper-V Restore-VMSnapshot) support scripted rollback in seconds — this is far faster than manual GUI reverts and removes human delay between samples.
• Parallelize across VMs, not within one. Run multiple clean VM instances simultaneously (one per sample) rather than serially rolling back a single VM. This multiplies your throughput without touching the one-sample-per-clean-state rule.
• Batch the fixed-cost steps. Do OS/product updates and the pristine baseline snapshot once, then clone that snapshot per test rather than rebuilding the environment each time.
• Accept the time cost as the tradeoff for the smaller sample pool. Since I bumped the recommended pool to 30 samples × 3 runs = 90 total test cycles, rollback automation isn't optional at that scale — without it, the revised methodology becomes impractical for a solo forum reviewer. If full automation isn't feasible, it may be more honest to scale the sample count back down and say so explicitly, rather than skip rollbacks to save time.

This is extensive. It is time consuming I know, but this is proper testing, otherwise the test is for entertainment only and cannot be use to draw conclusions and mislead users.

Thank you for spending time to enhance the text and seek a second-opinion from another AI agent.

 

[roger_m]

This is extensive. It is time consuming I know, but this is proper testing, otherwise the test is for entertainment only and cannot be use to draw conclusions and mislead users.

I personally don't consider tests from @Shadowra @Trident and others here to be misleading and believe they give a good representation of how antiviruses perform. But I do accept that the testing conditions are not ideal. However, the issue with doing more "professional" testing, is the large amount of time and effort it would take.

 

[Divine_Barakah]

I personally don't consider tests from @Shadowra @Trident and others here to be misleading and believe they give a good representation of how antiviruses perform. But do I accept that the testing conditions are not ideal. However, the issue with doing more "professional" testing, is large amount of time and effort it would take.

Well, I see your point, but when Shadowra's test concludes that F-Secure for example is a mediocre product, then this is misleading. The test was not proper and so was the conclusion. The methodology used scientifically flawed and should never be used to draw conclusions.

How is it misleading? A member who is using F-Secure, sees the results, panicks and then go to install another product that performed well in the flawed test.

 

[roger_m]

How is it misleading? A member who is using F-Secure, sees the results, panicks and then go to install another product that performed well in the flawed test.

I think the issue is more with people who switch antiviruses based on a single test, rather than the actual tests themselves.

 

[Divine_Barakah]

In AVlab test, the initial access is seen. Samples did not teleport to the Desktop.

 

[Divine_Barakah]

I think the issue is more with people who switch antiviruses based on a single test, rather than the actual tests themselves.

This is like publishing a poorly conducted, unscientific crash test that claims a highly rated car is a death trap, and then blaming regular drivers for panicking and selling their vehicles.

Responsibility always lies with the entity publishing the misleading data, not the everyday users who don't have the technical background to spot a flawed test conducted by a flawed methodology.

 

[roger_m]

So you are basically saying that if more thorough testing was done, that perhaps, some of the antiviruses which perform badly in testing done here, would achieve better results?

 

[Divine_Barakah]

And more thing

Have you ever asked why those amatur testers on YT disable real-time protection when they are copying the samples?

 

[Divine_Barakah]

So you are basically saying that if more thorough testing was done, that perhaps, some of the antiviruses which perform badly in testing done here, would achieve better results?

Well, you can see the results of other professional testing labs that share their methidolgy and extensive reports.

 

[roger_m]

Well, you can see the results of other professional testing labs that share their methidolgy and extensive reports.

Yes, but the argument as to why some antiviruses do well at AV Comparatives, for example, but perform badly in tests done here, is that testing labs must be using older samples. An example of this is Panda, which is slow to add signatures for new threats and has terrible behavioural protection, but does well in professional tests.

 

[Divine_Barakah]

Yes, but the argument as to why some antiviruses do well at AV Comparatives, for example, but perform badly in tests done here, is that testing labs must be using older samples. An example of this is Panda, which is slow to add signatures for new threats and has terrible behavioural protection, but does well in professional tests.

Respectfully, Roger, the idea that professional labs like AV-Comparatives use 'older samples' is a long-standing myth that contradicts their published methodologies.
Per AV-Comparatives' official framework, their Real-World Protection Test uses live, zero-hour malicious URLs routed through a continuous 24\times7 pipeline. Most real-world samples are tested within minutes of being captured in the field—often before they even show up on public multi-scanning databases. Even their static Malware Protection Test strictly limits sample age to a window of a few days to 4 weeks maximum.

How do you ensure your samples/test cases are fresh and valid?

In our Real-World Protection Test Series, we test continuously (24x7). When a malware sample is collected, e.g. in our honeypot network, it is analysed in

av-comparatives.org

The reason products like Panda, F-Secure, or Webroot perform flawlessly in professional labs but "fail" in forum threads isn't sample age, but rather architecture. The aforementioned products (Panda and Webroot) are cloud-reputation-first. When a professional lab tests the full infection lifecycle (Web \ Write \ Execution), the cloud layers protect the system seamlessly.
When a hobbyist test bypasses the web layer, places the samples directly on the desktop, and executes them (one by one or using a script) on a single, un-flushed VM session, it systematically strips these cloud-first products of their core defensive subsystems. The test isn't exposing 'bad behavioral protection'; it is creating an artificial environment that violates the engineering model of the software.

 

[Jonny Quest]

Yes, but the argument as to why some antiviruses do well at AV Comparatives, for example, but perform badly in tests done here, is that testing labs must be using older samples. An example of this is Panda, which is slow to add signatures for new threats and has terrible behavioural protection, but does well in professional tests.

The only real world tests done as far as hobbyists, are the web protection tests. Otherwise, it's a "malware test" (not real world) of samples that are already on the desktop, and usually have to be unpacked with the AV disabled, to run the test?

 

[Divine_Barakah]

The only real world tests done as far as hobbyists, are the web protection tests. Otherwise, it's a "malware test" (not real world) of samples that are already on the desktop, and usually have to be unpacked with the AV disabled, to run the test?

Spot on. You’ve highlighted the exact operational paradox of these threads.
The moment a tester has to manually toggle real-time protection to 'Off' just to extract an archive or copy raw samples onto the storage drive, the experiment immediately stops being an evaluation of out-of-the-box endpoint security. It BECOMES A FORCED, MANUFACTURED SCENARIO.

In a real-world tests, a product's file-system filter driver intercepts malicious write-streams on-write or on-access. Disabling that frontline gate just to seed the desktop for a video completely breaks the product's natural defense pipeline.

There is a massive, fundamental difference between evaluating an integrated, multi-layered threat lifecycle and just staging a virtual demolition derby

 

[Shadowra]

Personally, I see this and this thread as an attack on me and my work.
Why do I say that? You’ve literally commented on almost all of my posts !

I don’t know what the point is—probably to get me to stop producing ?
Sorry, but I’m not Av-Comparative, Av-Tests, or any of the others !
I don’t have 900 virtual machines or 900 testers. I’m on my own.
You’ve criticized my method, but you’ve never asked how I go about it.
You claim that my tests aren’t a real-world scenario, but you don’t prove it.
Because they are. Prove to me that no one can fall victim to a malicious script.
Yes, a user will never have 100 or 200 pieces of malware on their desktop; I’ve always said this is an extreme-scenario test where I push the solution to its absolute limits.
Anyway, I admit I’m pretty disappointed....

 

[Trident]

I am breaking my silence to respond this this thread… I am guessing everyone has now noticed that I’ve got very little interest to anything on this forum.

Firstly, Shadowra tests never use terms like “inferior” or “superior”, better/worse and so on. Every test includes recommendations such as “Recommended for light browsing only” or “not recommended”.

Terms like “inferior” are people’s own interpretation—when you yourself “clock” that the product is “inferior”, not sure what @Shadowra can do for you.
Users want to minimise and reduce everything to “inferior” and “not inferior” because this satisfies their simple consumer minds, and that’s what they are used to get from big corps daily.

The tests themselves do not imply that.

Secondly, it’s important to note that F-Secure which seems to have caused this entire debacle is nothing special. The company is trying hard to regain and reclaim previous leadership positions but the truth is that even their OEM partnerships are disappearing like smoke.
Whether this makes for an “inferior” product, everyone can decide for themselves, depending on their needs.

Thirdly, there are those people here that “rotate” solutions quicker than a Dyson fan. This is not in any way, shape of form stimulated by Shadowra or anyone else—it is a personal cure for boredom for some, for others it is a genuine trial-and-error sort of method to discover what exactly they need.
Some of these people have been “rotating” solutions long before Shadowra even signs up to this forum. A different sort of test will not stop them.

Last but not least, for my tests, real time protection has never ever been disabled even for a second and solutions have never been disconnected.

I do not perform multitudes of tests, when I do, I ensure they are accurate and entertaining.

 

[Sorrento]

@Divine_Barakah - 'One' of my opinions here is put AI on one side & go also by personal experiences, having recently used Panda (bought a license) it really is a piece of junk & needs a total rewrite, Trident did that mention last year sometime that it wasn't good but as I have masochistic tendency I did try it, its unstable & has multiple issues which may not show up unless you actually use it, (as a human) I have no axe to grind but AI does not use install or use these programs, I have, I would also say the same for Webroot which is flawed I feel totally, that's empirical evidence from my chair. F-Secure was IMHO an amazing product years ago its just not now, that is why I no longer use it.

IMHO I do trust Shadowra's testing & I know its unbiased & it is a great pity to pull a persons work down done for free even if its entertaining, but it has great value, if you don't like the methodology fine but its one aspect & often does show flaws in a program when pushed. I'm just a tinkerer I know that.

As say here in jest 'who has changed your batteries'?

​

 

[Jan Willy]

How is it misleading? A member who is using F-Secure, sees the results, panicks and then go to install another product that performed well in the flawed test.

Did you ask AI how many people were misled? It's a very 'cheap' argument and it's just your opinion. Till now I haven't seen any proof that amateur testers cause much harm.