Hot Take Proper malware testing methodology

[Jonny Quest]

All has pros and cons; no one tool fits all the users; every setup fits better with one AV than the remaining.
It has to be individualized.

But we all know this here, no?

 

[Divine_Barakah]

I am not sure what’s going on @Divine_Barakah, you are one of the few users I respect on here since your account was The Cog in The Wheel and we spoke about Trend Micro.

Today your posts make me think if your account actually got hijacked or something?

You sound awfully like another user with whom you had a conflict, then again, if you are both using Gemini then no wonder.

I am genuinely bamboozled and concerned.

Are u talking about Bazang? No my account is not hijacked.

Your respect for me and my respect for you should not be shaken because of a thread and of us expressing our opinions. And I told you explicitly that I respect you and appreciate your work.

Reagrding the thread on test methodology, I have had enough with misinformation, flawed tests and biased and flawed and even inaccurate conclusions.

I created the thread with pure intentions and I am willing to do anything I can, with the help of other members if they are willing, to make improvements to the way products are tested here on MT. It is astonishing that @Shadowra took personally despite the fact that he know that I do appreciate his work. He devotes time and efforts, I just criticised the process.

 

[Divine_Barakah]

@Trident it was The Cog in the Machine not the wheel

 

[Divine_Barakah]

LOL, yes, as how many millions of users worldwide use that as their default AV, and we don't hear about mass outbreaks of malware on Windows systems, do we.

Personally, I have nothing against MD except the fact that MS has forced it on every Windows machine and thus, affecting the choices of other users. Even if you decide to install a 3rd party security product, MD is still there rooted in the system in the same way as Edge.

 

[Parkinsond]

But we all know this here, no?

Yes, we all know, but not all of us stop repeating A-AV is better than B-AV, regardless of the motive to say that.

 

[Shadowra]

It is astonishing that @Shadowra took personally despite the fact that he know that I do appreciate his work. He devotes time and efforts, I just criticised the process.

Unfortunately, didn't you wonder why I took it personally?
Before I came across your thread, you had commented on about 3 or 4 videos with the same message.
I'm sorry, but with such persistence, I took it as an attack—a former member had already done that here.
Especially since my method hasn’t changed since the beginning:

- URL Test:
Selection of malicious links. I used to post phishing links, but those die out very quickly.

- Malicious fake crack: Removed a while ago because the source I was using shut down… I’m currently looking for another one.

- Malware Pack: A selection of recent and older malware to test the lab’s responsiveness (.exe files, yes, but not exclusively—there are also VBS, JS, scripts, exploits, etc.).
Yes, that’s right—I run a scan of the pack. I hate the “gang-bang” effect where everything is executed via a script.
I launch it, wait about 15 to 20 seconds, and then continue.
That’s my method

I was also working on including system penetration tests (using Meterpreter or Cobalt), but unfortunately I’m short on time right now.
I hope I’ve clarified the situation.

 

[Divine_Barakah]

Unfortunately, didn't you wonder why I took it personally?
Before I came across your thread, you had commented on about 3 or 4 videos with the same message.
I'm sorry, but with such persistence, I took it as an attack—a former member had already done that here.
Especially since my method hasn’t changed since the beginning:

- URL Test:
Selection of malicious links. I used to post phishing links, but those die out very quickly.

- Malicious fake crack: Removed a while ago because the source I was using shut down… I’m currently looking for another one.

- Malware Pack: A selection of recent and older malware to test the lab’s responsiveness (.exe files, yes, but not exclusively—there are also VBS, JS, scripts, exploits, etc.).
Yes, that’s right—I run a scan of the pack. I hate the “gang-bang” effect where everything is executed via a script.
I launch it, wait about 15 to 20 seconds, and then continue.
That’s my method

I was also working on including system penetration tests (using Meterpreter or Cobalt), but unfortunately I’m short on time right now.
I hope I’ve clarified the situation.

Well it is unfortunate that you took it personally. Just imagine if F-secure and other products u tested took it personally

Everything I mentioned remains valid and unquestionable, supported by methodologies adopted by professional labs the results of which are more credible and reflect real-world scenarios.

I apologise if you felt attacked, but just remember I have nothing against you. I just criticised the process and I am entitled to that.

 

[Andy Ful]

Malware testing is a hard task. The results of a single concrete test can depend heavily on the testing methodology and the pool of test samples.
So, the results of the AV-Comparatives Real-World test are often different from AV-Test, AVLab, or SE Labs tests and also from tests conducted by MT members.
None of those tests can fully describe the in-the-wild reality.

The methodology suggested by AI is close to that used by AV-Comparatives. Each day, they choose a larger pool of suspicious fresh samples and identify only a few as malware. Over four months, they include about 400 representative samples in the test (3-4 samples a day).
In theory, it is almost impossible to conduct a similar test at home, for example:

1. How to identify really fresh samples (AV-Comparatives uses a web-crawler).
2. How to choose a good representative sample (professional Labs have many years of experience in that).
3. How to perform tests for many AVs at the same time (a difference of a few minutes can be important).
4. How to conduct the test without delay (from identifying a suspicious sample to finally testing AVs).

So the home-made tests are usually based on a different methodology. The samples are usually random, and AVs show a different face compared to other tests.

 

[Divine_Barakah]

Malware testing is a hard task. The results of a single concrete test can depend heavily on the testing methodology and the pool of test samples.
So, the results of the AV-Comparatives Real-World test are often different from AV-Test, AVLab, or SE Labs tests and also from tests conducted by MT members.
None of those tests can fully describe the in-the-wild reality.

The methodology suggested by AI is close to that used by AV-Comparatives. Each day, they choose a larger pool of suspicious fresh samples and identify only a few as malware. Over four months, they include about 400 representative samples in the test (3-4 samples a day).
In theory, it is almost impossible to conduct a similar test at home, for example:

1. How to identify really fresh samples (AV-Comparatives uses a web-crawler).
2. How to choose a good representative sample (professional Labs have many years of experience in that).
3. How to perform tests for many AVs at the same time (a difference of a few minutes can be important).
4. How to conduct the test without delay (from identifying a suspicious sample to finally testing AVs).

So the home-made tests are usually based on a different methodology. The samples are usually random, and AVs show a different face compared to other tests.

Thank you, Andy, for bringing your expertise here. I do appreciate the insights you shared. Since, as you mentioned, it is impossible to conduct a similar test at home, or even here at MT, what are the practical and credible steps that can be taken to develop a proper methodology to test the products here with the available means?

 

[Sunqfu]

Thank you, Andy, for bringing your expertise here. I do appreciate the insights you shared. Since, as you mentioned, it is impossible to conduct a similar test at home, or even here at MT, what are the practical and credible steps that can be taken to develop a proper methodology to test the products here with the available means?

Asking the Gemini maybe.

 

[Divine_Barakah]

Asking the Gemini maybe.

The forum's peanut gallery cannot resist taking a cheap shot.

I asked Andy, one of the most expert and credible members, for his suggestions on how to develop a proper, feasible methodology with the available means so that we can further develop how products should be tested.

I'm more than happy to hear constructive ideas from anyone, but I'm particularly interested in Andy's technical perspective on this given his extensive development background.

 

[Trident]

I can assist with developing platforms and tools.

 

[silversurfer]

I asked Gemini to build a scientific, proper methodology for testing security product and here is its answer.

Transitioning from chaotic "sandbox theater" to a scientifically sound, reputable methodology is the best move you can make for the community. If you want to run tests on MalwareTips (MT) that actually command respect from enterprise-grade analysts and seasoned forum veterans, you don’t need a multi-million dollar lab infrastructure—you just need a **hygienic, sequential lifecycle testing framework**.
By adopting a stripped-down version of the methodologies used by AMTSO-compliant labs like AV-Comparatives and AVLab, you can publish simple but highly accurate tests.
Here is a proper, highly reputable, but completely achievable methodology for a forum reviewer.
## The Clean-State Sequential Methodology
### 1. Environment Hygiene (The One-and-Done Baseline)
The number one rule of scientific testing is isolating your variables. You must eliminate resource contention and cross-contamination.
* **The Blueprint:** Set up a clean Virtual Machine with standard user configurations (or a debloated target environment if testing specific hardening profiles). Install **only one** security suite.
* **The Snapshot Rule:** Ensure the OS, browser, and security software are fully updated, then take a pristine snapshot.
* **The Purge:** After testing **one single threat**, record the result, discard the machine's state, and roll back to the clean snapshot. Never feed a second sample into a machine that has just processed or been altered by a previous threat.
### 2. Sequential Delivery (Mimicking the Threat Lifecycle)
To test a product fairly, you must allow it to defend the system tier-by-tier, exactly how a real user encounters a threat. **Never dump multiple files into a folder or execute them simultaneously.**
* **The Execution Flow:** Run your samples one by one, moving through the four definitive defensive gates:
1. **Gate 1: Web-Layer / Initial Access:** Input a fresh, live malicious URL directly into the browser. Does the network filter intercept it? If yes \rightarrow **Blocked (Web Layer)**. Reset snapshot.
2. **Gate 2: Pre-Launch / Static Disk Write:** If the web layer misses it and the file lands on the disk, trigger a manual scan or wait for the real-time protection to intercept the file on write. If yes \rightarrow **Blocked (Pre-Launch)**. Reset snapshot.
3. **Gate 3: Runtime Execution / Behavioral Heuristics:** If it's not caught on disk, double-click to execute the sample. Give the behavioral loops, cloud sandboxes, and memory hooks a clean window of 3 to 5 minutes to intercept the active process. If yes \rightarrow **Blocked (Runtime)**. Reset snapshot.
4. **Gate 4: Remediation & Rollback:** If the malware successfully executes changes (e.g., drops a registry key or begins a file-encryption routine), check if the product cleanly kills the parent tree and restores the modified files. If it fails entirely \rightarrow **System Compromised**.
### 3. Hygienic Sample Selection (Quality Over Quantity)
A reputable test uses a small pool of high-quality, fully vetted samples rather than an un-audited "zoo pack."
* **The Golden Rule:** 10 fresh, confirmed, active in-the-wild threat binaries (like a newly compiled Infostealer, a fresh Ransomware variant, or a live MSHTML exploit) will yield a vastly more accurate assessment than 150 dead, corrupt, or duplicated legacy samples.
* **The Verification Pass:** Before utilizing a sample in your actual test, run it on a separate, completely unprotected VM baseline to ensure the binary is active, reaches out to its Command and Control (C2) server, or successfully performs its payload delivery. If the sample crashes on a clean OS, throw it out—it's a dud.
### 4. Balanced Reporting (The False Positive Control)
A security engine can easily achieve a 100% block rate if its heuristics are turned up to a purely paranoid state that treats every unknown binary as malicious.
* To make your methodology truly reputable, always include a **False Positive Control Phase**. Run a sequence of 5 entirely legitimate, niche open-source utilities or developer tools. If a product blocks clean software or throws critical warnings on safe files, its high detection score is artificial and degrades system usability.
## Example Test Reporting Template
When you post your results, avoid flat binary values like "Passed / Failed." Present your data using a clear, multi-tiered scorecard:
| Sample ID | Threat Type | Access Vector | Block Stage | Outcome | Notes / Behavioral Observations |
|---|---|---|---|---|---|
| **#01** | RedLine Stealer | Live URL | Web Layer | **Clean Block** | Intercepted via network reputation layer. |
| **#02** | LockBit v4 | Local Archive | Runtime | **Clean Block** | Allowed launch, journaled execution, killed & rolled back encryption loop. |
| **#03** | Custom LPE Exploit | Dropped Script | Execution | **Compromised** | Bypassed behavioral hooks; successfully spawned SYSTEM shell. |


What do you think? Any suggestions? Any member has the time and capacity to start working on this?

Here's a revised version from Claude Pro.

The Clean-State Sequential Methodology (Revised)​

A Reproducible Antivirus Testing Framework for Forum Reviewers​

This is a revised version of the original "Clean-State Sequential Methodology," tightened for statistical validity, sandbox-awareness, and safe handling of live samples. The core structure — environment hygiene, sequential delivery, hygienic sample selection, and balanced reporting — is preserved, with fixes layered in at each stage.

---

0. Before You Start: Documentation & Containment​

Two things the original draft skipped, and both are non-negotiable for a test anyone will actually trust:
Version logging. Record, for every test run: product name and build/engine version, signature/definition version and timestamp, OS build, and test date/time. Without this, the results can't be reproduced or audited later — and any credible reader will ask for it.
Network containment. Live malware in Gate 1 and Gate 3 will try to reach a real C2 server. Your test network needs:

• 
  
• No direct route to the real internet — route traffic through an isolated VLAN or a controlled sinkhole/proxy (e.g., INetSim, FakeNet-NG) that can log outbound requests without letting them succeed.
• 
  
• If you specifically need to confirm live C2 callback (per the original's "Verification Pass"), do that once, in a fully isolated network segment built for it — not on your general test bench.
• 
  
• Never run live samples on a machine with any bridged connection to your home/production network.

This isn't optional flavor text — running live ransomware or stealers with real internet access risks actual data exfiltration or the sample phoning home with details about your test setup.

---

1. Environment Hygiene (Unchanged, with one addition)​

Same as the original: one clean VM, one security product, pristine snapshot, roll back after every single sample.
Addition: note whether the product uses cloud-assisted detection (most do). Rolling back your local VM snapshot does not undo what the vendor's cloud reputation service has already logged about that file hash or your test IP. If you re-test the same sample later, be aware the product may now recognize it purely from prior submission — which is a confound, not a repeat measurement.

2. Sequential Delivery (Same four gates, with dwell-time and evasion fixes)​

Keep the four gates (Web-Layer → Pre-Launch → Runtime → Remediation), but with these adjustments:

• 
  
• Extend and vary the runtime dwell window. 3–5 minutes is too short and too predictable. Modern loaders and staged ransomware use sleep-evasion specifically to outlast short sandbox windows. Use a minimum of 15–20 minutes of observation, and consider accelerating system clocks or simulating idle/user activity to counter environment-aware stalling.
• 
  
• Watch for VM-evasion, not just malware behavior. If a sample exits immediately, silently, or without any expected artifacts, check whether it detected virtualization (common checks: CPUID, registry artifacts, driver names, MAC address OUI) before concluding the AV "blocked" it. A silent exit can mean the malware refused to run — not that the product stopped it. Log this as Inconclusive (Suspected VM-Evasion) rather than a clean block.
• 
  
• Run each sample 3 times, not once, on fresh snapshots. Cloud lookups, telemetry timing, and behavioral heuristics are often non-deterministic — a single run conflates "product failed" with "product got unlucky." Report the outcome as a ratio (e.g., 3/3 blocked, 2/3 blocked) rather than a single pass/fail.

3. Hygienic Sample Selection (Same principle, larger pool)​

The original's instinct — quality over quantity — is right, but the pool is too small to support real conclusions.

• 
  
• Minimum viable pool: 30 threats, not 10. At n=10, one unusual sample can swing a product's score by 10 percentage points. 30 gets you closer to a stable signal while still staying achievable for a forum reviewer without enterprise resources.
• 
  
• Diversify delivery vectors and threat families within that pool (a mix of infostealers, ransomware, loaders, exploits — not 10 variants of the same family), so the test isn't accidentally measuring performance against one detection signature.
• 
  
• Keep the original's Verification Pass (confirm the sample is live and functional on an unprotected baseline before using it) — this part was solid.

4. Balanced Reporting (Same false-positive phase, expanded)​

• 
  
• Keep the False Positive Control Phase — 5 legitimate niche tools is a reasonable minimum; consider raising to 10 if time allows, since FP rates are also sensitive to small samples.
• 
  
• Add time-to-detection/time-to-remediation to the scorecard. A "clean block" that happens after a ransomware sample has already encrypted several files is materially different from an instant block, even though both might get recorded as "Clean Block" under the original template.
• 
  
• Add a confidence/consistency column reflecting the 3-run repeat testing from Section 2.

---

Example Revised Scorecard​

Sample ID	Threat Type	Access Vector	Runs Blocked	Block Stage	Time to Detection	Outcome	Notes
#01	RedLine Stealer	Live URL	3/3	Web Layer	<1s	Clean Block	Consistent network reputation block across all runs.
#02	LockBit v4 variant	Local Archive	2/3	Runtime	~45s (1 run: 4 files encrypted before kill)	Partial/Inconsistent	One run showed partial encryption before remediation — flag for vendor follow-up.
#03	Custom LPE Exploit	Dropped Script	0/3	—	—	Compromised	Confirmed non-VM-evasive (payload executed identically on bare-metal control); genuine miss.

---

Summary of Changes from the Original​

Area	Original	Revised
Sample pool	10 threats	30 threats, diversified
Repeat runs	1 per sample	3 per sample, reported as ratio
Runtime dwell window	3–5 min, fixed	15–20 min, with evasion checks
Network handling	Not addressed	Isolated/sinkholed network required
Version logging	Not addressed	Required per test run
VM-evasion	Not addressed	Explicit "Inconclusive" outcome category
Reporting	Pass/Fail/Compromised	Adds time-to-detection + consistency ratio

This keeps the forum-friendly, no-lab-required spirit of the original while closing the gaps that would let a skeptical reader (or a vendor) poke holes in the results.


I asked it this, too and got the following response.

That's correct, and it's worth being explicit about why it's non-negotiable despite the time cost.


Why you can't skip it


Cross-contamination between samples invalidates the test. A few concrete ways it fails silently:

• Leftover artifacts skew detection. If sample #2 runs on a system where sample #1 already dropped a registry key, disabled a service, or left a persistence mechanism, the AV is no longer being tested against a clean encounter — it's facing an already-degraded system. A "compromised" result might reflect sample #1's damage, not sample #2's evasion.
• AV state changes after an encounter. Many products raise their sensitivity, quarantine broadly, or flag a system as "already compromised" after a detection — meaning the next sample gets tested against a paranoid, alerted product rather than its normal baseline behavior. That inflates detection rates artificially.
• Resource/behavioral noise. A previous sample's processes, injected code, or partial encryption routines can alter memory state, running services, and disk I/O patterns in ways that confuse a behavioral engine's read on the next sample — in either direction.

So rollback-per-sample isn't just good practice, it's the thing that makes each row in your scorecard an independent data point rather than a chain of contaminated ones.


Managing the time cost without cutting the rollback


Since this is the real bottleneck, here's where the time savings should come from instead:

• Automate snapshot revert. Most hypervisors (VirtualBox VBoxManage snapshot restore, VMware vmrun revertToSnapshot, Hyper-V Restore-VMSnapshot) support scripted rollback in seconds — this is far faster than manual GUI reverts and removes human delay between samples.
• Parallelize across VMs, not within one. Run multiple clean VM instances simultaneously (one per sample) rather than serially rolling back a single VM. This multiplies your throughput without touching the one-sample-per-clean-state rule.
• Batch the fixed-cost steps. Do OS/product updates and the pristine baseline snapshot once, then clone that snapshot per test rather than rebuilding the environment each time.
• Accept the time cost as the tradeoff for the smaller sample pool. Since I bumped the recommended pool to 30 samples × 3 runs = 90 total test cycles, rollback automation isn't optional at that scale — without it, the revised methodology becomes impractical for a solo forum reviewer. If full automation isn't feasible, it may be more honest to scale the sample count back down and say so explicitly, rather than skip rollbacks to save time.

@Divine_Barakah @roger_m
Please remember to use optional forum feature spoilers for long content from AI
That would be easier reading for some forum people without to scroll down like endlessly...

 

[Divine_Barakah]

@Divine_Barakah @roger_m
Please remember to use optional forum feature spoilers for long content from AI
That would be easier reading for some forum people without to scroll down like endlessly...

Noted. I did not know this feature existed.

 

[Trident]

I have a personal threat hunting platform that I use. I will enrich it with my signature OrbitUX and release it to you guys by the end of the day. It’s not a big deal but a tad bit of help.