Do you consider this a block or a fail?

[Andy Ful]

Picture explains it all. In one of the test videos the testers response after this block is to run the sample as admin and passes SRP. And hard-configurator allows it (because it is set up that way). Next ransomware with admin rights destroys the system. What is the consensus of the members. Block or Fail

View attachment 211815

If the test was performed with Hard_Configurator, then the settings are not the standard ones.
On Windows 8+ in the recommended H_C settings, the user cannot use "Run as administrator" option to run files with admin rights. The file can be "Run As SmartScreen", which means that is run as administrator only if it is accepted as safe by Windows SmartScreen Application Reputation filter. If the malware from the test managed to fool the SmartScreen and infect the system, then it should be counted as a fail. This is possible (very rarely), and there was one such case in @askalan tests for H_C (no AV setup) made in the period January-March 2019.

Hard_Configurator - January 2019 Report

Disclaimer: Experimental setup for testing the effectiveness of Windows SmartScreen and script restrictions against 0-day malware samples. This test is suitable for users with more knowledge about Windows built-in security features. changed configuration from 7 January 2019: 1. Containment...

malwaretips.com

Hard_Configurator - February 2019 Report

Containment: VirtualBox 5.1.38 Windows: 10 LTSB VPN: CyberGhost Product: Windows SmartScreen (activated by Hard_Configurator with recommended SRP and restrictions) Office: LibreOffice (standard settings) Disclaimer: Experimental setup for testing effectiveness of Windows SmartScreen and script...

malwaretips.com

Hard Configurator - march 2019 report

Hard Configurator report for march 2019 Containment: VirtualBox 5.1.38 Windows: 10 LTSB VPN: CyberGhost Product: Windows SmartScreen (activated by Hard_Configurator with recommended SRP and restrictions) Office: LibreOffice (standard settings) al-khasar test results: link Disclaimer...

malwaretips.com

 

[AlanOstaszewski]

I think the failure was in the testing methodology

My test testing methodology is:
1. I double click on the file first so that the inexperienced people who see my tests notice that you can only start the file (if you want to) with the "Run As SmartScreen" mode.

2. I run it with the "Run As SmartScreen" mode.

3. Block or fail

Where exactly is the failure?

 

[Andy Ful]

My test testing methodology is:
1. I double click on the file first so that the inexperienced people who see my tests notice that you can only start the file (if you want to) with the "Run As SmartScreen" mode.

2. I run it with the "Run As SmartScreen" mode.

3. Block or fail

Where exactly is the failure?

I think that all our posts do not apply to your tests, because @Windows_Security has not mentioned in his post if "Run As SmartScreen" was used in the test to run executables. I assumed (and probably other MT members), that standard "Run as administrator" was used in the test to run executables. If so, then the test procedure could be questionable.

Edit.
If I correctly remember, @Windows_Security uses SRP + allowing elevation of signed executables only, and does not use "Run As SmartScreen".

 

[shmu26]

I assumed (and probably other MT members), that standard "Run as administrator" was used in the test to run executables.

+1
My apologies to @askalan

 

[RoboMan]

Wait, were we talking about an @askalan test? I just assumed we were creating an hipothetical case, so I just talked about ordinary final users. Didn't mean to trash talk about you or any test of yours @askalan, I know you know a lot more than me

 

[Andy Ful]

Wait, were we talking about an @askalan test? I just assumed we were creating an hipothetical case, so I just talked about ordinary final users. Didn't mean to trash talk about you or any test of yours @askalan, I know you know a lot more than me

We did not, but it is probable that @Windows_Security did (I am not sure).
The fact is that MalwareHub tests are very demanding for the tested products, more demanding than real-world scenario. In the real world, the one malware that managed to bypass forced SmartScreen and infected the system (in @askalan tests), would be possibly a payload which would not be downloaded at all due to H_C script restrictions, or blocked by SRP. But, this is also true for AVs tested on MalwereHub, because they can detect some scripts and can block malicious downloads via the web shield. That is why @askalan counted this sample (correctly) as H_C's protection failure.

 

[Andy Ful]

I examined @askalan's video on MalwareHub, and now I am sure that the picture from @Windows_Security post is from this video. It means that in this particular test, the fact of blocking the malware by SRP, and next allowing it when using "Run As SmartScreen" should be counted as the H_C protection failure.
This classification follows from the very demanding procedure used in MalwareHub tests.

 

[shmu26]

I examined @askalan's video on MalwareHub, and now I am sure that the picture from @Windows_Security post is from this video. It means that in this particular test, the fact of blocking the malware by SRP, and next allowing it when using "Run As SmartScreen" should be counted as the H_C protection failure.

Lemme make sure I got this right:
1 file was run with standard rights, and SRP blocked it. (expected behavior)
2 "Run As SmartScreen" was set to grant elevated rights.
3 file was run by "Run As SmartScreen" and was not blocked. (Smartscreen failed)
4 purpose of vid was to test Smartscreen.

Is this what happened?

 

[AlanOstaszewski]

Lemme make sure I got this right:
1 file was run with standard rights, and SRP blocked it. (expected behavior)
2 "Run As SmartScreen" was set to grant elevated rights.
3 file was run by "Run As SmartScreen" and was not blocked. (Smartscreen failed)

Is this what happened?

Exactly.

 

[AlanOstaszewski]

Lemme make sure I got this right:
1 file was run with standard rights, and SRP blocked it. (expected behavior)
2 "Run As SmartScreen" was set to grant elevated rights.
3 file was run by "Run As SmartScreen" and was not blocked. (Smartscreen failed)
4 purpose of vid was to test Smartscreen.

Is this what happened?

You can also see it here:

 

[shmu26]

Exactly.

Thank you.
I apologize again for my earlier comment, I simply didn't know the context.
Not having watched the vid yet, was this the "famous" case of the signed malware that Smartscreen failed on?
If not, any theories why Smartscreen missed it?

 

[Andy Ful]

Thank you.
I apologize again for my earlier comment, I simply didn't know the context.
Not having watched the vid yet, was this the "famous" case of the signed malware that Smartscreen failed on?
If not, any theories why Smartscreen missed it?

SmartScreen Application Reputation filter is based on Artifical Intelligence which can calculate the reputation of applications. It can take as input many factors like application prevalence, digital certificate, reputation history, etc. If the application is signed with a common digital certificate then other factors (like prevalence and reputation history) are also important. So, most signed malware can be blocked by SmartScreen anyway.
Yet, if the never-seen-malware uses Extended Validation Certificate (requires verification of the requesting entity's identity by a certificate authority) or digital certificate stolen from a very popular application, then it can usually bypass SmartScreen. That happened twice in @skalan tests (one malware cannot infect the testing system anyway).
In H_C, the SmartScreen is used only for application installers (not payloads). So in the real world, the chances of infection in the home environment via SmartScreen are close to 0, because H_C settings apply other protection layers to prevent payloads.

 

[Djnjd09]

Oh, By the way, Is it a Work Computer or Home Computer?