Can anyone confirm this data corruption bug?

Discussion in 'Shadow Defender' started by Kawamei, Apr 11, 2017.

  1. Kawamei

    Kawamei Level 1

    Apr 11, 2017

    I noticed in the thread over at Wilders that some people were occasionally experiencing lost applications settings, reset windows settings and such when using Shadow Defender. All of these symptoms could be traced to corrupt files, and so far I've been able to reproduce the phenomenon on 2 physical machines (albeit running in a VM).
    It seems no one else has really tried to systematically replicate the error so far.

    Here's what I did:

    I used an untouched WinXP SP3 VM with a virtual disk size of 8GB with no additional software installed. I installed a Hasher (corz checksum) and Shadow Defender. Next, i copied a folder with 100MB worth of files with various file sizes to the VM, placing it in a temporary folder (let's call it "Folder0"). I used an extracted old Java installation folder, with the largest file being 40MB, most other files much smaller. The mix seems to be important, as corruption affects smaller files more often, but there has to be enough traffic over the Shadow Defender driver.

    Next, i created hashes for Folder0 and wrote a quick&dirty batchfile to create 20 duplicates of this folder.

    Something like that works:

    @echo off
    FOR /L %%i IN (0,1,20) DO @CALL :COPY %%i
    echo Done.
    xcopy "Folder0" "Folder%1" /S /E  /I /F
    and saved it as a .bat next to "Folder0".

    Enter Shadow Mode and run the file.

    After it's done, Verifying all folders (running checksum from the parent folder) in most cases does not show any corrupt files.

    However, after deleting all folders except Folder0 and repeating the process, I am guaranteed to see corrupt files, more often than not a lot of them. It's really scary :)

    Probably more scary, at least for me, would be if I am really the only one experiencing this problem :eek:
    Mr.X, Danielx64 and Rengar like this.
  2. Danielx64

    Danielx64 Level 9

    Mar 24, 2017
    Windows 10
    Maybe it just me but I rather use deepfreeze but that's a different story. I will be watching this though
  3. Amelith Nargothrond

    Mar 22, 2017
    Windows 10
    Interesting, i'll try to reproduce later today.
  4. Kawamei

    Kawamei Level 1

    Apr 11, 2017
    I have to add that sometimes it's necessary to repeat the process several times, but I never completely failed to reproduce the bug.

    I am not sure what the occurrence of this error depends on. Whether it's the amount of I/O traffic the driver has to handle, or the amount of cache "Space used by Shadow Defender" vs remaining free space on the disk (after all, it rarely works the first time without deleting Folders 1-20 first).

    Also, you may need to increase the number of copies (replace the "20" in the batch with anything you like).

    I've tried to reproduce the errors many time with Shadow Mode deacivated, just to rule out hardware failure on the host system, and there never was a single corrupt file.

    I'll try it again on the new version ( released today, maybe it's already fixed. ;)
    Mr.X likes this.
  5. Kawamei

    Kawamei Level 1

    Apr 11, 2017
    Here's a tutorial copied over from Wilders to make the process easier:


    Nice to see that the development of Shadow Defender is continuing, so I'm back with some news about the mythical corruption issue :)

    For new users, check it out here:

    I have since zero-filled my HDDs, then run extended self-tests on then, after that memtest86, and finally rebuilt my System. After a while, I started testing Shadow Defender again, this time in a 32bit Windows 7 SP1 VMWare environment, and I cound NOT reproduce the corruption, at least not immediately.

    I have used a second, small 4GB virtual Hard Disk, putting it in Shadow Mode and trying to reproduce corruption, to no avail. After a while, I tried the same on the system disk, and the corruption was back almost instantly. I then created another, 16GB virtual disk, filled it with random files until the number of file approximately matched my system drive, and could easily reproduce corruption on it, as well.

    So now I believe that it must be some kind of buffer-overflow tied to the state of MFT, meaning that the number of metadata entries must be above a certain threshold for bad files to occur.

    Anyway, I've devised a relatively hassle-free way for users to check for this phenomenon on their systems, in an unattended way:

    1. Windows 7 SP1 32bit installed on a 16GB Virtual Disk in VMWare (or your preferred virtualization environment, though I've only tested it in VMWare)
    2. Shadow Defender (that's pretty obvious) :)
    3. Java Runtime Environment (Java is not really needed, I just found that the Java folder with its mix of large and small files is ideal for finding corruption, prevents stress on the HDD and my script relies on a Java folder being present in "Program Files")
    4. HashDeep (It's IMHO the best open source commandline hasher, you can get it here:

    Next, do the following:

    1. Create a "temp" folder in the root of C:\

    Be advised that all folders whose name starts with "Copy" inside this temp folder (the script will be run from it) will be deleted!

    2. Put "hashdeep.exe" from the downloaded archive in C:Windows, or any folder listed in the PATH environment variable)

    3. Install Java Runtime Environment (Using the default installation Folder, "C:\Program Files\Java")

    4. Open a command prompt in C:\Program Files\Java (this is important, because of the file paths in the hash file)

    5. Run this command:

    hashdeep -j0 -e -rl * >"C:\temp\audit.txt"
    That's our audit file for hashdeep. :)

    6. Create a .bat (maybe name it shadowtester.bat ?) :) file in C:\temp and paste the following in it:

    @echo off
    setlocal enabledelayedexpansion
    FOR /L %%i in (0,1,30) DO @CALL :COPY %%i
    echo Done.
    xcopy "C:\Program Files\Java" "Copy%1" /S /E /V /I /F /Y
    FOR /D %%G IN ("Copy*") DO (
    PUSHD %%~nxG
    hashdeep -j0 -e -v -v -v -a -k "C:\temp\audit.txt" -rl *
      if !ERRORLEVEL! EQU 1 (
      ECHO Corruption found
      EXIT /b !errorlevel!
    ) ELSE ( POPD )
    FOR /D %%G IN ("Copy*") DO RMDIR "%%G" /S /Q
    This quick&dirty batchfile will infinitely loop, creating 30 copies of "C:\Program Files\Java" in C:\temp, check their integrity, delete them, and start again.

    Finally, install Shadow Defender, enable Shadow Mode and let the script run, the longer, the more certain you can be corruption does not happen on your system. If it stops with a "Corruption found" message, you are affected (and probably should repeat the procedure on your real system) :)

    On my system, corrution is found relatively quickly, quite often within a couple of minutes.

    You may need to increase the number of copies by changing the last counter in the FOR /L loop on the 4th line on the script to something larger than 30, around 50 copies can fit on my Virtual System, but it shouldn't be necessary
Similar Threads Forum Date
Need Help Anyone Seen a Great Standalone System Watcher Program? Apps - Questions & Help Mar 5, 2018
Hacker Who Never Hacked Anyone Gets 33-Month Prison Sentence Security News Feb 27, 2018
Q&A StopAd for Windows - Anyone using it? VPN and Privacy Feb 26, 2018
  • About Us

    Our community has been around since 2010, and we pride ourselves on offering unbiased, critical discussion among people of all different backgrounds about security and technology . We are working every day to make sure our community is one of the best.
  • Need Malware Removal Help?

    If you're being redirected from a site you’re trying to visit, seeing constant pop-up ads, unwanted toolbars or strange search results, your computer may be infected with malware. We offer free malware removal assistance to our members in the Malware Removal Assistance forum.
  • Quick Tip

    Without meaning to, you may click a link that installs malware on your computer. To keep your computer safe, only click links and downloads from sites that you trust. Don’t open any unknown file types, or download programs from pop-ups that appear in your browser.