I don't specifically remember how I had it set up, but I would usually lean towards the more paranoid settings back then. I did read about a windows zero day vulnerability with media files a few days after this happened but I had wiped the system by that point. I remember as I ran the file I looked at the file size (it was a TV show and the size was only a few megabytes) and I immediately thought it could be malware but what's the worry my hips will notify me. It sketched me out for a while after and I noticed it connecting out when I checked the outbound connections. I felt like the French with their Maginot line.
From the sounds of it, you were the victim of an exploit hitting a legitimate app. There's no satisfactory solution to this imo, even today. Ie if an mp4 does a buffer overflow to VLC player and VLC player is a trusted app, no whitelisting will stop this, and from my discussion in another thread with an AV developer it sounded like no BB would realistically catch this either.
If you want to cast the movie (I never watch anything on a laptop anymore), I'm not sure how well sandboxing solutions would work with Chromecast, if someone has tried and chromecast did work for them, that would be an interesting approach, sandboxing the player only for "risky" movies/mp4s etc.
Another solution would be to enforce very strict policies even for legitimate apps but this would be a nightmare to maintain with updates etc.
Probably the best solution to isolate this threat vector would be to run downloaded movies inside a container ( real container, that uses kernel namespaces ) on your media server which can access only LAN IPs and use XWindows on your desktop machine to control the container. It would take something really advanced to take this down eg a kernel exploit on the media server to cross the containerization gap.
For even more security a VM in your media server instead of a container would also be viable ( albeit responsiveness of vt-x VMs is disappointing ).
For less security an AppContainerized player.