From the sounds of it, you were the victim of an exploit hitting a legitimate app. There's no satisfactory solution to this imo, even today. Ie if an mp4 does a buffer overflow to VLC player and VLC player is a trusted app, no whitelisting will stop this, and from my discussion in another thread with an AV developer it sounded like no BB would realistically catch this either.
If you want to cast the movie (I never watch anything on a laptop anymore), I'm not sure how well sandboxing solutions would work with Chromecast, if someone has tried and chromecast did work for them, that would be an interesting approach, sandboxing the player only for "risky" movies/mp4s etc.
Another solution would be to enforce very strict policies even for legitimate apps but this would be a nightmare to maintain with updates etc.
Probably the best solution to isolate this threat vector would be to run downloaded movies inside a container ( real container, that uses kernel namespaces ) on your media server which can access only LAN IPs and use XWindows on your desktop machine to control the container. It would take something really advanced to take this down eg a kernel exploit on the media server to cross the containerization gap.
For even more security a VM in your media server instead of a container would also be viable ( albeit responsiveness of vt-x VMs is disappointing ).
For less security an AppContainerized player.