And on top of that, there are some ways that baselining/trusting can break down too:
If you restrictively baseline based off binaries already on your system, that will affect a lot of apps that auto-update by downloading and executing binaries (including AV engines, Chrome, Firefox, etc). And if you basically say "whitelisted binaries are allowed to get and process other whitelisted binaries" that starts opening you up to exploit-based or fileless attacks. For example, on macOS an old vulnerable version of the Sparkle app updating framework could be fooled into downloading and unpacking arbitrary things from an attacker.
All in all, it takes a lot of dedication from a power user to make this scheme work well. It's simply not realistic to expect the Average Joe or even Above Average Joe to get a lot of benefit from this kind of a setup. They are much better served by the combo of web antiphishing, strong signatures, and a cloud white/blacklist system that controls the sensitivity of a behavior blocker. (Unsurprisingly, that's what every top notch AV suite goes for!)