Identifying in the applications execution flow (e.g. sequence of API calls to determine what it's trying to do instead of assuming a trigger on one or two functions each time) can be a clever idea and work well only if the sequence differs from genuine software (e.g. identifying process hollowing or thread hijack injection can be good but not for other things), however it limits you to specific threats of ransomware as you've already said, plus might not be as effective as alternate options.
The alternate, and most preferred method, would be to just monitor how programs which are untrusted are attempting to alter the files (e.g. is it attempting to perform write requests, is it attempting to change the the extension, etc.); for the write requests interception you can add functionality to identify a change in it's entropy (or just put in detection methods for when files are being encrypted by software). If the activity on a file is suspicious, then trigger an alert for suspicious file modification. I believe this how the Emsisoft Behavior Blocker works for ransomware.
As an extension to doing the above, you can place a few dummy files on the system - maybe in common areas like Documents, Photos, maybe the root of the drive too (since some malware enumerates the drives first and then starts it's way up down as opposed to target areas). If the dummy file becomes modified, you can track which process was responsible with IoGetProcess() from kernel-mode in the device driver which is intercepting the I/O requests, and if the modification symbolizes encryption then you can flag an alert and deal with the situation.
It can be a quite complicated feature to implement with such functionality because it needs to be worked on carefully as it'll be delicate, and it'll lean towards many false positive detection's unless you had a great cloud white-list for verified publishers/genuine software, and perform proper testing to master the filtering to be more adapt to only unknown, suspicious programs.
