There are many different methods when it comes to sandboxing programs however without manual analysis on specific sandboxing software it's hard to determine what techniques are being utilised (unless specified publicly by the vendor).
API hooking -
most sandboxing software will evolve around this method since it allows the sandbox software to alter the execution flow of the target program and intercept upon certain API calls (which are targeted) and then continue to either redirect/block the actions entirely or just log it and allow it to pass through. Of course nothing is bullet-proof and everything has flaws however this method is also commonly found for the internals of most BB/HIPS/Dynamic Heuristics functionality in traditional AV/IS software (and also can be used for self-protection mechanisms in security software).
On x64 systems you won't really find kernel-mode sandboxing techniques being utilised - you can find sandbox software making use of File system Mini-filter drivers to restrict access to specific folders/files, but you won't find sandboxing software using kernel-patching techniques due to KPP (Kernel Patch Protection) which is integrated into PatchGuard (one of the Windows security features which was introduced at the start of Windows Vista for x64 versions of the OS only).
On x86 systems sandboxing software can be much more secure since it can utilise kernel-patching techniques (e.g. hook the System Service Dispatch Table) and thus this makes bypassing the sandbox software much more difficult - this is due to not being able to escape the hooks without unhooking of the SSDT which would require kernel-mode components (and if the sandbox restricted the program properly it would be unable to load a device driver to give it the advantage and power of escaping the sandbox).
Most of the time you'll find sandboxing software using user-mode (ring 3) hooks (e.g. IAT/EAT hooking techniques - this is the practise of modifying the pointer addresses for a function in the IAT/EAT table to the pointer address of the callback function, which will result in code execution flow modification so the new callback function is invoked when the target hooked function is called by the affected program/s). The main downside of these techniques is that sandboxed malware can potentially find methods of escaping the sandbox due to manual implementation of the function prologue (e.g. NTAPI) they want to call/system calls, thus evading the set hooks without alarming the sandboxing software - instead of this an alternate would be to just attempt to remove the hooks and then call the functions as it normally would.
However kernel-mode patching for sandboxing is much more difficult than just some kernel-mode hooking techniques since it's system-wide and thus results in affecting all running programs (so lots of work to polish it up to work right is necessary), therefore it won't work as well as user-mode hooking (which would only affect the target programs which have the sandbox code injected into them), regardless of if kernel-mode patching is more secure than user-mode patching methods - user-mode methods are much more stable since it won't result in a system-wide crash if something goes wrong, also.
Aside from API hooking methods there are other sandboxing methods which may come into role:
One idea for sandboxing would be to run the target program as a child to a container parent process which has restricted access (running with lower priorities) and therefore if the system did end up becoming compromised due to the malware bypassing the sandbox, it'd face the new problem of being ran with restricted privileges due to the OS restrictions (since the process would be of a low authority and wouldn't be able to carry out specific actions without being higher up). I believe many sandbox software use this method, maybe even Sandboxie also.
As an example, I believe Sandboxie utilises kernel-mode patching on x86 systems via use of it's device driver/s however on x64 systems it will be working with user-mode patching methods (e.g. IAT hooking) - it most likely uses device drivers on x64 systems also for some things (possibly a file system mini-filter driver for file/folder restriction). Please do correct me if I am wrong.
When it comes to isolation,
virtualisation is a much better and secure approach compared to sandboxing and therefore if you are considering of testing malware then I recommend you just stick to a Virtual Machine since escaping a VM is a complete new story than a sandbox... Most VM exploits are due buffer overflow attacks and exploiting of vulnerabilities found in the VM additional features (such as: Shared Folder, shared Clipboard, etc) - and are much harder to identify and actually exploit.
Hope this helped and if I misunderstood your question then I do apologise!
Stay safe,
Wave.
