1. Self-protection (for processes - I can give you information on registry/file protection too if you want but you'll have to ask nicely): you can take a few different approaches for this one. I will give you some methods (some better than others):
Method one:
The first method is by far the easiest and it works by altering the security descriptors of your process... Through this you can actually prevent any non-administrator processes from opening a handle to your own process, leaving it an Access Denied error as response. This is good to prevent any malware running on the system which has not been granted administrator privileges from attacking your process, however it's one of the worst methods I can think of since it's not very good in terms of protection.
Method two:
This second method I will briefly explain used to be used by a lot of vendors however now most vendors have moved to using method four. This method evolves around user-mode hooking, therefore injecting into all processes running on the system (either a DLL (Dynamic Link Library) or code via codecave injection) and then you will hook various functions to prevent the target processes from attacking your process.
As an example, within the DLL you are injecting, you may hook NtOpenProcess. NtOpenProcess is an Native API function which is eventually called once OpenProcess has been called (OpenProcess is a Win32 API function). To use functions like NtTerminateProcess (or TerminateProcess which will call this), NtSuspendProcess, NtAllocateVirtualMemory, NtWriteVirtualMemory, CreateRemoteThread, etc (or the functions to attack the process directly) you'll require a handle to the target process. Therefore, blocking access to open a handle will solve your problems... A callback to the function hook would look similar to this:
Code:
NTSTATUS NTAPI NtOpenProcess_CB(PHANDLE ProcessHandle, ACCESS_MASK DesiredAccess, POBJECT_ATTRIBUTES ObjectAttributes, PCLIENT_ID ClientId)
{
if (ClientId->UniqueProcess == (HANDLE)5555) { return 0xC0000022; }
return NtOpenProcessTramp(ProcessHandle, DesiredAccess, ObjectAttributes, ClientId);
}
The above callback code will check the process a handle is attempting to be opened too and if it is the process with the PID 5555 (e.g. the process we want to protect) it will return STATUS_ACCESS_DENIED. If the process a handle is being opened too is not the process we want to protect, it will return with the trampoline (therefore the original function is called without triggering the hook and causing a loop, therefore the handle opening is allowed).
Method three:
This third method will evolve via kernel-mode patching. From Windows Vista and on-wards Microsoft introduced something called PatchGuard/KPP which will prevent kernel-patching on x64 editions of Windows, however you can still do it for x86 systems. Kernel-mode patching would be kernel-mode hooking (e.g. System Service Dispatch Table hooking) for example. Through this you can hook functions like PsLookupProcessByProcessId, ObOpenObjectByPointer, NtTerminateProcess, NtOpenProcess, NtSuspendProcess, thread functions as well if you want to protect those also (which is also very important).
Method four:
This method is compatible with both x86 and x64 systems regardless of PatchGuard/KPP and is documented by Microsoft themselves. This method is also the EXACT same method that most popular and top AV products will use... Kernel-mode callbacks! For process protection, the callback ObRegisterCallbacks is utilised. It provides a pre and post operation. On the Pre operation function you can check the process being targeted and if it is the one you want to protect you can remove the access rights from the handle being generated which will result in Access Denied. You can also protect the process' threads via this method (and prevent duplicating handles). The only way to bypass this method (if it is implemented correctly and all flaws are patched up) would be to execute code from kernel-mode itself, calling functions like ObOpenObjectByPointer to bypass the access checks, or to remove the callback for example...
2. BB/HIPS
This is usually developed via user-mode API hooking. This will work by injecting code into all running processes (e.g. via DLL) which will then hook various functions (such as CreateRemoteThread, NtWriteVirtualMemory, RtlCreateUserThread, NtCreateKey, NtSetValueKey, NtDeleteKey, NtLoadDriver, NtSetSystemInformation (some rootkits will attempt to load device drivers silently via this method), service functions, GetProcAddress (or go lower than this), Ldr functions if necessary, etc). If you really want to go much more lower-level, you can hook functions like KiFastSystemCall (on older OS versions like Windows 7) or X86SwitchTo64BitMode and take even more control... Or if not, you can hook the actual Nt/Zw function stubs as opposed to simple IAT/EAT hooking methods which can be bypassed more easily.
If you are on x86 you can use SSDT hooking for some things like NtLoadDriver hooks... And then call IoGetCurrentProcess() to trace the actions back to the caller process.
3. Static heuristics
You can implement this via the use of generic signatures (e.g. HEX detection based on the bytes of malicious code/parts of HEX only found in multiple malicous samples), scanning of the IAT/EAT (Import Address Table/Export Address Table), checking the PE File Header for information on detecting signs of packing/calculating the Entropy level, checking digital signature and checking validation of it, etc. Study malware analysis and this will be much easier.
4. Dynamic Heuristics
Hooking dependant again really... You can also make an advanced memory scanner to improve detection since you'll be able to use some static analysis methods dynamically (e.g. trace when malware unpacks itself in memory, then utilise the static scanning methods).
5. Anti-Executable
This is very important for people who want default deny... You can learn how to develop your own Anti-Exe via my thread:
Developing your own Anti-Exe (C - Device Driver Development)
I recommend you make a real engine in C/C++/ASM and if you do not have experience with Win32 GUI development in another language (e.g. Native C++) then you may as well stick with C#.NET for it. You can work with IPC (Interprocess Communication) between C/C++ and .NET MSIL processes. An example would be via shared memory (memory mapped files) or pipes. As for device drivers, you can send data to them via IOCTls for example.