Call stacks are an understated yet often important source of telemetry for EDR products. They can provide vital context to an event and be an extremely powerful tool in determining false positives from true positives (especially for credential theft events such as handle access to lsass).

The Windows kernel provides a number of callbacks for AV/EDR drivers to subscribe to in order to receive notifications about system events. For example, this includes process creation/deletion events (PsSetCreateProcessNotifyRoutineEx), thread creation/deletion events (PsSetCreateThreadNotifyRoutine), and object access (ObRegisterCallbacks) etc.

Many of these callbacks run in the context of the thread that triggered the action. Hence, when a kernel driver’s process notify routine is called, it is running in the context of the process that triggered the callback (e.g. via calling CreateProcess) and interprets user mode virtual addresses in the context of that user process. Furthermore, the callback will run inline; the operating system is waiting on the callback to return before it can complete the target action of say creating a process or new thread.

This is demonstrated in the contrived kernel call stack below (obtained from windbg via kernel debugging). This shows a breakpoint set on a custom ObRegisterCallback routine (in this case a process handle operation) which was triggered via Outflank’s dumpert tool:

From this callback, an AV/EDR driver can inspect the object access request and take direct action, such as stripping permission bits from the requested handle if required.  Similarly, from a process or thread callback perspective, an AV/EDR can inspect the new process/thread and take preventative action such as blocking it from executing based on some kind of detection logic/heuristic (does the thread point to sketchy memory? etc.).

Additionally, to support the argument of just how useful call stack collection can be, the example above clearly demonstrates the use of direct system calls as there is no ntdll listed in the call stack prior to nt!KiSystemServiceCopyEnd.

As a word of caution, the ObjectCallback is not actually guaranteed to run in the context of the thread that triggered the action; it runs in what is called an arbitrary thread context (hence the current context may not be the actual process which triggered the callback). However, it appears you can reliably assume this is the case the majority of the time.

What should be clear from the example above is that one action an AV/EDR can perform inline from a kernel callback is the act of walking the call stack. In fact, this is exactly what SysMon does for process access events (Event id 10: Process access).

In the screenshot below we can see a Process access event generated by SysMon, which shows svchost obtaining a handle to lsass:

Figure 1: An example SysMon event for process access where lsass is the target image.

We can see the event contains a ‘CallTrace’ field; this shows the user mode call stack and essentially reveals the chain of events within the process that led to the handle request (albeit without full symbol resolution). This particular event was generated minutes after installing SysMon and occurred at a regular frequency afterwards. Given the call stack does not contain any anomalous memory regions it should be clear this is demonstrably a false positive.

If we load up the SysMon driver (SysmonDrv.sys) into IDA we can determine how exactly SysMon collects the call stack. The key function to look for is RtlWalkFrameChain and cross reference from there. SysMonDrv registers a callback (ObjectHandleCallback below) for process handle operations and on each invocation will call RtlWalkFrameChain to collect the user mode call stack (via the StackWalkWrapper function):

Figure 2: The decompilation generated by IDA for SysMonDrv’s object call back.

Note that SysMon uses a flag of 1 (‘mov r8d, 1’) in the call to RtlWalkFrameChain to indicate it wants to collect a user mode call stack only.

RtlWalkFrameChain is exported by ntoskrnl and (at a very high level) works as follows:

• Calls RtlCaptureContext to capture a ContextRecord / CONTEXT structure of the current thread
• Calls RtlpxVirtualUnwind which will take the CONTEXT structure and start unwinding the stack (based on the current state of execution recorded in the CONTEXT structure e.g. via Rip/Rsp etc.)

An example of the implementation of RtlVirtualUnwind can be found here: https://github.com/hzqst/unicorn_pe/blob/master/unicorn_pe/except.cpp#L773 and here: https://doxygen.reactos.org/d8/d2f/unwind_8c.html#a03c91b6c437066272ebc2c2fff051a4c.

Furthermore, ETW can also be configured to collect a stack trace (see: https://github.com/microsoft/krabsetw/pull/191). This again can be very useful for determining anomalous activity for a number of providers (For example, when applied to the Microsoft TI feed or looking for unbacked wininet calls). As a note, ETW collects the call stack in a slightly different way to the typical inline kernel callback approach demonstrated above. It queues an APC to the target thread first and then calls RtlWalkFrameChain. This is probably due to the fact that some ETW providers execute in an arbitrary thread context.

A quick look at the implementation of RtlVirtualUnwind reveals the (rather complicated looking) parsing of X64 unwind codes. Therefore, in order to understand how the call stack is walked via RtlVirtualUnwind, it is first necessary to understand a little bit about how code generation/execution works on X64. A full overview is beyond the scope of this blog post, but this excellent CodeMachine blog post contains everything that is needed to fully understand the techniques used in this blog post: https://codemachine.com/articles/x64_deep_dive.html.

As a brief recap, the CPU has no concept of a function, rather it is a higher level language abstraction. In x86, functions are implemented at the CPU level via using the frame pointer register (Ebp), which can be used as a reference to access local variables and arguments passed via the stack. Through following this chain of Ebp pointers (or function frames) it is possible to find the next stack frame up and hence walk the x86 stack.

In X64, things are more complicated as Rbp is no longer used as a frame pointer and so, alas, the method used above will not work. The key difference to understand is that X64 executables contain a new section called “.pdata”. This section is essentially a database containing every function in the executable and instructions (known as UNWIND_CODEs) as to how to “unwind” a given function in the event of an exception. “Unwind” here essentially means to reverse any operation it performed in its function prologue which modified the stack in some way (e.g. made space for local variables, pushed any non-volatile registers to the stack etc.). On X64, once a function has finished its prologue (and hence stack modifications), it does not modify the stack pointer until its epilogue reverses them, hence Rsp is static throughout the function body.

Some typical UNWIND_CODEs are:

• ALLOC_SMALL/LARGE (allocates a small/large amount of memory for local args e.g. sub rsp, 80h)
• PUSH_NONVOL (pushes a non-volatile register to the stack e.g. push rdi)

In windbg, the ‘.fnent’ command will parse this information for a specified function and display its unwind info, as demonstrated for kernelbase!OpenProcess below:

This shows that OpenProcess only has one unwind code, which is that it allocates a small area of memory on the stack. The total size of ‘UWOP_ALLOC_SMALL’ is calculated by multiplying the op info value by 8 and adding 8 (0xc * 8 + 8 =0x68). This can be confirmed by disassembling the first few bytes of kernelbase!OpenProcess (sub rsp, 68h):

A documented list of all the available UNWIND_CODES (and how to parse them) can be found here: https://docs.microsoft.com/en-us/cpp/build/exception-handling-x64?view=msvc-170

In order to walk the stack, windbg will calculate the total stack size for each function found by adding up the size of:

• any local variables
• any stack based parameters
• the return address (8 bytes)
• homing space
• stack space taken up by non-volatile registers

If we take a call to OpenProcess as an example:

The top entry, ntdll!NtOpenProcess (#00), is the current stack frame. The Child-SP value of 000000df`7d8fef88 is the value of Rsp after NtOpenProcess has finished its function prologue (i.e. the value of the stack pointer after any stack modifications NtOpenProcess needs to make have been performed).  The value of 8 in the ‘Memory’ column in the row below the current frame is the total stack size used by NtOpenProcess. Therefore, in order to calculate the Child-SP of the next frame, we can add the total stack size of the current frame (8) to the current Child-SP:

Note that NtOpenProcess has no unwind op codes (it doesn’t modify the stack) so the next Child-SP is simply found by skipping the return address pushed by the previous caller (KERNELBASE!OpenProcess). This is why its total stack size is listed as 8 bytes (e.g. the return address only).

This new Child-SP (000000df`7d8fef90) is the value of Rsp after KERNELBASE!OpenProcess has finished its function prologue. When KERNELBASE!OpenProcess calls ntdll!NtOpenProcess it will push its ret address on to the stack; therefore this return address will be the next value on the stack immediately after where its Child-SP points, as shown by Child-SP 01 in Figure 3.

This process can be repeated again for the next frame. Kernelbase!OpenProcess has a Child-SP of 000000df`7d8fef90 and its total stack utilisation is 0x70 bytes. Once again if we add them together we can get the next Child-SP for VulcanRaven!main:

This process is repeated until the debugger has completely walked the stack. Therefore, at a high level the stack walking process looks like this:

Figure 3: A diagram showing the stack walking process for X64.

The key take away relevant to this blog post is that by knowing the total stack size of a function, it is possible (without symbols) to follow this chain of child stack pointers and walk a call stack. This process will be mimicked in reverse when it comes to spoofing call stacks.

Having discussed the usefulness of call stack telemetry and given a brief overview of how unwinding call stacks works on x64, we can now pivot to the question this blog post addresses: is it possible to spoof a call stack so that when this collection takes place inline (say from within a kernel driver callback routine) a fake call stack is recorded?

The PoC in this blog post takes the following approach:

1. Identify a target call stack to spoof. For this SysMon was used and an arbitrary entry for event type 10 that opened a handle to lsass was chosen e.g. the example below:

2. For each return address in the target call stack above, parse its unwind codes and calculate the total stack space required so we can locate the next childSP frame.

3. Create a suspended thread and modify the CONTEXT structure so that the stack/rsp fits the exact outline of the target call stack to spoof (without any of the actual data being there). Hence we are initialising the thread’s state to “fit the profile” of another thread by pushing fake return addresses and subtracting correct child-SP offsets (e.g. stack unwinding in reverse). Care needs to be taken when handling certain unwind codes (UWOP_SET_FPREG) as this will reset rsp == rbp.

4. Modify the CONTEXT structure so that Rip points to our target function (ntdll!NtOpenProcess in this case) and set the appropriate arguments required by the x64 calling convention (e.g. via setting Rcx/Rdx/R8/R9).

5. Resume the thread and handle the inevitable error once the sys call returns (as it is returning up a fake call stack) via a vectored exception handler. From this exception handler, we can re-direct the thread to RtlExitUserThread (via re-setting Rip)  and let it gracefully exit.

For 5), this approach is an obvious limitation; a better approach would be to use VEH exception handling with either hardware or software breakpoints in a similar manner to this patchless AMSI bypass: https://gist.github.com/CCob/fe3b63d80890fafeca982f76c8a3efdf.

With this approach, we could place a breakpoint on the ret immediately after the NtOpenProcess sys call (00007ff8`7ca6d204 below) has returned:

Once a breakpoint exception is generated (and before the thread returns and crashes) we could handle the error in the same way as discussed previously. Additionally, recovering the state of the fake thread and being able to re-use it would be an improvement and stop the need to repeatedly create “sacrificial threads”.

Furthermore, this approach could also potentially be applied to the sleeping obfuscation problem; a fake thread with a legitimate call stack could be initialised to call ntdll!NtDelayExecution (or WaitForSingleObject etc.) and VEH exceptions used to re-direct flow to a main beacon function on return of the sleep duration.

The PoC code can be found here: https://github.com/countercept/CallStackSpoofer

The PoC comes with three example call stacks (wmi/rpc/svchost) to mimic, all of which were taken arbitrarily from SysMon logs via observing process handle access to lsass. These call stack profiles can be selected via the ‘–wmi’, ‘–rpc’, and ‘–svchost’ flags, as demonstrated below:

Figure 4: A screenshot showing VulcanRaven grabbing a handle to lsass and spoofing the call stack to look like RPC activity.

The screenshot above demonstrates a fake call stack being recorded by SysMon (contrast this to the expected use of OpenProcess which would have a call stack of: VulcanRaven.exe -> kernelbase!OpenProcess -> ntdll!NtOpenProcess). Just to stress, the examples in this PoC were chosen to mimic events found via SysMon but the call stack does not have to make sense and can be anything, as demonstrated below:

Figure 5: A screenshot from WinDbg showing a completely nonsensical call stack being spoofed when calling NtOpenProcess.

The most obvious example of why this technique would be of interest to an attacker is that most remote access trojans (beacon etc.) still tend to operate off floating / unbacked memory. Thus, when an attacker injects mimikatz directly into memory, the call stack for the subsequent handle access from this injected code will look highly anomalous.

As an example, a SysMon event is shown below for unbacked memory calling OpenProcess:

Figure 6: A SysMon event showing handle access to lsass originating from unbacked memory.

This was generated via using a modified version of Stephen Fewer’s ReflectiveDLLInjection https://github.com/stephenfewer/ReflectiveDLLInjection codebase.

In this example, a reflective DLL has been injected into cmd.exe and subsequently obtained a handle to lsass with PROCESS_ALL_ACCESS access. As the call originated from unbacked memory, SysMon records the last entry in the call stack as “UNKNOWN” (e.g. the last return address in the stack walk belonged to floating/unbacked code and not a legitimately loaded module) and hence is obviously suspicious.

However, if we modify the VulcanRaven PoC above to run as a reflective DLL, we generate the following event:

Figure 7: A screenshot showing Vulcan Raven modified to run as reflective DLL. Even though it is running from unbacked memory, the call stack for handle access to lsass is still spoofed to look legimate

The call stack (“CallTrace”) is spoofed to an arbitrary value pulled from SysMon as expected; it is impossible to tell from this call stack that the call to NtOpenProcess/OpenProcess actually originated from code running from unbacked memory, and the thread on the surface looks genuine (although the cmd.exe is contrived and obviously suspicious). Also note the different GrantedAccess to Figure 1, which in this case is PROCESS_ALL_ACCESS/0x1FFFFF.

It is clear though that an attacker could profile their chosen call stack to blend in with their chosen injected process (wmi, procexp, svchost etc. all regularly grab handles to lsass).

William Burgess