Series Overview
This series is intended for readers who are interested in reverse engineering but have only opened a debugger a handful of times. If you have trouble with certain concepts of reverse engineering, tooling, disassembly or debugging then you’ve come to the right place. Starting from the ground up we’ll work our way to advanced topics that aid in automating the reversal process such as heuristic analysis using a disassembly engine and return-oriented programming. If you’re new it’s recommended you start from the first article and work your way through the series, as it’s meant to guide you through the intricacies of the architecture and operating system structures. This series does expect the reader to have prerequisite knowledge of a native programming language such as C, C++, Rust, etc. Native meaning compiled to a native machine language, as opposed to interpreted. I do not cover reverse engineering Java Byte Code. If you don’t have a background in a compiled programming language this series may be confusing and esoteric. Otherwise, you’re in good hands!
This series is written for reverse engineering on a 64-bit Windows OS. Windows 10 will be the OS that the author is working in, and all examples will be relevant to Windows 10 and the Intel64/AMD64 architecture. You’ll certainly be able to take what you learn from this series and apply it to other architectures and operating systems, however, you’ll have to adapt to any changes present on those platforms. Also worth noting that I will address 64-bit Assembly in detail with a small subsection regarding 16-bit and 32-bit assembly to help solidify the readers’ understanding of x64 Assembly.
All that being said, if you’re familiar with reverse engineering and interested in a specific topic then feel free to skip around, and visit the sections you find most interesting! It’s by no means linear, but if you’re starting out then going in order will be much less confusing.
Note: The documentation referenced will be the Intel and AMD SDM, among other books, articles, and blogs.
I’ve decided on this series that, in order to reduce the length of my articles, I’m going to cover topics in their own separate post. They will be linked here so they’re easy to find from the main navigation bar on the left side of the site.
Index
— The Foundation
ð Basic Architecture – This article will cover general-purpose registers, RFLAGS, data types, and a few examples of assembly at work using status flags.
ð The Stack – Addresses the stack layout, stack operations, the purpose of a stack, calling conventions, alignment, and stack faults.
ð Exceptions and Interrupts – The basics of exceptions, how software-generated and hardware generated exceptions are handled, the most common exceptions, SEH, VEH, and the role of the OS.
ð Accelerated Assembly Part 1 | Part 2 – Covering x64 assembly from simple operations, conditional compares and jumps, to bit shifting and stack manipulation.
— High-Level Structure Redefinition
Following learning how to identify key instruction sequences you’ll refine your skills by learning how to identify structure accesses in IDA Pro and x64dbg. You’ll learn how to take the low-level operations and construct a high-level view of the structure being modified. We won’t be using a decompiler until after we’ve covered how to do it from the plain and dry dead-listing. Once we get to using the decompiler I’ll illustrate how to transfer that high-level reconstruction to IDA to build a more accurate and understandable representation of what the code is doing.
— Heuristic Analysis
Learning heuristic analysis and how to write your own analytical processes can make reverse engineering much easier by determining how a specific process interacts with the OS and executes in general. It can reduce the amount of time required to hunt for pointers after updates, or aid in decryption during runtime operations. In this section, now having a background on reverse engineering, we’ll leverage a disassembly library to perform heuristic analysis on an example application. We’ll develop a tool that uses heuristics to automate the identification of code sequences, and perform behavioral analysis based on how the application interacts with the OS. Knowing how to perform heuristic analysis on a target is an invaluable skill for any reverse engineer.
— Hooking Techniques
There’s a crazy amount of ways to hook something, so in this article, we’ll cover the 3 major techniques used for hooking functions in applications. If you’re unsure what a hook is, don’t worry we’ll address that too as well as it’s purpose. We’ll discuss a more advanced hooking technique provided with an example and further reading material for those interested. In the section covering reverse engineering classes and the vtable we’ll introduce one more hooking technique – usually described as easier than the others.
— Atomic Operations
What are atomic operations? How do they work? How do we identify them in assembly? In this article, I’ll answer all of those questions and include a small excerpt on how atomic operations work on the microarchitecture level – for all the detail lovers.
— Classes, Virtual Functions, and Inheritance
C++ is a popular language that many games and applications are written in. Knowing this we’re going to leverage your new-found knowledge and add one more skill to your reverse engineering tool-chain – reverse engineering object-oriented code. This involves being able to identify and reconstruct classes, virtual functions, globals, class construction, and how to reconstruct the inheritance hierarchy from bare-bones disassembly. I’ll introduce a hooking technique commonly used against C++ applications, but that will require an explanation of what the vtable is. This is gonna be a long one.
— Return Oriented Programming
Ever heard of return-oriented programming (ROP, for short)? If not, it’s covered in great detail in this section. This means you’ll learn how to apply it to a variety of applications, how it works on a fundamental level, and it’s common usage in exploitation. This is a very useful skill to have for reverse engineers and vulnerability researchers, whatever your skill level. We’ll cover a specific example where I used return-oriented programming to aid in the spoofing of return addresses in an effort to stay hidden from a monitoring service. And if this article isn’t enough there will be a load of supplementary material provided including a personal example used on an old game modding project.
— Game Example
Whether you’re new or experienced with reverse engineering it’s always neat to see how others approach the same problem. In this article, we’ll be reverse engineering a game, and applying what we’ve learned in this series to create a trainer that offers various features. But we need to keep it simple so we’ll only be manipulating player data to get us out ahead of the pack. There are a variety of other ways to put your skills to use, but who doesn’t love games? It will utilize a lot of your skillset and we’ll break down the thought processes when approaching a new project.
— Virtualization and Reverse Engineering
This post will not be written until the EPT series is complete, however, I plan to explain how a hypervisor can be utilized in a research scenario. This article will cover EPTP switching, page hooks, #VE, and EPT violations, and implementing a syscall hook as described in the article Syscall Hooking via Extended Feature Enable Register. We’ll essentially create our own sandboxed environment, run an author created malware, and log the Windows APIs the malware invokes along with their parameters and stack information. This will take you full circle from our CPU virtualization and EPT series back to applying it with reverse engineering. The possibilities are endless.
4 thoughts on “Applied Reverse Engineering Series”