Advanced reverse engineering techniques in unpacking
2019-05-16, 15:00–15:40, Main Hall

There are a lot of packers/protectors used to hide the functionality of the software. Sometimes this software is legal, sometimes malicious. It is vital to be able to unpack such software for future investigation. But the main issue is that many commercial protections use different algorithms to make automation of unpacking difficult. We will discuss more advanced techniques that are powerful and can be used to break strong protection. We will talk about debugging without debugging API. Year, it's strange but it's real life.

During the debugging, we often talk about debugging API on windows or ptrace routine on Linux. These mechanisms are provided by OS developers. So it is strongly recommended to use them for user-mode debugging (debugging in ring3). But software protection systems can use a lot of techniques for detecting and preventing debugging.

In practical reverse engineering anti-anti debugging plugins can be used. The most famous of them:
- Phantom and StrongOD (for OllyDbg);
- ScyllaHide (for x64dbg, IDA Pro)

But such plugins can only protect from well-known detection algorithms. If some unknown technique will be used they will fail.
So we will talk about how to implement your own tracing/debugging engine without debugging API and hide such an engine from anti-debug. We will dive into kernel development and implement our engine from scratch.

Reverse engineer and security researcher.
Speaker at nonamecon 2018.