Sign In Start Free Trial

Mastering Malware Analysis

By : Alexey Kleymenov, Amr Thabet

4.5 (10)

Mastering Malware Analysis

4.5 (10)

By: Alexey Kleymenov, Amr Thabet

Overview of this book

With the ever-growing proliferation of technology, the risk of encountering malicious code or malware has also increased. Malware analysis has become one of the most trending topics in businesses in recent years due to multiple prominent ransomware attacks. Mastering Malware Analysis explains the universal patterns behind different malicious software types and how to analyze them using a variety of approaches. You will learn how to examine malware code and determine the damage it can possibly cause to your systems to ensure that it won't propagate any further. Moving forward, you will cover all aspects of malware analysis for the Windows platform in detail. Next, you will get to grips with obfuscation and anti-disassembly, anti-debugging, as well as anti-virtual machine techniques. This book will help you deal with modern cross-platform malware. Throughout the course of this book, you will explore real-world examples of static and dynamic malware analysis, unpacking and decrypting, and rootkit detection. Finally, this book will help you strengthen your defenses and prevent malware breaches for IoT devices and mobile platforms. By the end of this book, you will have learned to effectively analyze, investigate, and build innovative solutions to handle any malware incidents.

Preface

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

Free Chapter

Section 1: Fundamental Theory

Section 1: Fundamental Theory

A Crash Course in CISC/RISC and Programming Basics

A Crash Course in CISC/RISC and Programming Basics

Basic concepts

Registers

Memory

Virtual memory

Stack

Branches, loops, and conditions

Exceptions, interrupts, and communicating with other devices

Assembly languages

CISC versus RISC

Types of instructions

Becoming familiar with x86 (IA-32 and x64)

Registers

Special registers

The instruction structure

opcode

dest

src

The instruction set

Data manipulation instructions

Data transfer instructions

Flow control instructions

Arguments, local variables, and calling conventions (in x86 and x64)

stdcall

Arguments

Local variables

cdecl

fastcall

thiscall

The x64 calling convention

Exploring ARM assembly

Basics

Instruction sets

Basics of MIPS

Basics

The instruction set

Diving deep into PowerPC

Basics

The instruction set

Covering the SuperH assembly

Basics

The instruction set

Working with SPARC

Basics

The instruction set

Moving from assembly to high-level programming languages

Arithmetic statements

If conditions

While loop conditions

Summary

Section 2: Diving Deep into Windows Malware

Section 2: Diving Deep into Windows Malware

Basic Static and Dynamic Analysis for x86/x64

Basic Static and Dynamic Analysis for x86/x64

Working with the PE header structure

Why PE?

Exploring PE structure

MZ header

PE header

File header

Optional header

Data directory

Section table

PE+ (x64 PE)

PE header analysis tools

Static and dynamic linking

Static linking

Dynamic linking

Dynamic link libraries

Application programming interface

Dynamic API loading

Using PE header information for static analysis

How to use PE header for incident handling

How to use a PE header for threat intelligence

PE loading and process creation

Basic terminology

What's process?

Virtual memory to physical memory mapping

Threads

Important data structures: TIB, TEB, and PEB

Process loading step by step

PE file loading step by step

WOW64 processes

Dynamic analysis with OllyDbg/Immunity Debugger

Debugging tools

How to analyze a sample with OllyDbg

Types of breakpoints

Step into/step over breakpoint

INT3 breakpoint

Memory breakpoints

Hardware breakpoints

Modifying the program execution

Patching—modifying the program's assembly instructions

Change EFlags

Modifying the instruction pointer value

Changing the program data

Debugging malicious services

What is service?

Attaching to the service

Summary

Unpacking, Decryption, and Deobfuscation

Unpacking, Decryption, and Deobfuscation

Exploring packers

Exploring packing and encrypting tools

Identifying a packed sample

Technique 1 – checking PE tool static signatures

Technique 2 – evaluating PE section names

Technique 3 – using stub execution signs

Technique 4 – detecting a small import table

Automatically unpacking packed samples

Technique 1 – the official unpacking process

Technique 2 – using OllyScript with OllyDbg

Technique 3 – using generic unpackers

Technique 4 – emulation

Technique 5 – memory dumps

Manual unpacking using OllyDbg

Technique 6 – memory breakpoint on execution

Step 1 – setting the breakpoints

Step 2 – turning on Data Execution Prevention

Step 3 – preventing any further attempts to change memory permissions

Step 4 – executing and getting the OEP

Technique 7 – call stack backtracing

Step 1 – setting the breakpoints

Step 2 – following the call stack

Step 3 – reaching the OEP

Technique 8 – monitoring memory allocated spaces for unpacked code

Technique 9 – in-place unpacking

Technique 10 – stack restoration-based

Dumping the unpacked sample and fixing the import table

Dumping the process

Fixing the import table

Identifying different encryption algorithms and functions

Types of encryption algorithms

Basic encryption algorithms

How to identify encryption functions

String search detection techniques for simple algorithms

The basics of X-RAYING

Simple static encryption

Other encryption algorithms

X-RAYING tools for malware analysis and detection

Identifying the RC4 encryption algorithm

The RC4 encryption algorithm

Key-scheduling algorithm

Pseudo-random generation algorithm

Identifying RC4 algorithms in a malware sample

Standard symmetric and asymmetric encryption algorithms

Extracting information from Windows cryptography APIs

Step 1 – initializing and connecting to the cryptographic service provider (CSP)

Step 2 – preparing the key

Step 3 – encrypting or decrypting the data

Step 4 – freeing the memory

Cryptography API next generation (CNG)

Applications of encryption in modern malware – Vawtrak banking Trojan

String and API name encryption

Network communication encryption

Using IDA for decryption and unpacking

IDA tips and tricks

Static analysis

Dynamic analysis

Classic and new syntax of IDA scripts

Dynamic string decryption

Dynamic WinAPIs resolution

Summary

Inspecting Process Injection and API Hooking

Inspecting Process Injection and API Hooking

Understanding process injection

What's process injection?

Why process injection?

DLL injection

Windows-supported DLL injection

A simple DLL injection technique

Working with process injection

Getting the list of running processes

Code injection

Advanced code injection-reflective DLL injection

Stuxnet secret technique-process hollowing

Dynamic analysis of code injection

Technique 1—debug it where it is

Technique 2—attach to the targeted process

Technique 3—dealing with process hollowing

Memory forensics techniques for process injection

Technique 1—detecting code injection and reflective DLL injection

Technique 2—detecting process hollowing

Technique 3—detecting process hollowing using the HollowFind plugin

Understanding API hooking

Why API hooking?

Working with API hooking

Inline API hooking

Inline API hooking with trampoline

Inline API hooking with a length disassembler

Detecting API hooking using memory forensics

Exploring IAT hooking

Summary

Bypassing Anti-Reverse Engineering Techniques

Bypassing Anti-Reverse Engineering Techniques

Exploring debugger detection

Direct check for debugger presence

Detecting a debugger through an environment change

Detecting a debugger using parent processes

Handling debugger breakpoints evasion

Detecting software breakpoints (INT3)

Detecting single-stepping breakpoints (trap flag)

Detecting a trap flag using the SS register

Detecting single-stepping using timing techniques

Evading hardware breakpoints

What is structured exception handling?

Detecting and removing hardware breakpoints

Memory breakpoints

Escaping the debugger

Process injection

TLS callbacks

Windows events callbacks

Obfuscation and anti-disassemblers

Encryption

Junk code insertion

Code transportation

Dynamic API calling with checksum

Proxy functions and proxy argument stacking

Detecting and evading behavioral analysis tools

Finding the tool process

Searching for the tool window

Detecting sandboxes and virtual machines

Different output between virtual machines and real machines

Detecting virtualization processes and services

Detecting virtualization through registry keys

Detecting virtual machines using PowerShell

Detecting sandboxes by using default settings

Other techniques

Summary

Understanding Kernel-Mode Rootkits

Understanding Kernel-Mode Rootkits

Kernel mode versus user mode

Protection rings

Windows internals

The infrastructure of Windows

The execution path from user mode to kernel mode

Rootkits and device drivers

What is a rootkit?

Types of rootkits

What is a device driver?

Hooking mechanisms

SSDT hooking

Hooking the SYSENTER entry function

Modifying SSDT in an x86 environment

Modifying SSDT in an x64 environment

Hooking SSDT functions

IRP hooking

Devices and major functions

Attaching to a device

Modifying the IRP response and setting a completion routine

DKOM

The kernel objects—EPROCESS and ETHREAD

How do rootkits perform an object manipulation attack?

Process injection in kernel mode

Executing the inject code using APC queuing

KPP in x64 systems (PatchGuard)

Bypassing driver signature enforcement

Bypassing PatchGuard—the Turla example

Bypassing PatchGuard—GhostHook

Disabling PatchGuard using the Command Prompt

Static and dynamic analysis in kernel mode

Static analysis

Tools

Tips and tricks

Dynamic and behavioral analysis

Tools

Monitors

Rootkit detectors

Setting up a testing environment

Setting up the debugger

Stopping at the driver's entry point

Loading the driver

Restoring the debugging state

Summary

Section 3: Examining Cross-Platform Malware

Section 3: Examining Cross-Platform Malware

Handling Exploits and Shellcode

Handling Exploits and Shellcode

Getting familiar with vulnerabilities and exploits

Types of vulnerabilities

Stack overflow vulnerability

Heap overflow vulnerabilities

The use-after-free vulnerability

Logical vulnerabilities

Types of exploits

Cracking the shellcode

What's shellcode?

Linux shellcode in x86-64

Getting the absolute address

Null-free shellcode

Local shell shellcode

Reverse shell shellcode

Linux shellcode for ARM

Null-free shellcode

Windows shellcode

Getting the Kernel32.dll's ImageBase

Getting the required APIs from Kernel32.dll

The download and execute shellcode

Static and dynamic analysis of exploits

Analysis workflow

Shellcode analysis

Exploring bypasses for exploit mitigation technologies

Data execution prevention (DEP/NX)

Return-oriented programming

Address space layout randomization

DEP and partial ASLR

DEP and full ASLR – partial ROP and chaining multiple vulnerabilities

DEP and full ASLR – heap spray technique

Other mitigation technologies

Analyzing Microsoft Office exploits

File structures

Compound file binary format

Rich text format

Office open XML format

Static and dynamic analysis of MS Office exploits

Static analysis

Dynamic analysis

Studying malicious PDFs

File structure

Static and dynamic analysis of PDF files

Static analysis

Dynamic analysis

Summary

Reversing Bytecode Languages: .NET, Java, and More

Reversing Bytecode Languages: .NET, Java, and More

The basic theory of bytecode languages

Object-oriented programming

Inheritance

Polymorphism

.NET explained

.NET file structure

.NET COR20 header

Metadata streams

How to identify a .NET application from PE characteristics

The CIL language instruction set

Pushing into stack instructions

Pulling out a value from the stack

Mathematical and logical operations

Branching instructions

CIL language to higher-level languages

Local variable assignments

Local variable assignment with a method return value

Basic branching statements

Loops statements

.NET malware analysis

.NET analysis tools

Static and dynamic analysis (with Dnspy)

.NET static analysis

.NET dynamic analysis

Patching a .NET sample

Dealing with obfuscation

Obfuscated names for classes, methods, and others

Encrypted strings inside the binary

The sample is obfuscated using an obfuscator

The essentials of Visual Basic

File structure

P-code versus native code

Common p-code instructions

Dissecting Visual Basic samples

Static analysis

P-code

Native code

Dynamic analysis

P-code

Native code

The internals of Java samples

File structure

JVM instructions

Static analysis

Dynamic analysis

Dealing with anti-reverse engineering solutions

Python—script language internals

File structure

Bytecode instructions

Analyzing compiled Python

Static analysis

Dynamic analysis

Summary

Scripts and Macros: Reversing, Deobfuscation, and Debugging

Scripts and Macros: Reversing, Deobfuscation, and Debugging

Classic shell script languages

Windows batch scripting

Bash

VBScript explained

Basic syntax

Static and dynamic analysis

Deobfuscation

Those evil macros inside documents

Basic syntax

Static and dynamic analysis

Besides macros

The power of PowerShell

Basic syntax

Static and dynamic analysis

Handling JavaScript

Basic syntax

Static and dynamic analysis

Anti-reverse engineering tricks

Behind C and C—even malware has its own backend

Things to focus on

Static and dynamic analysis

Other script languages

Where to start from

Questions to answer

Summary

Section 4: Looking into IoT and Other Platforms

Section 4: Looking into IoT and Other Platforms

Dissecting Linux and IoT Malware

Dissecting Linux and IoT Malware

Explaining ELF files

ELF structure

System calls

Filesystem

Network

Process management

Other

Syscalls in assembly

Common anti-reverse engineering tricks

Exploring common behavioral patterns

Initial delivery and lateral movement

Persistence

Privilege escalation

Interaction with the command and control server

Attacking stage

Static and dynamic analysis of x86 (32- and 64-bit) samples

Static analysis

File type detectors

Data carving

Disassemblers

Actual tools

Engines

How to choose

Dynamic analysis

Tracers

Network monitors

Debuggers

Binary emulators

Radare2 cheat sheet

Anti-reverse engineering techniques

Learning Mirai, its clones, and more

High-level functionality

Propagation

Weaponry

Self-defense

Later derivatives

Other widespread families

Static and dynamic analysis of RISC samples

ARM

MIPS

PowerPC

SuperH

SPARC

Handling other architectures

What to start from

Summary

Introduction to macOS and iOS Threats

Introduction to macOS and iOS Threats

Understanding the role of the security model

macOS

Security policies

Filesystem hierarchy and encryption

Directory structure

Encryption

Apps protection

Gatekeeper

App sandbox

Other technologies

iOS

System security

Data encryption and password management

Apps' security

File formats and APIs

Mach-O

Thin

Fat

Application bundles (.app)

Info.plist

macOS

iOS

Installer packages (.pkg)

Apple disk images (.dmg)

iOS app store packages (.ipa)

APIs

Static and dynamic analyses of macOS and iOS samples

Static analysis

Retrieving samples

Disassemblers and decompilers

Auxiliary tools and libraries

Dynamic and behavioral analysis

macOS

Debuggers

Monitoring and dynamic instrumentation

Network analysis

iOS

Installers and loaders

Debuggers

Dumping and decryption

Monitors and in-memory patching

Network analysis

Attack stages

Jailbreaks on demand

Penetration

Deployment and persistence

macOS

iOS

Action phase

macOS

iOS

Other attack techniques

macOS

iOS

Advanced techniques

Anti-reverse-engineering (RE) tricks

Misusing dynamic data exchange (DDE)

User hiding

Use of AppleScript

API hijacking

Rootkits for Mac—do they exist?

Analysis workflow

Summary

Analyzing Android Malware Samples

Analyzing Android Malware Samples

(Ab)using Android internals

File hierarchy

Android security model

Process management

Filesystem

App permissions

Security services

Console

To root or not to root?

Understanding Dalvik and ART

Dalvik VM (DVM)

Android runtime (ART)

APIs

File formats

DEX

ODEX

OAT

VDEX

ART

ELF

APK

Bytecode set

Malware behavior patterns

Attack stages

Penetration

Deployment

Action phase

Advanced techniques—investment pays off

Patching system libraries

Keylogging

Self-defense

Rootkits—get it covered

Static and dynamic analysis of threats

Static analysis

Disassembling and data extraction

Decompiling

Dynamic analysis

Android debug bridge

Emulators

Behavioral analysis and tracing

Debuggers

Analysis workflow

Summary

Other Books You May Enjoy

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

4.5 (10)

5 star

70%

4 star

20%

3 star

0

2 star

10%

1 star

0

Memory breakpoints

Memory breakpoints are used not to stop on specific instructions, but to stop when any instruction tries to read a specific part of memory or modifies it. This type of breakpoint is done by modifying the memory protection of this page of memory, either by making it non-accessible if the breakpoint is on accessing (or reading) this memory page or read-only if the breakpoint is on modifying (or writing) on this memory page.

They are accessible by right-clicking on Breakpoint | Memory, on access or Memory, on write, as shown in the following screenshot:

Figure 24: OllyDbg breakpoint menu

You may wonder why there is no memory on-execute using execute protection for memory, and the reason is that execute protection wasn't enforced until Windows 8. If you have your virtual machine running on Windows XP or Windows 7, I will show you how to enforce this protection and how to create memory breakpoints on execute in Chapter 3, Unpacking, Decryption, and Deobfuscation...

Search

Your notes and bookmarks