Step 2 – preparing the key

There are two ways to prepare the encryption key. As you may know, the encryption keys for these algorithms are usually of a fixed size (112 bits or 128 bits, and so on). Here are the steps the malware authors commonly take to prepare the key:

1. First, the author uses their plain text key and hashes it using any of the known hashing algorithms, such as MD5, SHA128, SHA256, or others:

CryptCreateHash(hProv,CALG_MD5,0,0,&hHash);
CryptHashData(hHash,secretkey,secretkeylen,0);

2. Then, they create a session key from this hash using CryptDeriveKey—for example, CryptDeriveKey(hProv,CALG_3DES,hHash,0,&hKey);. From here, they can easily identify the algorithm from the second argument value that's provided to this API. The most common algorithms/values are as follows:

CALG_DES = 0x00006601,// DES encryption algorithm.
CALG_3DES = 0x00006603,// Triple DES encryption algorithm.
CALG_AES = 0x00006611,// Advanced Encryption Standard (AES).
ALG_RC4...