OpenSSL for Restless Souls

The Mystery of False Positives

Imagine you're trying to unlock a door with a key. You try your key and the door opens - success! But wait... when you step inside, you realize this isn't your house at all. That's essentially what can happen with OpenSSL decryption.

OpenSSL, the popular cryptographic tool, sometimes tells you a password worked when it actually didn't. This tutorial will explain why this happens and how to deal with it.

1. The Correct Way

Encrypting a File

First, let's see how to properly encrypt and decrypt a file with OpenSSL using AES-256-CBC encryption:

# Create a file with our secret message
$ echo -n "hello world" > ./toencrypt.txt

# Encrypt it using AES-256-CBC with a password
$ openssl enc -aes-256-cbc -a -in ./toencrypt.txt \
  -pass pass:`echo -n "correctpass" | shasum -a 256 - | sed -E 's/  -//'` \
  > ./encres.txt

Decrypting Successfully

Now let's decrypt it with the correct password:

$ openssl enc -aes-256-cbc -d -a -in ./encres.txt \
  -pass pass:`echo -n "correctpass" | shasum -a 256 - | sed -E 's/  -//'`

hello world

This works exactly as expected - we get back our original "hello world" message.

2. The Surprising Behavior

Testing Wrong Passwords

Now let's try something interesting. We'll write a script to test many wrong passwords and see what happens:

#!/bin/bash

# Encrypted file from our example
enc_file="./encres.txt"

# Loop through 0001 to 9999
for i in {1..9999}; do
    # Format password as wrongpassXXXX
    pass="wrongpass$(printf "%04d" $i)"
    
    # Generate SHA256 hash of password
    pass_hash=$(echo -n "$pass" | shasum -a 256 | sed -E 's/[[:space:]]+-//')
    
    # Attempt decryption, suppress error messages
    result=$(openssl enc -aes-256-cbc -d -a -in "$enc_file" -pass pass:"$pass_hash" 2>/dev/null)
    exit_code=$?
    
    # Check if OpenSSL reported successful decryption
    if [ $exit_code -eq 0 ]; then
        echo "========================================================"
        echo "False positive found with password: $pass"
        echo "Hexdump of decrypted output:"
        echo -n "$result" | hexdump -C
        echo "========================================================"
        echo
    fi
done

Unexpected Results

When running this script, you might see output like this:

========================================================
False positive found with password: wrongpass0030
Hexdump of decrypted output:
00000000  17 c6 30 30 02 9d 94 d1  7d af 12 a7 35 07 71     |..00....}...5.q|
0000000f
========================================================

========================================================
False positive found with password: wrongpass1194
Hexdump of decrypted output:
00000000  5b d9 d7 5b ea b9 2b d8  a5 cb e8 b9 f7 a9 e7     |[..[..+........|
0000000f
========================================================

These are false positives - OpenSSL reports success with wrong passwords! The decrypted output is garbage, but OpenSSL had no way to know it wasn't what we originally encrypted.

3. Why This Happens

Padding Oracle Mystery

OpenSSL uses PKCS#7 padding for block ciphers like AES. When decrypting, it checks if the padding is valid, not if the data makes sense.

With a wrong password but correct padding, OpenSSL happily decrypts the data, resulting in garbage output.

Probability of False Positives

For AES (block size 16 bytes), the chance of random padding being valid is about 1/256. With our 9999 attempts, we'd expect around 39 false positives (9999/256 ≈ 39).

In our test, we found 4 - within expected statistical variation.

The Math Behind It

When OpenSSL decrypts, it looks at the last byte to determine padding. Valid padding values are 1 to 16, each with probability 1/256.

The probability that a random decryption appears valid is:
P(valid padding) ≈ 1/256 ≈ 0.39%

4. How to Detect Real Success

Hexdump Analysis

Examine the raw bytes of decrypted output. Legitimate text will show readable ASCII characters on the right side:

# Compare good vs bad decryption
$ openssl enc -aes-256-cbc -d -a -in encres.txt -pass pass:correctpass | hexdump -C
00000000  68 65 6c 6c 6f 20 77 6f  72 6c 64                 |hello world|
0000000b

$ openssl enc -aes-256-cbc -d -a -in encres.txt -pass pass:wrongpass | hexdump -C
00000000  17 c6 30 30 02 9d 94 d1  7d af 12 a7 35 07 71     |..00....}...5.q|
0000000f

AI Verification (draft)

Train a small LLM model to analyze decrypted content and answer "Does this look like valid decryption?":

import transformers

# Load a pre-trained model
model = transformers.AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased")
tokenizer = transformers.AutoTokenizer.from_pretrained("distilbert-base-uncased")

def is_valid_decryption(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
    outputs = model(**inputs)
    prediction = outputs.logits.argmax().item()
    return bool(prediction)  # True for valid, False for invalid

Fine-tune the model on examples of valid vs invalid decryptions to improve accuracy.

Best Practice

Combine hexdump analysis with AI verification for most reliable results. The human eye can spot patterns while AI handles bulk verification.

Key Takeaways

False Positives Happen

OpenSSL can report success with wrong passwords due to valid padding

Verify Output

Always check decrypted content matches expected format

Statistics Matter

Expect about 1 false positive per 256 wrong password attempts

OpenSSL is a powerful tool, but like all tools, it's important to understand its quirks. Happy (and careful) encrypting!