A subtle yet potent denial-of-service (DoS) vulnerability, dubbed "HollowByte" by Okta’s Red Team, has been revealed, affecting unpatched OpenSSL servers where a mere eleven bytes of attacker-controlled data can compel a server to reserve up to 131 KB of memory for a message that never fully materializes. This allocated memory, particularly on systems leveraging glibc (GNU C Library) for memory management, remains inaccessible until the affected process is restarted, leading to a persistent memory leak and potential system instability. The revelation by Okta on Thursday, July 18, 2024, has cast a spotlight on OpenSSL’s decision to integrate the fix for this critical issue without a Common Vulnerabilities and Exposures (CVE) identifier, a public advisory, or even a specific changelog entry, making it exceptionally challenging for system administrators and downstream distributors to identify and address the threat.
The HollowByte Vulnerability: A Deeper Dive into its Mechanics
At its core, the HollowByte vulnerability exploits a fundamental trust mechanism in older versions of OpenSSL’s TLS (Transport Layer Security) handshake process. Every TLS handshake message initiates with a 4-byte header. Crucially, three of these bytes are dedicated to declaring the anticipated length of the message body. Prior to the patch, vulnerable OpenSSL versions would immediately expand their receive buffer to accommodate this declared size upon receiving the header, even before a single byte of the actual message body had arrived and critically, before the handshake’s own internal validation checks could run.
For an inbound ClientHello message, a standard part of initiating a TLS connection, the maximum size ceiling is approximately 131 KB. An attacker can leverage this by sending only the initial 4-byte header, declaring a large body size up to this 131 KB limit, and then abruptly terminating the connection without ever sending the proclaimed message body. This forces the server’s worker thread to block indefinitely, awaiting data that will never arrive, tying up resources. The critical distinction of HollowByte from traditional connection-exhaustion attacks, such as Slowloris, lies in how the memory is handled post-disconnection.
The Crucial Role of Glibc and Persistent Memory Fragmentation
While typical connection-exhaustion attacks aim to overwhelm a server by exhausting its available connection slots, HollowByte introduces a more insidious problem: memory fragmentation and persistent consumption. When the attacker severs the connection, OpenSSL does attempt to free the allocated buffer. However, the interaction with glibc, the ubiquitous C standard library found on most Linux systems, prevents this memory from being truly returned to the operating system for immediate reuse.
Glibc employs sophisticated memory management strategies, often holding onto small and medium-sized memory chunks in various arenas for future reuse rather than immediately releasing them back to the kernel. This optimization is designed to improve performance by reducing the overhead of repeated system calls for memory allocation. In the context of HollowByte, the attack’s efficacy stems from varying the claimed message size across multiple, successive connections. Okta’s testing demonstrated that this variation in requested buffer sizes was sufficient to confuse glibc’s allocator, preventing it from effectively reusing the "freed" chunks.
The result is a phenomenon known as heap fragmentation. The server’s memory becomes littered with numerous small, unusable blocks of memory, effectively reducing the contiguous memory available to the application and the system. The resident set size (RSS) – the portion of memory held in RAM by a process – climbs steadily and remains elevated long after the attacker has ceased their activity. This "frozen" memory is not released until the entire OpenSSL process (or the service using it, like NGINX) is restarted, necessitating manual intervention and leading to prolonged degradation of service.
Impact and Scale of the Attack
Okta’s testing provided concrete figures illustrating the severity of the HollowByte attack. In one scenario involving an NGINX server with 1 GB of RAM, the system was subjected to an Out-Of-Memory (OOM) kill, with 547 MB of its memory permanently consumed by fragments. On a more robust 16 GB server, the attack managed to lock up 25% of the total system memory without ever hitting the server’s configured connection limit. This critical observation led Okta’s Red Team to conclude that "standard connection-limiting defenses won’t stop it," as the attack doesn’t rely on exhausting the number of concurrent connections but rather on depleting available memory through repeated, subtle fragmentation.
The absence of public exploit code or proof-of-concept repositories on platforms like GitHub as of July 18, 2024, suggests that the immediate risk of widespread, automated exploitation might be mitigated. However, the simplicity of the attack vector (11 bytes) means that a motivated attacker could quickly develop and deploy such tools, making the lack of official identification even more concerning.
OpenSSL’s Controversial Handling and Lack of Transparency
The most contentious aspect of the HollowByte saga is OpenSSL’s decision regarding its disclosure and classification. The fix, integrated into OpenSSL releases 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21, all dated June 9, was implemented by Matt Caswell, a prominent OpenSSL developer. His pull request, specifically for this patch, explicitly states the security team’s choice to "handle this as a ‘bug or hardening’ only fix."
OpenSSL’s official security policy delineates four severity tiers for vulnerabilities: Critical, High, Moderate, and Low. Notably, "bug or hardening" is not one of these categories. Even issues classified as "Low" typically warrant a CVE identifier, an entry in the project’s changelog, and a listing on the dedicated vulnerabilities page. HollowByte received none of these standard recognitions. The Hacker News confirmed the absence of any mention of this fix in the release notes or the 23 entries of OpenSSL’s 4.0.1 changelog.
This lack of transparency poses significant challenges. Without a CVE, security scanners cannot automatically detect the vulnerability. Without an advisory or changelog entry, system administrators have no official notification that a security-relevant update has been applied, or even that such a vulnerability existed.
OpenSSL has not publicly provided a detailed explanation for this specific classification. Potential arguments for their decision might include the bounded nature of the allocation (131 KB per connection is finite, unlike unbounded growth), and the general understanding that TLS servers inherently allocate memory per connection. From this perspective, a bounded allocation might not be seen as a true "vulnerability" in the traditional sense. However, Okta’s counter-argument, emphasizing that the memory never returns, fundamentally alters this calculus, transforming a bounded allocation into a persistent resource exhaustion mechanism.
Comparing to Other OpenSSL Vulnerabilities
The decision to triage HollowByte below the "Low" severity tier appears inconsistent when compared to OpenSSL’s past vulnerability classifications. For example, in January, OpenSSL assigned CVE-2025-66199 (rated Low) to a TLS 1.3 certificate-compression bug. This flaw involved a peer-supplied length growing a heap buffer before validation, potentially consuming around 22 MiB per connection. While requiring specific conditions (certificate compression compiled in, available algorithm, extension negotiated, client certificates requested on servers), it was still deemed worthy of a CVE. HollowByte, in contrast, requires none of these specific conditions, making it arguably simpler to exploit.
Furthermore, the same June 9 release that contained the HollowByte fix also assigned CVE-2026-34183 (rated Moderate) to an unbounded memory growth vulnerability within the QUIC PATH_CHALLENGE handler. Both this and HollowByte are memory-exhaustion DoS attacks, yet one received a CVE and a moderate rating, while the other was quietly integrated as a "bug fix." This disparity raises questions about the consistency and transparency of OpenSSL’s vulnerability management process. The June 9 release also notably closed 18 other CVEs, including a High-severity use-after-free bug in PKCS7_verify(), ensuring that users upgrading to these versions received these critical fixes, albeit without specific notice for HollowByte.
Challenges for Patching and Downstream Distributors
The implications of OpenSSL’s non-disclosure extend significantly to downstream distributors and system administrators. Major Linux distributions like Red Hat often employ a "backporting" strategy, where security fixes are applied to older, stable versions of software packages rather than forcing a full version upgrade. This means a patched OpenSSL package on a Red Hat system might still report its original, older version number.
Normally, the presence of a CVE identifier and corresponding security advisories (often linked to OVAL feeds) would enable distributors to clearly communicate the specific vulnerabilities addressed and allow automated tools to verify patch status. In the absence of a CVE for HollowByte, this standard process breaks down entirely. System administrators cannot rely on typical patch pipelines, vulnerability scanners, or official advisories to identify if their systems are vulnerable or have been patched.
Instead, they are forced into a manual, labor-intensive process: either consulting the package changelog (which may not explicitly mention the "bug or hardening" fix) or directly contacting maintainers to inquire whether their builds rebased on the June 9 releases or specifically incorporated the relevant pull requests. These pull request identifiers are 30792 for master and 4.0, 30793 for 3.6, 3.5, and 3.4, and 30794 for 3.0. Organizations that compile OpenSSL from source must proactively upgrade to the specified patched releases and ensure all services linking against the old library are restarted.
The Unaddressed DTLS Flaw
Adding another layer of concern, the fix for HollowByte explicitly covers only TLS. Matt Caswell noted in the pull request that addressing the same vulnerability in DTLS (Datagram Transport Layer Security) would have been "far more invasive," leading the project to defer a fix for it. Analysis by The Hacker News revealed that the DTLS handshake file remained byte-identical between OpenSSL 3.6.2 and 3.6.3 (the patched version), confirming that the vulnerability persists in DTLS. Even in OpenSSL 4.0.1, the newest release, the DTLS path continues to size its buffer based on the length declared by the peer, leaving it vulnerable to the same memory exhaustion attack.
OpenSSL has not classified this lingering DTLS vulnerability, nor has it committed to a timeline for its remediation. As with the TLS fix, no mention of this unaddressed DTLS path appears in release notes, changelogs, or the vulnerabilities page, with the only public acknowledgement residing within the pull request comments. This omission means that DTLS-based applications, which are often used in latency-sensitive or unreliable network environments, remain exposed to a known, unfixed memory exhaustion attack.
OpenSSL’s Role in Internet Security and Disclosure Standards
OpenSSL is a cornerstone of internet security, providing the cryptographic functions that underpin countless websites, applications, and services. Its ubiquity means that vulnerabilities, even seemingly minor ones, can have widespread repercussions. The project operates as an open-source endeavor, relying on community contributions and a dedicated core team. The transparency of its security advisories and patching processes is crucial for maintaining trust and enabling rapid, effective responses to threats across the global digital infrastructure.
The handling of HollowByte raises important questions about OpenSSL’s vulnerability disclosure policy and its commitment to transparency. While the project has a history of addressing critical flaws and providing detailed CVEs, the decision to quietly patch a DoS vulnerability that bypasses standard defenses and causes persistent memory leaks could erode confidence among its user base and security researchers.
Responsible disclosure typically involves public advisories, clear identification (like CVEs), and detailed changelogs to ensure that all stakeholders are aware of risks and can take appropriate action. Deviating from these established norms, especially for a widely used library like OpenSSL, introduces unnecessary risk and burden onto the very community it serves.
Recommendations and Mitigation
For system administrators and organizations utilizing OpenSSL, immediate action is paramount:
- Upgrade OpenSSL: The most direct mitigation is to upgrade to the latest patched versions: OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, or 3.0.21. Ensure that any services or applications linking against OpenSSL are restarted after the upgrade to load the new library.
- Verify Patch Application: Due to the lack of a CVE, verify that the specific patches corresponding to pull requests
30792,30793, or30794have been applied, especially if relying on distribution-specific packages. Consult package changelogs or contact maintainers directly. - Monitor Memory Usage: Implement robust monitoring for memory consumption on TLS-serving processes. Anomalous, persistent increases in resident set size (RSS) that do not correlate with increased connection counts could indicate an ongoing HollowByte attack.
- Consider DTLS Implications: If using DTLS-based applications with OpenSSL, be aware that the HollowByte vulnerability remains unaddressed in this protocol. Implement additional application-level protections or explore alternative solutions if DTLS is critical to your infrastructure.
Ongoing Questions and Future Implications
As of the latest information, "The Hacker News" has reached out to OpenSSL for clarification on why HollowByte was triaged below "Low" and whether the fix will extend to the extended-support 1.1.1 and 1.0.2 branches. Inquiries have also been directed to Okta regarding whether the memory fragmentation effect survives with allocators other than glibc, which could broaden or narrow the scope of affected systems. The answers to these questions will be crucial in fully understanding the vulnerability’s impact and OpenSSL’s long-term strategy for addressing such subtle yet impactful flaws.
The HollowByte incident serves as a critical reminder of the complexities of securing fundamental internet infrastructure and the ongoing challenges faced by open-source projects in balancing rapid development with transparent and consistent security disclosure practices. The implications extend beyond just this single vulnerability, potentially influencing how future "bug or hardening" fixes are communicated and managed within the broader cybersecurity ecosystem.
