Tag:

hollowbyte

OpenSSL Servers Vulnerable to "HollowByte" DoS Attack as Okta Discloses Quietly Patched Memory Leak Flaw.

by admin July 18, 2026

written by admin

A subtle yet potent denial-of-service (DoS) vulnerability, dubbed "HollowByte" by Okta’s Red Team, has been revealed, affecting unpatched OpenSSL servers where a mere eleven bytes of attacker-controlled data can compel a server to reserve up to 131 KB of memory for a message that never fully materializes. This allocated memory, particularly on systems leveraging glibc (GNU C Library) for memory management, remains inaccessible until the affected process is restarted, leading to a persistent memory leak and potential system instability. The revelation by Okta on Thursday, July 18, 2024, has cast a spotlight on OpenSSL’s decision to integrate the fix for this critical issue without a Common Vulnerabilities and Exposures (CVE) identifier, a public advisory, or even a specific changelog entry, making it exceptionally challenging for system administrators and downstream distributors to identify and address the threat.

The HollowByte Vulnerability: A Deeper Dive into its Mechanics

At its core, the HollowByte vulnerability exploits a fundamental trust mechanism in older versions of OpenSSL’s TLS (Transport Layer Security) handshake process. Every TLS handshake message initiates with a 4-byte header. Crucially, three of these bytes are dedicated to declaring the anticipated length of the message body. Prior to the patch, vulnerable OpenSSL versions would immediately expand their receive buffer to accommodate this declared size upon receiving the header, even before a single byte of the actual message body had arrived and critically, before the handshake’s own internal validation checks could run.

For an inbound ClientHello message, a standard part of initiating a TLS connection, the maximum size ceiling is approximately 131 KB. An attacker can leverage this by sending only the initial 4-byte header, declaring a large body size up to this 131 KB limit, and then abruptly terminating the connection without ever sending the proclaimed message body. This forces the server’s worker thread to block indefinitely, awaiting data that will never arrive, tying up resources. The critical distinction of HollowByte from traditional connection-exhaustion attacks, such as Slowloris, lies in how the memory is handled post-disconnection.

The Crucial Role of Glibc and Persistent Memory Fragmentation

While typical connection-exhaustion attacks aim to overwhelm a server by exhausting its available connection slots, HollowByte introduces a more insidious problem: memory fragmentation and persistent consumption. When the attacker severs the connection, OpenSSL does attempt to free the allocated buffer. However, the interaction with glibc, the ubiquitous C standard library found on most Linux systems, prevents this memory from being truly returned to the operating system for immediate reuse.

Glibc employs sophisticated memory management strategies, often holding onto small and medium-sized memory chunks in various arenas for future reuse rather than immediately releasing them back to the kernel. This optimization is designed to improve performance by reducing the overhead of repeated system calls for memory allocation. In the context of HollowByte, the attack’s efficacy stems from varying the claimed message size across multiple, successive connections. Okta’s testing demonstrated that this variation in requested buffer sizes was sufficient to confuse glibc’s allocator, preventing it from effectively reusing the "freed" chunks.

The result is a phenomenon known as heap fragmentation. The server’s memory becomes littered with numerous small, unusable blocks of memory, effectively reducing the contiguous memory available to the application and the system. The resident set size (RSS) – the portion of memory held in RAM by a process – climbs steadily and remains elevated long after the attacker has ceased their activity. This "frozen" memory is not released until the entire OpenSSL process (or the service using it, like NGINX) is restarted, necessitating manual intervention and leading to prolonged degradation of service.

Impact and Scale of the Attack

Okta’s testing provided concrete figures illustrating the severity of the HollowByte attack. In one scenario involving an NGINX server with 1 GB of RAM, the system was subjected to an Out-Of-Memory (OOM) kill, with 547 MB of its memory permanently consumed by fragments. On a more robust 16 GB server, the attack managed to lock up 25% of the total system memory without ever hitting the server’s configured connection limit. This critical observation led Okta’s Red Team to conclude that "standard connection-limiting defenses won’t stop it," as the attack doesn’t rely on exhausting the number of concurrent connections but rather on depleting available memory through repeated, subtle fragmentation.

The absence of public exploit code or proof-of-concept repositories on platforms like GitHub as of July 18, 2024, suggests that the immediate risk of widespread, automated exploitation might be mitigated. However, the simplicity of the attack vector (11 bytes) means that a motivated attacker could quickly develop and deploy such tools, making the lack of official identification even more concerning.

OpenSSL HollowByte Flaw Could Freeze Server Memory with 11-Byte TLS Requests

OpenSSL’s Controversial Handling and Lack of Transparency

The most contentious aspect of the HollowByte saga is OpenSSL’s decision regarding its disclosure and classification. The fix, integrated into OpenSSL releases 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21, all dated June 9, was implemented by Matt Caswell, a prominent OpenSSL developer. His pull request, specifically for this patch, explicitly states the security team’s choice to "handle this as a ‘bug or hardening’ only fix."

OpenSSL’s official security policy delineates four severity tiers for vulnerabilities: Critical, High, Moderate, and Low. Notably, "bug or hardening" is not one of these categories. Even issues classified as "Low" typically warrant a CVE identifier, an entry in the project’s changelog, and a listing on the dedicated vulnerabilities page. HollowByte received none of these standard recognitions. The Hacker News confirmed the absence of any mention of this fix in the release notes or the 23 entries of OpenSSL’s 4.0.1 changelog.

This lack of transparency poses significant challenges. Without a CVE, security scanners cannot automatically detect the vulnerability. Without an advisory or changelog entry, system administrators have no official notification that a security-relevant update has been applied, or even that such a vulnerability existed.

OpenSSL has not publicly provided a detailed explanation for this specific classification. Potential arguments for their decision might include the bounded nature of the allocation (131 KB per connection is finite, unlike unbounded growth), and the general understanding that TLS servers inherently allocate memory per connection. From this perspective, a bounded allocation might not be seen as a true "vulnerability" in the traditional sense. However, Okta’s counter-argument, emphasizing that the memory never returns, fundamentally alters this calculus, transforming a bounded allocation into a persistent resource exhaustion mechanism.

Comparing to Other OpenSSL Vulnerabilities

The decision to triage HollowByte below the "Low" severity tier appears inconsistent when compared to OpenSSL’s past vulnerability classifications. For example, in January, OpenSSL assigned CVE-2025-66199 (rated Low) to a TLS 1.3 certificate-compression bug. This flaw involved a peer-supplied length growing a heap buffer before validation, potentially consuming around 22 MiB per connection. While requiring specific conditions (certificate compression compiled in, available algorithm, extension negotiated, client certificates requested on servers), it was still deemed worthy of a CVE. HollowByte, in contrast, requires none of these specific conditions, making it arguably simpler to exploit.

Furthermore, the same June 9 release that contained the HollowByte fix also assigned CVE-2026-34183 (rated Moderate) to an unbounded memory growth vulnerability within the QUIC PATH_CHALLENGE handler. Both this and HollowByte are memory-exhaustion DoS attacks, yet one received a CVE and a moderate rating, while the other was quietly integrated as a "bug fix." This disparity raises questions about the consistency and transparency of OpenSSL’s vulnerability management process. The June 9 release also notably closed 18 other CVEs, including a High-severity use-after-free bug in PKCS7_verify(), ensuring that users upgrading to these versions received these critical fixes, albeit without specific notice for HollowByte.

Challenges for Patching and Downstream Distributors

The implications of OpenSSL’s non-disclosure extend significantly to downstream distributors and system administrators. Major Linux distributions like Red Hat often employ a "backporting" strategy, where security fixes are applied to older, stable versions of software packages rather than forcing a full version upgrade. This means a patched OpenSSL package on a Red Hat system might still report its original, older version number.

Normally, the presence of a CVE identifier and corresponding security advisories (often linked to OVAL feeds) would enable distributors to clearly communicate the specific vulnerabilities addressed and allow automated tools to verify patch status. In the absence of a CVE for HollowByte, this standard process breaks down entirely. System administrators cannot rely on typical patch pipelines, vulnerability scanners, or official advisories to identify if their systems are vulnerable or have been patched.

Instead, they are forced into a manual, labor-intensive process: either consulting the package changelog (which may not explicitly mention the "bug or hardening" fix) or directly contacting maintainers to inquire whether their builds rebased on the June 9 releases or specifically incorporated the relevant pull requests. These pull request identifiers are 30792 for master and 4.0, 30793 for 3.6, 3.5, and 3.4, and 30794 for 3.0. Organizations that compile OpenSSL from source must proactively upgrade to the specified patched releases and ensure all services linking against the old library are restarted.

The Unaddressed DTLS Flaw

Adding another layer of concern, the fix for HollowByte explicitly covers only TLS. Matt Caswell noted in the pull request that addressing the same vulnerability in DTLS (Datagram Transport Layer Security) would have been "far more invasive," leading the project to defer a fix for it. Analysis by The Hacker News revealed that the DTLS handshake file remained byte-identical between OpenSSL 3.6.2 and 3.6.3 (the patched version), confirming that the vulnerability persists in DTLS. Even in OpenSSL 4.0.1, the newest release, the DTLS path continues to size its buffer based on the length declared by the peer, leaving it vulnerable to the same memory exhaustion attack.

OpenSSL has not classified this lingering DTLS vulnerability, nor has it committed to a timeline for its remediation. As with the TLS fix, no mention of this unaddressed DTLS path appears in release notes, changelogs, or the vulnerabilities page, with the only public acknowledgement residing within the pull request comments. This omission means that DTLS-based applications, which are often used in latency-sensitive or unreliable network environments, remain exposed to a known, unfixed memory exhaustion attack.

OpenSSL’s Role in Internet Security and Disclosure Standards

OpenSSL is a cornerstone of internet security, providing the cryptographic functions that underpin countless websites, applications, and services. Its ubiquity means that vulnerabilities, even seemingly minor ones, can have widespread repercussions. The project operates as an open-source endeavor, relying on community contributions and a dedicated core team. The transparency of its security advisories and patching processes is crucial for maintaining trust and enabling rapid, effective responses to threats across the global digital infrastructure.

The handling of HollowByte raises important questions about OpenSSL’s vulnerability disclosure policy and its commitment to transparency. While the project has a history of addressing critical flaws and providing detailed CVEs, the decision to quietly patch a DoS vulnerability that bypasses standard defenses and causes persistent memory leaks could erode confidence among its user base and security researchers.

Responsible disclosure typically involves public advisories, clear identification (like CVEs), and detailed changelogs to ensure that all stakeholders are aware of risks and can take appropriate action. Deviating from these established norms, especially for a widely used library like OpenSSL, introduces unnecessary risk and burden onto the very community it serves.

Recommendations and Mitigation

For system administrators and organizations utilizing OpenSSL, immediate action is paramount:

Upgrade OpenSSL: The most direct mitigation is to upgrade to the latest patched versions: OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, or 3.0.21. Ensure that any services or applications linking against OpenSSL are restarted after the upgrade to load the new library.
Verify Patch Application: Due to the lack of a CVE, verify that the specific patches corresponding to pull requests 30792, 30793, or 30794 have been applied, especially if relying on distribution-specific packages. Consult package changelogs or contact maintainers directly.
Monitor Memory Usage: Implement robust monitoring for memory consumption on TLS-serving processes. Anomalous, persistent increases in resident set size (RSS) that do not correlate with increased connection counts could indicate an ongoing HollowByte attack.
Consider DTLS Implications: If using DTLS-based applications with OpenSSL, be aware that the HollowByte vulnerability remains unaddressed in this protocol. Implement additional application-level protections or explore alternative solutions if DTLS is critical to your infrastructure.

Ongoing Questions and Future Implications

As of the latest information, "The Hacker News" has reached out to OpenSSL for clarification on why HollowByte was triaged below "Low" and whether the fix will extend to the extended-support 1.1.1 and 1.0.2 branches. Inquiries have also been directed to Okta regarding whether the memory fragmentation effect survives with allocators other than glibc, which could broaden or narrow the scope of affected systems. The answers to these questions will be crucial in fully understanding the vulnerability’s impact and OpenSSL’s long-term strategy for addressing such subtle yet impactful flaws.

The HollowByte incident serves as a critical reminder of the complexities of securing fundamental internet infrastructure and the ongoing challenges faced by open-source projects in balancing rapid development with transparent and consistent security disclosure practices. The implications extend beyond just this single vulnerability, potentially influencing how future "bug or hardening" fixes are communicated and managed within the broader cybersecurity ecosystem.

July 18, 2026 0 comment

Cybersecurity & Hacking

HollowByte DDoS flaw bloats OpenSSL server memory with 11-byte payload

by admin July 18, 2026

written by admin

The Unseen Backbone: Understanding OpenSSL’s Critical Role

OpenSSL stands as an indispensable open-source cryptography library, serving as the bedrock for secure communication across the internet. It provides the implementation for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols, which encrypt data transmitted between web browsers and servers, email clients and servers, and countless other applications. From securing e-commerce transactions to protecting sensitive personal data, OpenSSL’s functions are pervasive, underpinning a vast array of digital services. Its widespread adoption spans popular web servers like NGINX and Apache, various programming language runtimes such as Node.js, Python, Ruby, and PHP, and critical databases including MySQL and PostgreSQL. Furthermore, it is a standard component pre-installed on most Linux distributions, responsible for managing TLS encryption, digital certificates, and cryptographic operations.

Given its ubiquitous presence, any vulnerability within OpenSSL carries profound implications for global cybersecurity. Historically, major flaws like the "Heartbleed" bug in 2014 demonstrated the catastrophic potential of OpenSSL vulnerabilities, exposing sensitive data from millions of servers worldwide. While HollowByte does not involve data theft or remote code execution, its ability to induce a denial-of-service condition through resource exhaustion is a severe operational threat, capable of disrupting critical services, causing significant financial losses, and eroding user trust.

Dissecting HollowByte: An 11-Byte Path to Server Paralysis

The mechanics of the HollowByte vulnerability were brought to light by Okta’s Red Team in a detailed advisory published earlier this week. Their research meticulously describes how a seemingly innocuous design flaw in the TLS handshake process could be weaponized for a potent DoS attack. The vulnerability exploits a specific interaction during the initial phase of establishing a secure connection.

In a standard TLS handshake, each message exchanged between a client and a server begins with a 4-byte header. Within this header, a crucial three-byte field is designated to declare the expected size of the subsequent handshake data. This mechanism is designed to inform the receiving party about the incoming data’s length, allowing for appropriate buffer allocation. However, researchers discovered that vulnerable versions of OpenSSL prematurely allocate memory based on this declared length before validating the actual size of the incoming payload.

This pre-allocation without validation creates the perfect window for exploitation. An unauthenticated attacker can initiate a TLS connection and transmit an extremely small, malicious input—just 11 bytes—containing a header that falsely declares a significantly larger message body will follow. The server, trusting the header’s claim, allocates a substantial chunk of memory to accommodate the purported large message. Crucially, the worker thread responsible for this connection then enters a blocking state, indefinitely awaiting the arrival of the promised data that, by design, will never materialize.

The attacker can repeat this process across numerous concurrent connections. Each malicious connection triggers the server to allocate considerable amounts of memory based on exaggerated claims, while transmitting only a minimal volume of data. This efficient yet insidious method allows an attacker to rapidly deplete a server’s available memory resources.

The Memory Fragmentation Trap: Why Bloat Persists

A key aspect that exacerbates the impact of HollowByte is its interaction with the underlying memory management system, specifically the GNU C Library (glibc), which is widely used in Linux environments. While OpenSSL eventually frees the buffers associated with a dropped connection, glibc’s approach to memory handling introduces a critical caveat.

Okta researchers explain that glibc does not immediately return small-to-medium memory allocations to the operating system. Instead, it retains these freed chunks for potential reuse by the application. This optimization aims to reduce the overhead of constantly requesting and releasing memory from the kernel. However, in the context of a HollowByte attack, this mechanism becomes a liability.

HollowByte DDoS flaw bloats OpenSSL server memory with 11-byte payload

By launching waves of connections with randomized claimed sizes, an attacker prevents the memory allocator from efficiently reusing these freed chunks. This leads to severe "heap fragmentation," where the server’s memory becomes littered with numerous small, unusable blocks, even though the total "freed" memory might appear substantial. Consequently, the server’s Resident Set Size (RSS), which represents the portion of memory held in RAM, climbs continuously and remains permanently bloated. Even after the attacker disconnects and the malicious connections are terminated, the server does not fully recover its memory footprint. The only effective way to reclaim the wasted space and restore the server to its optimal memory state is through a full restart of the affected process. This requirement for manual intervention adds a significant operational burden and prolongs service disruption.

Timeline of Discovery and OpenSSL’s Response

The timeline surrounding the HollowByte vulnerability, while not explicitly detailed in public advisories, can be inferred from the available information. Okta’s Red Team likely discovered the flaw during routine security research or penetration testing activities. Following responsible disclosure protocols, they would have privately reported their findings to the OpenSSL project maintainers.

OpenSSL’s response was swift, albeit discreet. Instead of assigning a public Common Vulnerabilities and Exposures (CVE) identifier, the OpenSSL team opted for a "silent fix." This approach, while sometimes employed to expedite patching and minimize immediate exploitation risks before a full public disclosure, can also lead to challenges in tracking and prioritizing patches for system administrators. The fix was incorporated into the latest development branch and subsequently backported to several older, still-supported releases.

The patched versions include OpenSSL 4.0.1, with backports to 3.6.3, 3.5.7, 3.4.6, and 3.0.21. The core of the fix lies in modifying how OpenSSL handles buffer allocation. Instead of relying solely on the header’s declared length, the patched versions now grow the buffer incrementally only as the actual data arrives, effectively ignoring any potentially malicious oversized claims in the header. This "grow-as-you-go" approach prevents premature, excessive memory allocation, thereby mitigating the DoS vector.

Okta’s public advisory, issued "earlier this week" (referring to the week of June 24, 2026, based on the original article’s context), served as the formal public disclosure, urging organizations to apply the patches immediately. The decision by OpenSSL to categorize this as a "hardening fix" rather than a critical security vulnerability, despite its significant impact, has led security experts like Okta to emphasize the practical severity of the issue, strongly recommending an immediate upgrade.

Far-Reaching Impact and Operational Disruptions

The ubiquitous nature of OpenSSL means the HollowByte vulnerability has a vast potential blast radius. Any system or application that relies on a vulnerable version of the library for TLS communication is susceptible. This includes, but is not limited to:

Web Servers: NGINX and Apache, which power a significant portion of the internet, are directly impacted.
Application Servers and Runtimes: Node.js, Python, Ruby, and PHP environments often link against OpenSSL, making applications built on these platforms vulnerable.
Databases: MySQL and PostgreSQL, when configured for encrypted connections, utilize OpenSSL.
Operating Systems: Most Linux distributions come with OpenSSL pre-installed, affecting various system services.
Network Appliances and IoT Devices: Many embedded systems and network devices also rely on OpenSSL for secure communication, potentially exposing a wide array of infrastructure.

Okta’s testing demonstrated the practical impact of HollowByte across different server capacities. For low-capacity environments, the attack could easily lead to complete memory depletion, causing service crashes and prolonged downtime. Even on higher-specification servers, which typically have more robust resources, the attack proved effective, causing memory loss of up to 25% while remaining below typical security alerting thresholds for bandwidth usage. This stealthy characteristic makes HollowByte particularly dangerous, as attacks might go unnoticed by conventional network monitoring tools focused on traffic volume.

The consequences of a successful HollowByte DoS attack extend beyond mere technical disruption:

Operational Disruption: Unavailability of critical services, leading to loss of productivity and inability to conduct business.
Financial Loss: Direct revenue loss from inaccessible services, costs associated with incident response, system restarts, and potential hardware upgrades.
Reputational Damage: Erosion of customer trust, negative public perception, and potential legal or compliance repercussions.
Resource Strain: IT and security teams are diverted to handle the incident, troubleshoot memory issues, and implement emergency patching.
Cascading Failures: In complex, interconnected systems, the failure of one OpenSSL-dependent service could trigger failures in others.

While DoS vulnerabilities are sometimes perceived as less severe than those enabling data theft or code execution, their capacity for widespread operational paralysis makes them a significant threat in today’s interconnected digital landscape.

Broader Implications and Mitigation Strategies

The HollowByte vulnerability serves as a stark reminder of several critical aspects of modern cybersecurity:

Supply Chain Security: As a foundational component, OpenSSL is deeply embedded in the software supply chain. A flaw in such a widely used library can ripple through countless applications and systems, making it challenging for organizations to identify all affected assets and apply patches uniformly. This highlights the importance of robust software bill of materials (SBOM) practices and continuous vulnerability management for third-party components.

Patch Management Challenges: The lack of a CVE identifier for HollowByte, while potentially speeding up initial fixes, creates hurdles for organizations relying on automated vulnerability scanning and patch management systems. System administrators must proactively monitor OpenSSL releases and security advisories from researchers like Okta, rather than waiting for CVEs to appear in their vulnerability databases. The need to backport fixes across multiple OpenSSL versions also complicates deployment, as organizations may be running older, but still supported, releases.

Detection and Response: The stealthy nature of HollowByte, characterized by a tiny attack payload and an impact that might initially fall below typical bandwidth-based alerting thresholds, demands more sophisticated detection mechanisms. Organizations need to implement robust memory usage monitoring (e.g., tracking RSS, heap usage) and anomaly detection systems that can flag unusual resource consumption patterns, even if network traffic remains low. Proactive threat hunting for suspicious TLS handshake patterns could also be beneficial.

The Role of Security Research: Okta’s Red Team’s discovery and detailed disclosure are crucial examples of how independent security research strengthens the overall cybersecurity ecosystem. By uncovering such subtle yet impactful flaws, researchers help improve the security posture of widely used open-source projects, benefiting countless users.

Recommendations for Organizations:
Security experts, including Okta, unequivocally recommend that organizations prioritize upgrading their distribution’s OpenSSL packages immediately. Even though OpenSSL categorized it as a "hardening fix," the practical implications of HollowByte warrant urgent attention.

Immediate Patching: Identify all systems running vulnerable OpenSSL versions (specifically those older than 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21) and apply the available patches without delay.
Inventory Management: Maintain an accurate inventory of all software and libraries, including their versions, to quickly identify affected assets during future vulnerability disclosures.
Enhanced Monitoring: Implement advanced monitoring solutions that track system resource utilization, particularly memory (RSS, heap usage), and look for unusual spikes or persistent bloat, even in the absence of high network traffic.
Incident Response Planning: Ensure incident response plans are updated to address DoS scenarios, including procedures for identifying compromised systems, mitigating attacks, and restoring services.
Stay Informed: Regularly consult official OpenSSL project announcements, vendor security advisories, and reputable cybersecurity news sources for the latest vulnerability information.

The HollowByte vulnerability underscores the continuous arms race between attackers and defenders in the digital realm. As critical infrastructure increasingly relies on complex, interconnected software components, the security of foundational libraries like OpenSSL remains paramount, demanding vigilance, rapid response, and a collaborative effort from the entire cybersecurity community.

July 18, 2026 0 comment