A recent deep dive into the intricate workings of Large Language Model (LLM) APIs, specifically focusing on the reasoning capabilities of models like OpenAI’s GPT and Anthropic’s Claude, has brought to light potential security implications previously overlooked by many developers and users. This exploration, initiated as a weekend hobby project, has revealed that the "thinking" or "reasoning" data generated by these advanced AI systems, when transmitted to the client application, is not merely an opaque black box but carries potential vulnerabilities related to replayability and side-channel leakage. While not exposing a critical system-breaking flaw, the findings offer a nuanced understanding of how LLM architecture might inadvertently expose sensitive information.
The Genesis of an Investigation: An Intriguing Error Message
The investigation began with an unexpected encounter during the setup of an OpenClaw agent, a sophisticated AI application. While the immediate utility of another AI in the author’s personal life was questionable, the configuration process exposed a peculiar error message. This message contained a digital signature within what was described as the LLM’s "thinking" block. This anomaly immediately piqued the interest of a cryptographer, raising fundamental questions: Why would an AI’s internal reasoning process be signed? What security implications arise if these reasoning blocks are indeed subject to tampering? This curiosity set the stage for an extensive weekend of analysis, involving approximately 5 million tokens processed by Codex.
Decoding the "Encrypted Reasoning" Mechanism
At the core of the discovery lies a feature of LLM APIs that is often not explicitly highlighted: the transmission of the model’s internal reasoning process, commonly referred to as "chain-of-thought" (CoT). While users interacting with models like ChatGPT see summarized outputs, the underlying API for developers often includes a hidden, encrypted copy of the raw CoT data. This data is sent to the client application as a JSON object, containing Base64-encoded strings. Documentation from providers like OpenAI and Anthropic indicates that this data is intended to be opaque, not for direct inspection, but rather to be sent back to the server in subsequent turns to maintain conversational context, especially in stateless or zero-retention environments.
However, upon closer examination, these encoded strings exhibit characteristics of authenticated ciphertexts. Their size fluctuates based on the complexity of the model’s reasoning, and any attempt to alter them results in a recognizable API error. This suggests a cryptographic layer designed to ensure the integrity and authenticity of the reasoning process as it’s passed between the server and the client.
- OpenAI’s Reasoning Blocks: Analysis suggests that OpenAI’s reasoning blocks may be based on a standard like Fernet tokens, comprising various opaque fields that mutually authenticate each other. While modifications to individual fields are prevented, other aspects can be altered. The presence of 12-byte Initialization Vectors (IVs) hints at the use of encryption algorithms like GCM or ChaCha, although the IV length might be considered short.
- Anthropic’s Reasoning Blocks: Anthropic’s implementation, while also involving authenticated ciphertexts, appears more complex. Notably, despite being labeled as a "signature," empirical testing indicated no actual cryptographic signature was present. The system primarily focuses on mutual authentication of opaque fields, preventing manipulation of individual components while allowing for modifications elsewhere. The IV lengths here also suggest similar encryption primitives.
The "Why" Behind the Encrypted Transmissions
The rationale for sending this encrypted reasoning data to the client, rather than keeping it entirely server-side, is rooted in the operational design of modern LLM APIs. In scenarios employing stateless, zero-retention, tool-loop, or client-managed conversation modes, the client application is responsible for maintaining the conversation history. Encrypted reasoning allows LLM providers to return hidden model states to the client in a format that is unreadable and unmodifiable by the client, yet verifiable and decryptable by the server upon replay. This ensures continuity in complex reasoning processes without compromising the integrity of the model’s internal state.
The "So What?": Potential Vulnerabilities
The implications of these encrypted reasoning blobs, though not immediately catastrophic, warrant careful consideration. The cryptographic protections generally prevent direct manipulation. However, two key areas of potential vulnerability have been identified:
1. Replayability of Reasoning Blobs
A significant finding is the ability to replay unmodified reasoning blocks. While direct tampering triggers API errors, unaltered blocks can be replayed without issue. This replayability extends beyond a single session, working across different accounts and even different models within OpenAI’s ecosystem (though Claude models proved more resistant).
From a cryptographic standpoint, this suggests the potential use of a single, global encryption key for all reasoning data transmitted to clients. This has implications for zero-data retention modes, as all users’ reasoning data could be escrowed under a single, infrequently changing key, rather than being protected on a per-account basis.
Furthermore, this replayability introduces a new threat vector for applications that expose chat interfaces to potentially malicious users. If an attacker can inject crafted JSON into the chat stream, they might be able to insert their own reasoning blobs, leading to unpredictable model behavior. This underscores the critical need for input sanitization in such applications.
While replayed blocks are accepted by the LLM, their "semantic activity"—whether the model truly processes and acts upon the decrypted data—can vary. Extensive experimentation with Codex revealed that while replayed blocks are often quietly absorbed, in some instances, models like GPT 5.5 demonstrably incorporate the replayed information. For example, one experiment showed a replayed reasoning block leading to the appearance of a specific number in a new session on a different account, raising questions about data persistence and inter-session influence. This confirmed that encrypted blocks can indeed be semantically active, although extracting actionable sensitive data remains challenging.
2. Side-Channel Leakage Through Reasoning
Beyond direct replay, the author explored whether metadata associated with reasoning blocks could be exploited as a side channel for data extraction. This includes observable signals such as the length of the reasoning blocks, token counts (like OpenAI’s reasoning_tokens field), and even wall-clock response times.
The hypothesis posits that if an LLM is instructed to keep a secret (e.g., a single bit or a string) from the user, but is allowed to reason about it internally, the computational complexity of that reasoning could leak information. For instance, a scenario where computing the bit as ‘1’ requires significantly more processing than computing it as ‘0’ would manifest as differences in reasoning block size, token counts, or response times.
An experiment was designed where a model was asked to compute a checksum when a secret bit was ‘1’. By analyzing the size of the reasoning blobs or the response times, it was possible to distinguish between the ‘0’ and ‘1’ states of the secret bit with a high degree of accuracy. This demonstrates that reasoning processes themselves can be inherently "leaky," even when the model is instructed not to reveal sensitive information directly. This signal is particularly concerning as it could potentially be extracted even from chat-only interfaces where direct access to encrypted blobs is not available.
The Pursuit of Platform Secrets
A more ambitious goal was to determine if this side-channel leakage could be leveraged to extract highly sensitive "platform secrets," such as the models’ top-secret system prompts—the underlying instructions that govern their behavior and safety protocols. This would involve a painstaking process of asking true/false questions about successive characters of the prompt.
However, this line of inquiry encountered significant resistance. Tools like Codex and Claude Code, after identity verification, actively refused to assist in extracting confidential information. Switching to OpenCode with Kimi 2.6, which exhibited fewer ethical qualms, yielded mixed results. While application-specific secrets that were deliberately embedded could be extracted, attempts to extract non-existent or non-embedded model prompts proved unsuccessful. The author concluded that while extracting embedded secrets is feasible, extracting system prompts that are not actively being reasoned about remains a "maybe," highlighting the need for providers to carefully consider the implications of their reasoning data.
Responses and Recommendations from LLM Providers
The findings were reported to OpenAI and Anthropic through their respective bug bounty programs. OpenAI initially deemed the report unreproducible, and while the author provided scripts, it was too late to rectify the situation. Anthropic acknowledged the findings but reasonably concluded that there were no significant security implications in the observed side channels or replayability. They indicated a potential update to their developer documentation to caution application developers about these aspects. Both providers granted permission to publish these findings.
Despite these responses, the author maintains that LLM providers should not dismiss these observations entirely.
Recommendations for Providers:
- Enhanced Key Management: Providers should proactively improve their key management practices. If reasoning state is deemed valuable enough for encryption, it should be properly secured. Reasoning data should not be replayable across sessions or accounts. Patching potential vulnerabilities before they are exploited is crucial.
- Addressing Side-Channel Leakage: The side-channel leakage is a more fundamental issue tied to the nature of model reasoning. If models can be induced to perform secret-dependent reasoning, leakage is almost inevitable. Future mitigation might involve implementing policy gates before models engage in reasoning about sensitive information. However, this itself presents a challenge, as policy enforcement often requires reasoning.
The author concludes by expressing a degree of relief that such complex issues fall outside the purview of a cryptographer, highlighting the intricate challenges faced by those responsible for the security and integrity of advanced AI systems. The exploration of encrypted reasoning blobs underscores the evolving landscape of AI security, where even seemingly opaque internal processes can harbor subtle vulnerabilities that require ongoing vigilance and innovative solutions.
