Part 2: The Many Ways LLMs Leak Data—and How to Solve It

As detailed in Part 1 of this article, enterprises face a critical threat from LLM data leaks—with 45% of organizations already affected. Now we dive deeper into the three attack vectors enabling these breaches: prompt injection, jailbreaking, and flowbreaking.

Where traditional cyberattacks exploit code vulnerabilities, these methods manipulate LLMs’ linguistic reasoning. Understanding them is critical for developing effective defenses—whether you’re a CISO evaluating AI security tools or a startup building secure applications.

Prompt injection

Prompts are the primary mode of interacting with LLMs – instructions delivered to the model, typically through a chat or voice interface. Examples of prompts include “tell me a joke,” “identify all cybersecurity investments Cota Capital has made,” or “translate the titles of all Taylor Swift songs into Spanish.”

The above examples are all non-malicious; prompt injection aims to concatenate malicious instructions with non-malicious instructions, thereby deceiving the model and making it execute unsafe actions. The goal is to contradict and overrule the preset model guardrails, and if needed, the application guardrails through sleight of hand and the use of clever and deceptive wording. This can be done through the following techniques:

Jailbreaking

Similar to prompt injections, jailbreaking attacks try to manipulate models into returning malicious outputs or executing malicious actions through input prompts. However, there are two key differences:

• Jailbreaking attacks directly target the models themselves, and not applications on top of these models – therefore, they are usually only bypassing model guardrails, and not the second-level safety features built into applications on top of these 
• Jailbreaking attacks do not concatenate trusted and untrusted inputs 

There are some additional techniques specific to jailbreaking, such as hijacking, where the model is “forced” to ignore its existing guardrails. An example of this is the DAN (Do Anything Now)prompt, where the model is led to believe it is empowered to provide any output, irrespective of its safety.

Flowbreaking

Flowbreaking is an entirely different category of attack that targets nuances of output generation by models. These attacks take advantage of the brief window between when a model generates an output and when that output is flagged as inappropriate and possibly retracted. Typically, these attacks only target the models themselves, and not applications built on top of them.

So far, two main flowbreaking techniques have been identified:

• Second thoughts – Models generate an inappropriate output, which is then retracted a few seconds (or less) later; however, the resultant data leakage can be captured through a screenshot or other similar methods
• Stop and roll – In this case, the model reasons through a given input, but its processing is stopped by the user manually (through a kill switch or stop button) before the guardrails activate

The intricacies of the model and guardrail architecture that enable these attacks is unknown, but they have been demonstrated on several of the best-known AI models including OpenAI’s o1-mini, Microsoft o365 copilot, and Claude 2 by security researchers and ethical hackers. 

How to prevent an AI from being misguided

While multiple potential risk vectors targeting AI models are being identified, several solutions that help safeguard these models are also being developed. These include:

• Prompt filtering – These solutions analyze the input prompt to identify malicious intent and content, and prevent the model from responding to these. 
• Data marking – These solutions aim to guard against indirect prompt injections and other similar risks by clearly highlighting and analyzing externally accessed data for any malicious intent or content.
• Metaprompts – These are overall guardrails that set out clear definitions for what the model is expected to do, irrespective of the input prompts or externally accessed content. 
• Data access controls – These solutions prevent models from accessing confidential data, especially from within a company’s ecosystem.
• Identity and user access management – These solutions, which increasingly need to protect both human users and agentic users, segment access to data or tasks based on a user’s role, thereby preventing unauthorized data access. 
• Output guardrails – These are safety filters that review the model outputs before release.

A net-new world

As the adoption of AI across enterprise use cases grows, new risks and attack vectors will continue to be uncovered. This has led to a need for a completely new set of AI-native products that use AI for security—and that provide security for AI. These products are truly disruptive:

• Net-new space
• Net-new markets served
• Net-new technologies underpinning these solutions

Incumbents in the security space do not have a true head start over newcomers due to the rapidly evolving nature of the ecosystem. Furthermore, the talent required to solve these problems would need to have expertise in AI architectures – enabling a new breed of founders and builders to disrupt this market. 

As the risk of data leakage continues to grow, solutions that ensure safety, security, and reliability of AI are essential to truly unlock the vast efficiency and productivity benefits possible. From our perspective, the ideal solution would:

• Combine the different potential solutions listed above to create a holistic solution that reviews input prompts, externally accessed data and tools, and output responses
• Utilize multiple different models that are trained across all known vulnerabilities (direct and indirect prompt injection, jailbreaks, flowbreaking) to provide multi-layered security for known attack vectors
• Detect and flag anomalous behavior that could be indicative of a new type of attack vector
• Balance usability and security, ensuring that there is minimum additional latency or filtering out of genuine user requests, either of which could harm user experience
• Be compatible with evolving architectures and protocols in the AI space

At Cota Capital, we continue to invest in net-new security companies that are building innovative solutions at the forefront of security for AI. If you are a builder in this space, reach out to us.