What Is Prompt Injection?

Prompt injection is, conceptually, a close cousin of SQL injection or cross-site scripting: a class of attack where untrusted input gets treated as instructions rather than data. The mechanism is different — there's no query syntax to escape — but the underlying problem is familiar to anyone who's dealt with traditional injection vulnerabilities.

How it works

Large language models process instructions and data in the same channel — natural language text. A system prompt tells the model how to behave; user input is meant to be data the model processes according to those instructions. Prompt injection exploits the fact that the model doesn't have a hard, structural boundary between "instructions I should follow" and "data I should process" — both arrive as text, and a sufficiently crafted piece of input text can cause the model to treat it as a new instruction, overriding or ignoring what it was originally told to do.

Direct vs. indirect prompt injection

Direct prompt injection happens when a user directly types input designed to override the model's instructions — asking it to "ignore previous instructions" or similar techniques, attempting to get the model to reveal its system prompt, bypass restrictions, or behave in unintended ways.

Indirect prompt injection is often the more dangerous variant in real applications: malicious instructions embedded in content the model processes as part of its normal function — a document it's asked to summarize, a webpage it's asked to read, an email it's processing — rather than typed directly by the user interacting with it. This matters because the person triggering the attack doesn't need direct access to the chat interface at all; they just need to get malicious content into something the model will process.

Why it's a genuinely hard problem

Unlike SQL injection, which has a well-established, largely effective fix (parameterized queries), prompt injection doesn't currently have an equivalently reliable structural solution, because the instruction/data separation that would prevent it isn't a native part of how current language models process text. Filtering and detection approaches help, but they're not comprehensive fixes — a sufficiently creative injection can often route around a given set of filters.

What a successful prompt injection can actually do

The real-world impact depends entirely on what the model has access to:

• Information disclosure — revealing system prompts, internal instructions, or data present in the model's context that shouldn't be exposed
• Unauthorized actions — if the model can call functions or take actions on a user's behalf, injection can potentially trigger actions the legitimate user never intended
• Misinformation or manipulation — causing the model to produce false, harmful, or brand-damaging output
• Bypassing intended restrictions — getting the model to perform tasks it was explicitly instructed not to do

Practical mitigations

While prompt injection can't currently be fully eliminated, its impact can be significantly reduced:

• Limit what the model has access to — the less privileged access a model has, the less damage a successful injection can do, regardless of how convincing the injected instructions are
• Require explicit authorization for sensitive actions, independent of the model's own decision — don't let the model's output alone be sufficient to trigger a high-impact action
• Treat model output as untrusted when it's used downstream — don't render it as HTML or execute it without appropriate handling
• Separate system instructions from user-supplied content as clearly as the platform allows, even though this doesn't provide a hard guarantee
• Monitor and log model interactions, particularly around sensitive actions, so unusual patterns can be detected even if not fully prevented

How does Averto fit in?

Testing for prompt injection — including indirect injection through content the model processes rather than direct user input — is part of how Averto approaches AI-integrated applications, alongside verifying what real-world impact a successful injection could actually achieve given the model's specific access and capabilities.