Posted By
Part one of this series covered why combining vision, voice and text breaks traditional agent architecture. It also breaks the security model sitting underneath it. A text based filter is built to catch a suspicious sentence. It has no way to catch a suspicious pixel or a suspicious waveform, and a multimodal agent processes both by default. The same combination that breaks the architecture opens a second problem. That gap does not live in the code. It lives inside the images, audio and documents the agent is asked to process, and the frameworks most security reviews are built around, OWASP's LLM Top 10 among them, have no dedicated category for inspecting any of it.
Image based prompt injection hides instructions in what the agent sees
A prompt injection attack used to mean malicious text hidden in a webpage or a document, something a text filter had a real chance of catching. Image-based prompt injection changes that. Researchers have shown that instructions embedded directly into an image, as visible text, as patterns invisible to the human eye, or even printed on a physical sign in the real world, can steer a vision capable model without a single word of the original prompt changing. The agent is still doing exactly what it was told. It is just being told by the picture instead of the person holding the keyboard.
Where these instructions hide
Some of these instructions are typographic, plain text rendered small enough or blended into a background well enough that a person skims past it while the model reads every pixel. Others use adversarial pixel patterns invisible to a human eye but legible to whatever model is doing the classification. Security researchers have documented all these categories against production systems, not just lab demonstrations, which is what makes this a present risk rather than a theoretical one.
Why this matters more for agents than for chatbots
A chatbot that misreads one image gives one wrong answer. An agent that misreads one image can act on it, and if that agent's output feeds a second agent further down a pipeline, the bad instruction travels with it. Researchers have flagged this exact propagation risk in multi agent systems, where an image processed early in a workflow can quietly influence an agent with far more privilege later.
Audio adversarial attacks exploit what the model hears, not what a person hears
Voice agents introduce the same problem from a different direction. A perturbation added to an audio clip can be inaudible or nearly inaudible to a person while still being interpreted as an instruction by the model processing it.
Why inaudible does not mean harmless
Researchers publishing at a major security conference demonstrated an attack against audio language models with success rates reported as high as 96 percent across a range of open models, and the hidden commands were able to trigger real actions, downloading files, sending emails, running searches. None of it required the audio to sound unusual to a human listener in the room.
The defense gap this creates
A text-based content filter has nothing to inspect here. There is no suspicious phrase to flag, no keyword to block, no unusual sentence structure to catch. The instruction exists only in the waveform. A filter built to check the words in a transcript has nothing to check against, since the transcript is not where the instruction lives.
Cross modal data leakage turns supporting files into disclosure risk
Sensitive information does not need to appear in a conversation to leave an organization. It can travel inside a file the agent was only asked to summarize.
What gets exposed without anyone typing anything
Image metadata routinely carries GPS coordinates, device identifiers and internal file paths that were never meant to leave the building, the kind of hidden data that has deanonymized people before. OCR run over a screenshot pulls out whatever text is visible in it, including the parts a person assumed were too small or too incidental to matter. A voice transcript captures whatever was audible in the room, not just the sentence the speaker intended to send.
Why fusion makes the leakage worse
None of these signals looks dangerous in isolation. A username by itself is not identifying. A voice clip by itself is not identifying. A photo with its metadata stripped is not identifying. Put them together inside one agent reasoning across all three, and the combination can identify a specific person or location with a level of confidence none of the individual pieces would have allowed on its own.
Why traditional content filters were never built for this
OWASP's Top 10 for LLM applications, the most referenced framework in this space, is structured around what happens to text: prompt injection, sensitive information disclosure, output handling. These categories are useful, and prompt injection has held the top spot on that list for two editions running. But the framework itself inspects characters and tokens. It has no native way to inspect a pixel pattern or an audio waveform for the same kind of malicious intent.
The filters are looking in the wrong place
A filter tuned to catch a suspicious instruction in a text prompt has nothing to flag when the same instruction arrives as a typographic overlay on an image or a perturbation in an audio clip. It passes the check simply because that layer of the input was never being inspected in the first place.
Building a multimodal aware threat model
None of this means multimodal agents should stay unbuilt. It means the AppSec checklist that worked for a text only chatbot needs real additions before it gets applied to one.
What changes in the checklist
Input validation needs a check for every modality entering the agent, not just the text prompt, since a pixel pattern and an audio waveform both need their own inspection step. Privilege boundaries between agents in a pipeline matter more once one compromised modality can influence a downstream agent with more access than the first one had. Metadata stripping belongs before an image or document enters any processing step, not after. And any high stakes action triggered by a non-text input, a voice command, an image instruction, deserves the same human confirmation step a high stakes text based action would get.
A reasonable place to start
Map every modality entering the agent to its own validation step before calling the pipeline reviewed. A threat model that only accounts for the text prompt has not actually covered the agent. It has covered one third of it.
Vision, voice and text each carried their own attack surface long before anyone combined them into a single agent. What changes once they are combined is that a compromise in one modality can now influence what the agent does with the other two. That is the part a text centric framework has no category for. Part one of this series covered why combining these modalities breaks traditional architecture. This is what that same combination does to the threat model sitting underneath it.
Contact Opcito's experts to talk through what a multimodal aware threat model needs to cover.
Related Blogs
