Pentesting AI Applications: How to Test AI, LLM, and Agent Systems

What AI Penetration Testing is, and What it is Not?

AI penetration testing can mean two different things: using AI to perform a penetration test or testing an AI system for security weaknesses.

This guide focuses on the second, i.e., assessing AI-powered applications, LLMs, agents, chatbots, RAG pipelines, and connected tools for vulnerabilities that traditional application testing may miss.

Unlike conventional penetration testing, the goal is not just to find flaws in the underlying application. It is to determine whether an AI system can be manipulated to expose sensitive data, bypass authorization, misuse tools, or take unintended actions.

AI application penetration testing evaluates AI-powered applications for vulnerabilities across their models, prompts, data sources, tools, agents, APIs, identities, and connected systems.

What Counts as an AI Application?

AI applications can have very different architectures. Therefore, start by classifying the assets that make up the AI environment before beginning AI application security testing.

Identify multiple AI asset classes, including:

LLM Applications

Applications where an LLM is directly exposed through a web interface, API, chatbot, or other application layer.

Retrieval Pipelines

RAG systems involving document ingestion, vector databases, embeddings, retrieval mechanisms, rerankers, and knowledge sources.

MCP Servers

Model Context Protocol (MCP) servers exposing tools, resources, prompts, or other capabilities to AI assistants.

Autonomous Agents

AI systems capable of planning, using tools, accessing resources, and taking actions on behalf of users.

Multi-Agent Systems

Architectures where multiple agents communicate, delegate tasks, or share tools and resources.

Conversational Interfaces

Chatbots and AI assistants that interact directly with customers, employees, or other users.

Model Endpoints

Direct inference APIs, model gateways, streaming interfaces, and other interfaces through which models are accessed.

Model Supply Chain

Models, adapters, training data, registries, dependencies, artifacts, and MLOps infrastructure.

Multimodal Interfaces

Systems that accept or generate images, documents, audio, video, or other non-text inputs and outputs.

The same application can fall into multiple categories. For example, an enterprise AI assistant may simultaneously involve an LLM application, RAG pipeline, MCP servers, autonomous agents, and conversational interfaces.

Siemba’s AI security testing methodology requires recording these assets in an AI asset register, allowing the assessment team to determine which testing phases apply to each.

Why Traditional AppSec Testing Misses AI Flaws

Traditional web, mobile, and API penetration testing remains essential. However, AI introduces new trust boundaries that may not exist in conventional applications. Reasons include non-determinism, no fixed input space, and semantic rather than syntactic attacks. An LLM can interpret natural-language instructions, context, previous conversation history, retrieved documents, tool descriptions, and multiple conflicting instructions.

This is why a scanner can miss an AI app’s vulnerabilities.

Consider a typical flow:

User → Application → LLM → Tool → API → Business System

A traditional API test may establish that the API correctly enforces authorization.

But an AI agent may have its own service identity and invoke that API with significantly greater authority.

Similarly, a RAG application may have secure document storage while still allowing an attacker to manipulate the retrieval layer through poisoned content.

Siemba's methodology therefore maps trust boundaries and authority before exploitation begins. It specifically asks whether content crossing a boundary is being treated as data or instruction, and inventories the tools, credentials, integrations, and permissions reachable by the AI.

The AI Application Attack Surface

Test an AI application as a complete system rather than an isolated model. It may have the following layers that act as an AI attack surface:

Prompt Layer

Siemba evaluates whether an attacker can manipulate the application's instruction hierarchy.

Testing includes:

1. Direct prompt injection
2. Indirect prompt injection
3. System prompt extraction
4. Jailbreaking
5. Persona and role manipulation
6. Multi-turn escalation
7. Format and transformation attacks
8. Encoded instructions
9. Adversarial inputs

Indirect prompt injection is particularly important because malicious instructions may be hidden inside documents, webpages, emails, tickets, calendar entries, code comments, or HTTP responses consumed by the model.

The objective is not simply to make the model say something unexpected. The tester determines what the manipulation can reach.

The Model Endpoint

Evaluate the underlying AI model endpoint for authentication, authorization, data exposure, abuse controls, and resource consumption.

Testing may determine whether unauthorized users can access the model, whether model responses expose sensitive information, and whether usage controls can be bypassed.

For detailed model-specific testing, organizations can extend the assessment into dedicated LLM security testing.

RAG and Retrieval Pipeline

RAG introduces a second major security boundary.

Siemba's methodology tests the complete retrieval pipeline, including:

• Ingestion paths
• Corpus writers
• Document poisoning
• Retrieval manipulation
• Keyword and embedding stuffing
• Reranker manipulation
• Retrieval-time authorization
• Vector-store security
• Namespace isolation
• Metadata-filter bypass
• Chunk-boundary attacks
• Citation integrity
• Poison persistence

The methodology specifically identifies every writer to a corpus. It includes internal uploads, customer submissions, crawled content, SaaS sources, tickets, emails, and other connected systems as a potential injection source.

Testing therefore asks two separate questions:

Can an attacker influence what the AI retrieves?

and

Can the AI retrieve information the attacker should not be authorized to access?

Tool and Function Calling

AI applications increasingly allow models to call functions and external tools.

A security assessment should determine whether the model can:

• Invoke unauthorized functions
• Manipulate function parameters
• Access resources outside its permissions
• Execute sensitive operations without approval
• Abuse tool chaining
• Bypass application authorization

For MCP-enabled environments, this also includes testing connected MCP servers and tool manifests.

A malicious tool description matters only when testing shows it can influence the AI to perform an unauthorized action or access protected information.

AI Agent Autonomy and Excessive Agency

AI agents differ from traditional chatbots in one key way: they can act.

Evaluate the authority available to agents, including:

• Agent identities
• Service accounts
• Execution credentials
• Tool permissions
• API access
• Write permissions
• Delete permissions
• Approval workflows
• Memory
• Inter-agent communication

A good AI security testing methodology inventories each tool's effective permission scope. It then determines whether the agent acts using the invoking user's authority or its own standing authority. Agents operating with their own standing authority receive additional attention because compromise of the agent can create a larger blast radius.

Testing may include:

• Excessive agency
• Goal hijacking
• Tool abuse
• Approval bypass
• Data exfiltration
• Memory poisoning
• Agent impersonation
• Confused-deputy attacks
• Inter-agent injection
• Delegation abuse
• Cascading compromise

Read more: AI Agent Security Testing for Enterprise Chatbots.

Training and Fine-Tuning Pipeline

The AI supply chain can also become an attack surface.

Security testing may examine training data sources, fine-tuning datasets, model repositories, third-party models, dependencies, model artifacts, data integrity, and access controls.

A compromised training or model pipeline can introduce security problems before the application even reaches production.

Output Handling and Downstream Trust

Never automatically treat AI output as trusted application input.

An LLM's response may eventually be passed to a browser, database, an API, a shell, business workflow, another model, and a function or tool.

Testing should determine whether manipulated output can result in traditional vulnerabilities such as XSS, SSRF, injection, unauthorized transactions, or code execution.

This is exactly where AI-specific vulnerabilities can become conventional application compromises.

AI Security Testing Methodology

Siemba's AI security testing methodology tests the complete AI application rather than treating the underlying model as the only attack surface. It covers hosted and self-hosted models, first- and third-party integrations, AI features embedded into conventional applications, and purpose-built AI systems.

The methodology is built around three principles:

1. Test the system, not just the model.
2. Measure AI behavior across multiple trials rather than relying on a single response.
3. Follow an attack to the authority it can actually reach.

This means a successful jailbreak is not automatically considered a critical vulnerability. Siemba validates whether the behavior results in data exposure, unauthorized tool execution, privilege escalation, business impact, or any other demonstrable security consequence.

Select the phases of the AI security testing methodology based on the AI assets identified during scoping, rather than applying the same checklist to every environment.

Phase 1: Scoping an AI Pentest

The assessment begins by establishing:

• AI applications
• Model endpoints
• Model providers
• Streaming interfaces
• Tools and function schemas
• MCP servers
• Retrieval sources
• Corpus writers
• Agent identities
• Service accounts
• API keys
• OAuth grants
• Connected integrations

The engagement also establishes rules of engagement covering testing windows, cost ceilings, containment plans, destructive-action approvals, and trial counts.

For AI agents with write or destructive capabilities, Siemba's methodology calls for staged environments or canary targets wherever possible.

Phase 1 outputs an asset and authority register.

Phase 2: Threat Modeling and Authority Mapping

Before attempting attacks, Siemba maps where instructions and authority move through the system.

For example:

User input → Prompt assembly → Retrieved content → Model → Tool → API → Resource

Each boundary is examined to determine whether information is being treated as data or instruction.

The specialized team also inventories tools, credentials, permissions, execution identities, integrations, and reachable resources.

The methodology then models the potential blast radius of a fully compromised model or agent and identifies the highest-value authority reachable from each asset.

This authority mapping helps determine testing priorities.

Phase 3: LLM and Multimodal Interface Testing

Siemba tests the instruction boundary between system/developer intent and user or third-party content.

Testing includes:

Prompt Injection

Attempts to override system or developer instructions.

Indirect Injection

Instructions hidden in external content the model consumes.

System Prompt Extraction

Attempts to recover system instructions through progressive elicitation, transformation, translation, or partial completion.

Jailbreaking

Persona manipulation, role reframing, hypothetical scenarios, and other techniques designed to bypass restrictions.

Data Leakage

Attempts to cause the model to disclose protected information.

Adversarial Inputs

Encoded, malformed, unusual, or otherwise adversarial inputs designed to bypass controls.

For multimodal systems, testing expands to:

• OCR injection
• Image-based instructions
• PDF layers and comments
• Metadata
• Embedded objects
• Encoded content
• Audio injection
• Video/frame manipulation
• Parser exploitation
• Cross-modal filter inconsistencies

Phase 4: Retrieval Pipeline Testing

Where RAG is present, Siemba tests the ingestion and retrieval lifecycle.

The assessment can include:

1. Identifying every source that can write to the corpus.
2. Planting controlled poisoning payloads.
3. Testing retrieval ranking manipulation.
4. Testing authorization at query time.
5. Testing vector database controls.
6. Testing tenant and namespace isolation.
7. Measuring whether poisoning persists.
8. Testing whether restricted content can be retrieved through indirect queries.

The objective is to establish whether untrusted content can reach the context window as an instruction and whether retrieval actually respects the application's authorization model.

Phase 5: MCP Server Testing

For MCP-enabled systems, Siemba assesses both conventional and AI-specific attack surfaces.

Testing covers authentication, authorization, tool inventory, metadata, descriptions, poisoning, transport security, delegated permissions, and server configuration.

The methodology evolves with the protocol. MCP server security testing is therefore scoped to the protocol version and capabilities deployed during the engagement, rather than assuming a static checklist will remain sufficient indefinitely.

Phase 6: Autonomous and Multi-Agent Testing

Agentic systems receive deeper testing because successful attacks can translate directly into actions.

Siemba tests whether an attacker can manipulate agents into:

• Calling unauthorized tools
• Accessing restricted resources
• Exfiltrating data
• Modifying information
• Deleting information
• Bypassing approval workflows
• Escalating privileges
• Manipulating memory
• Delegating malicious instructions

For multi-agent systems, testing extends across agent-to-agent communication to identify transitive trust, confused-deputy behavior, inter-agent injection, and cascading compromise.

Phase 7: Conversational Interface Testing

Siemba tests conversational systems against the business actions they can perform.

Rather than simply testing whether a chatbot produces a prohibited response, Siemba validates whether an attacker can actually:

• Access another user's information
• Modify an account
• Change an entitlement
• Trigger a transaction
• Manipulate an order
• Obtain restricted knowledge-base content

This provides a more meaningful measure of conversational AI security than a simple jailbreak pass/fail result.

Phase 8: Model Endpoint and AI Supply Chain Testing

AI security extends beyond the application itself.

Model Endpoint Testing

Siemba evaluates API authentication, key scope, rotation, tenant isolation, quotas, model-version controls, gateway bypass, batch endpoints, asynchronous interfaces, resource exhaustion, and denial-of-wallet scenarios.

AI Supply Chain Testing

Testing can include model artifact integrity, checksums and signing, model provenance, unsafe deserialization, registry permissions, dataset poisoning, training-pipeline authorization, third-party models, adapters, dependencies, containers, and secrets in MLOps environments.

Phase 9: Reporting, Revalidation and Closure

Siemba's methodology places significant emphasis on evidence and repeatability.

A component is not considered clean simply because an automated scanner did not produce a finding. Testing must cover the relevant techniques against the identified entry points, with negative results supported by the agreed trial count.

For confirmed findings, Siemba records payload, response, reproduction rate, downstream authority reached, evidence, business impact, and applicable framework mapping.

This matters because AI responses are non-deterministic.

The methodology typically uses 10 trials for exploratory testing and 25 trials for confirmed findings, with reproduction rates measured and reported.

Why Reproduction Rate Matters in AI Security Testing

Traditional vulnerabilities are often deterministic.

An SQL injection attack either works or does not.

AI vulnerabilities can behave differently. A prompt injection might succeed two out of ten attempts or even 21 out of 25 attempts.

Those results communicate very different levels of reliability.

Siemba therefore measures reproduction rates rather than treating every successful response as equally exploitable. The methodology also records the downstream authority reached, which helps distinguish model behavior from actual security impact.

Mapping AI Security Tests to the OWASP LLM Top 10

The OWASP GenAI LLM Top 10 2026 (August 2026) provides an important framework for organizing GenAI and LLM security testing. The latest edition still has prompt injection as the foremost security threat to LLMs.

A useful AI penetration testing report should map individual findings to relevant OWASP categories rather than simply mentioning OWASP in the methodology.

For example:

For organizations building AI security programs, OWASP mapping also makes it easier to translate penetration-testing findings into broader application security and governance programs.

Read more: OWASP Top 10 for LLMs: Security Testing & Mitigation Guide for AI Applications.

How to Test an AI Application, Step by Step

If you are asking “how to test AI for security?”, use the following high-level process:

1. Inventory the AI application and identify models, agents, APIs, RAG systems, and tools.
2. Map trust boundaries between users, models, data, and connected systems.
3. Identify sensitive data that the AI can access.
4. Test authentication and authorization across AI interfaces and connected resources.
5. Run direct prompt-injection tests.
6. Run indirect injection tests through documents, web content, and retrieved data.
7. Test jailbreak and multi-turn manipulation.
8. Test sensitive information disclosure.
9. Test RAG isolation and retrieval controls.
10. Test tool and function authorization.
11. Test excessive agency and business logic.
12. Test AI output handling.
13. Validate confirmed attack paths manually.
14. Map findings to relevant OWASP categories.
15. Remediate and independently retest.

The important final step is validation. A theoretical jailbreak should not automatically be treated as equivalent to confirmed data exposure or unauthorized system access.

How to Automate AI Security Testing

Automation is valuable for increasing test coverage, so how do we automate AI security testing?

Siemba's methodology uses adversarial payload libraries and repeatable test suites to evaluate many attack variations.

Automation can help test prompt injection, jailbreaks, encoding attacks, RAG poisoning, tool manipulation, file uploads, multimodal inputs, API abuse, resource exhaustion, and model endpoint controls.

However, automation alone cannot establish the complete security impact. A scanner can identify that a prompt produced an unexpected response.

But we need an expert pentester to establish:

What information was exposed?

What authority was reached?

Could the action actually be executed?

Can the attack be reproduced?

That is why Siemba's methodology explicitly states that automated tooling alone is insufficient to declare a component clean.

How to Test Your AI Deployment: Pre-Production vs. Live

Organizations should consider AI security testing at multiple points in the development lifecycle, as shown in the following table.

Automated security validation can help identify regressions between expert-led assessments.

This is particularly useful because AI applications can change without traditional application code changing.

Testing by AI Application Type

We can have the following four AI application types, with different testing approaches.

AI Chatbots

For AI agents and chatbots, focus on prompt injection, jailbreaks, sensitive information disclosure, knowledge-base access, session isolation, and output handling.

RAG Applications

Focus on retrieval authorization, data isolation, document poisoning, indirect prompt injection, and source integrity.

AI Agents

Focus on excessive agency, tool invocation, authorization, business logic, privilege boundaries, and human approval.

MCP-Connected Systems

Focus on MCP server authentication, authorization, tool poisoning, malicious tool metadata, and third-party server risks.

Each asset type requires specialized testing, even when they use the same underlying LLM.

AI Red Teaming vs. AI Penetration Testing

AI penetration testing and AI red teaming overlap, but they answer different questions.

AI penetration testing focuses on identifying and validating vulnerabilities across the AI application's attack surface.

AI red teaming takes a more adversarial, objective-driven approach. The question becomes:

Can an attacker use the available weaknesses to achieve a defined objective?

For example, a pentest may identify that an agent has excessive permissions.

A red team exercise may attempt to use those permissions, combined with prompt injection and other weaknesses, to access a sensitive system.

Both approaches can be valuable depending on the organization's maturity and objectives.

What an AI Pentest Report Should Contain

A useful AI penetration testing report should provide more than a list of failed prompts.

It should include:

• Scope and architecture
• AI attack surface
• Testing methodology
• Test cases
• Confirmed vulnerabilities
• Proof of concept
• Reproduction steps
• Business impact
• Severity and risk
• OWASP mapping
• Remediation recommendations
• Evidence for each finding
• Retest status

The distinction between theoretical behavior and confirmed security impact is particularly important for AI.

A report should clearly separate:

Potential jailbreak → Confirmed vulnerability → Confirmed data exposure → Confirmed business impact

This makes the results more useful to security, engineering, and leadership teams.

How Often Should You Test AI Applications?

No single testing frequency fits every AI application.

A practical approach is to combine:

• Full penetration testing periodically
• Automated security testing during development and deployment
• Regression testing after significant AI changes
• Targeted testing when new tools, models, or data sources are introduced
• Retesting after remediation

Testing frequency should increase with the application's exposure and rate of change.

An internal AI assistant with limited permissions may require a different testing cadence from an internet-facing AI agent that can modify customer accounts or execute financial transactions.

AI Application Pentest Checklist

Before an engagement, security teams should identify:

1. AI applications
2. Model endpoints
3. Model providers
4. Model versions
5. System prompts
6. User roles
7. RAG pipelines
8. Knowledge sources
9. Vector databases
10. Corpus writers
11. MCP servers
12. Tools and functions
13. AI agents
14. Multi-agent workflows
15. Agent identities
16. Service accounts
17. API keys
18. OAuth grants
19. Sensitive data
20. Business-critical actions
21. Human approval workflows
22. Multimodal inputs
23. Third-party models
24. Model supply-chain components

This inventory forms the foundation for threat modeling and determines which phases of the security testing methodology apply. Here is a checklist to tally for each asset in the attack surface.

Build a Security Testing Program for Your AI Applications

AI security cannot be treated as a one-time model evaluation.

Every new model, tool, data source, agent capability, API integration, or workflow can introduce another attack path. At Siemba, we approach AI application security as a combination of automated adversarial testing and expert-led penetration testing.

Whether you are securing an AI chatbot, API, RAG application, autonomous agent, or MCP-connected environment, Siemba can help identify and validate AI-specific attack paths before they become production incidents. Pentest your AI application: Get a free attack surface scan.

To learn more → Book a demo with Siemba.

Frequently Asked Questions

How do you test AI for security?

Test the AI system across its full attack surface, including prompts, system instructions, RAG data, APIs, tools, agents, authentication, authorization, and connected services. Use adversarial testing to identify issues such as prompt injection, sensitive data disclosure, excessive agency, unauthorized actions, and insecure output handling, then validate their real-world security impact.

How can I automate AI security testing?

AI security testing can be automated using repeatable adversarial test cases, attack payloads, and automated validation across AI applications and their integrations. Automation can test large numbers of prompts and attack scenarios consistently, while human experts should validate complex findings, business logic, attack paths, and actual impact.

How can I test the security of my AI deployment?

Start by inventorying your AI assets, including models, applications, agents, RAG pipelines, APIs, MCP servers, tools, and data sources. Then threat-model their permissions and trust boundaries, run targeted security tests, validate exploitable findings, assess blast radius, and re-test after remediation. This approach tests the deployed AI system and its connected environment, not just the underlying model.