GENERAL AI SECURITY / OWASP AI EXCHANGE
Start with the wider AI threat model.
Overview of OWASP AI Exchange: A Comprehensive Guide to AI Security
No video connection is made until you load the player.
A general introduction to AI security risks, controls, and governance. Watch the overview, then apply the written lessons and scenarios below to your own application boundary.
Overview of OWASP AI Exchange: A Comprehensive Guide to AI Security — Behnaz Karimi, German OWASP Day 2024, November 13, 2024. Hosted by media.ccc.de.
Licensed CC BY-SA 4.0 ↗ · Original source ↗. Embedded without editing. No OWASP endorsement of Avizium is implied.
0 of 3 knowledge checks answered correctly
Progress lasts until this page is refreshed. It records correct answers, not video viewing or certification.
01 / AI APPLICATION SECURITY
Treat model output as untrusted input
Apply application-security fundamentals to prompts, retrieved documents, and generated responses.
Revisit the AI security overview ↑
Review prompt: Use the LLM risk overview to identify where your application crosses from language into data access or execution.
Written lesson: controls to apply
- Keep authorization in application code. A system prompt and a guardrail are additional controls, not an access boundary.
- Treat documents and tool responses as untrusted content. Validate structured output and encode text for the destination; never execute generated shell commands or SQL directly.
- Test malicious retrieved text, unauthorized requests, and unsafe output handling alongside normal application tests. Record expected denials and retest evidence.
ARCHITECTURE REVIEW SCENARIO
An OpenShift support assistant reads a runbook containing instructions to ignore its task and reveal a secret. The document is useful evidence, but its instructions have no authority.
02 / AI APPLICATION SECURITY
Protect the data behind the answer
Connect retrieval permissions, model provenance, and dependency review to the evidence you already collect for secure delivery.
Revisit the AI security overview ↑
Review prompt: Consider how a model and data inventory complements your application dependency and deployment records. Apply the general AI security overview to retrieval-access controls and component provenance.
Written lesson: controls to apply
- Enforce tenant and document permissions before retrieval reaches the model. Test cross-tenant and revoked-access cases; a prompt asking for confidentiality is insufficient.
- Inventory model and embedding versions, data sources, libraries, images, and licenses. Review provenance and changes through the delivery pipeline.
- Minimize sensitive prompt and response retention. Define approved model endpoints, data classification, encryption, retention, and redacted audit evidence.
ARCHITECTURE REVIEW SCENARIO
A shared retrieval index contains documents from two customers. A user asks a legitimate question, but semantic similarity selects a document belonging to the other customer.
03 / AI APPLICATION SECURITY
Give agents bounded authority
Bring AWS IAM, OpenShift workload isolation, and change-control discipline to AI tools and agent workflows.
Revisit the AI security overview ↑
Review prompt: Identify which AI components can access data or trigger actions, then apply explicit authorization to each tool integration.
Written lesson: controls to apply
- Separate read-only analysis from change execution. Use narrowly scoped AWS roles or OpenShift service accounts and explicit tool allowlists.
- Require an independent approval for consequential changes, bound to the exact action and target. Reauthorize at execution; do not treat model confidence as approval.
- Limit egress, execution time, requests, and spend. Log redacted tool decisions and results, define a stop mechanism, and verify rollback and denied-action behavior.
ARCHITECTURE REVIEW SCENARIO
An agent that summarizes vulnerability findings is also allowed to change production security groups. A poisoned finding asks it to open inbound access to resolve a connectivity issue.
FROM LEARNING TO REVIEW
Build an AI security
evidence plan.
Carry the same discipline from STIG remediation and application security into AI features. Agree the system boundary, test the controls, and record the outcome.
Discuss AI application security ↗- 01
Map the architecture
Document users, models, retrieval stores, tools, AWS accounts, OpenShift namespaces, and external data paths.
- 02
Test allowed and denied behavior
Include prompt injection, cross-tenant retrieval, unsafe output, excessive tool access, and resource limits within an authorized scope.
- 03
Verify and retain evidence
Track model and application versions, owners, corrective actions, and redacted retest results. A STIG baseline alone does not establish AI application security.