Reference · AI Security · AppSec
AI Security Review Notes
A practical review checklist for AI features, agents, tool access, context, memory, logs, and output trust.
AI security review is mostly normal security work in a new wrapper. The questions are still about input, trust, authorization, data exposure, logging, and what happens when one system acts on another system’s behalf.
Start With The System Shape
I want a plain answer to:
- What does the AI feature do?
- Who can use it?
- What data does it see?
- What tools or actions can it request?
- What output does another person or system trust?
- What persists after the interaction ends?
If those answers are fuzzy, the security review will be fuzzy too.
Review Checklist
| Area | Questions |
|---|---|
| Context | What private or sensitive data is added to the prompt, retrieval context, or conversation state? |
| Instruction hierarchy | What instructions are trusted, and can user-controlled content compete with them? |
| Tools | What actions can the model request? Are they read-only, write-capable, or externally visible? |
| Authority | Whose permissions are used when the tool runs? Is authorization checked outside the model? |
| Memory | What is saved across sessions, users, tenants, or workflows? Who can read or change it? |
| Output trust | Does generated output update records, trigger actions, send messages, or influence decisions? |
| Logging | Can the team investigate behavior without exposing secrets or private prompts? |
Prompt Injection
I do not treat prompt injection like magic words. I treat it like untrusted content crossing a trust boundary.
The risk goes up when untrusted content can influence:
- Tool choice.
- Tool arguments.
- Hidden context.
- Data retrieval.
- Output that another system treats as trusted.
The fix is usually not “write a better prompt.” The fix is to narrow tool access, enforce authorization outside the model, validate outputs, and make sensitive actions explicit.
Tool Access
Tool access should be boring in the best way: narrow, explicit, authorized, and logged.
- Prefer read-only tools unless write access is truly needed.
- Enforce authorization server-side for every action.
- Treat model output as a request, not a decision.
- Validate tool inputs like any other user-controlled input.
- Log tool calls clearly enough that someone can reconstruct what happened.
Review Output
At the end, I want to be able to say:
- The model can see this.
- The model can ask for these actions.
- These systems enforce authorization.
- These outputs are verified before they matter.
- These logs let us investigate without creating a new exposure.
That is the difference between “we added AI” and “we understand what we added.”