AI Security

How It Works

The OWASP Top 10 for LLM Applications identifies the most critical security vulnerabilities in LLM-based systems. The 2025/2026 version includes: (1) Prompt Injection, where attackers manipulate model behavior through crafted inputs. (2) Insecure Output Handling, where LLM outputs are trusted without sanitization. (3) Training Data Poisoning, where malicious data corrupts model behavior. (4) Sensitive Information Disclosure, where models leak private training data. (5) Supply Chain Vulnerabilities in model dependencies. Each risk includes description, examples, and mitigation strategies.

Key Technologies

• Prompt injection (direct and indirect)
• Insecure output handling and code execution
• Training data poisoning and backdoors
• Model denial of service (resource exhaustion)
• Sensitive information disclosure through extraction

Real-World Examples

Prompt injection has been demonstrated against ChatGPT, Bing Chat, and numerous enterprise AI deployments. Indirect prompt injection (hiding instructions in documents the AI processes) is particularly dangerous for agentic systems. Samsung banned ChatGPT internally after employees inadvertently leaked proprietary source code. OWASP's LLM project has become the reference standard cited by regulators and enterprise security teams.

How It Works

AI security assessment evaluates an AI system across multiple dimensions: input validation (can the model be manipulated through prompts or data?), output safety (does the model produce harmful, biased, or confidential content?), infrastructure security (are model endpoints, APIs, and training pipelines protected?), supply chain integrity (are model weights, libraries, and data sources trustworthy?), and access control (who can query the model and at what rate?). Assessment frameworks combine automated scanning tools with manual red-team exercises.

Key Technologies

• Automated prompt injection testing tools
• Model output safety scanners
• API rate limiting and authentication
• Model weight integrity verification
• Dependency scanning for ML libraries

Real-World Examples

Google, Microsoft, and OpenAI all run internal red teams that probe their models for security issues before release. The AI Safety Institute (UK) and NIST (US) are developing standardized AI evaluation methodologies. Companies like HiddenLayer and Robust Intelligence provide commercial AI security assessment platforms. Bug bounty programs for AI systems (OpenAI, Anthropic) incentivize external security research.

How It Works

AI threats operate at multiple layers. Adversarial attacks craft inputs that cause models to misclassify or produce incorrect outputs (adversarial examples for vision, jailbreaks for LLMs). Data poisoning inserts malicious patterns into training data that activate during specific conditions. Model extraction uses queries to replicate a proprietary model's behavior. AI-enabled threats use AI tools to enhance traditional cyberattacks (automated phishing, vulnerability discovery, social engineering at scale). Supply chain attacks target ML libraries and model hosting platforms.

Key Technologies

• Adversarial examples and robustness testing
• Jailbreaking and guardrail bypassing
• Data poisoning and backdoor insertion
• Model extraction and intellectual property theft
• AI-enhanced social engineering and phishing

Real-World Examples

The 2024 xz utils backdoor attempt demonstrated how supply chain attacks target critical infrastructure. AI-generated deepfakes have been used in financial fraud (a $25M deepfake video call scam in Hong Kong, reported by CNN in February 2024). State-sponsored actors are using AI for influence operations and disinformation. The MITRE ATLAS framework catalogs adversarial techniques specific to machine learning systems.

How It Works

AI-assisted vulnerability management applies machine learning to the vulnerability lifecycle. Discovery uses LLMs and static analysis to identify vulnerabilities in source code, often finding patterns that traditional scanners miss. Prioritization uses ML models trained on exploit data to predict which vulnerabilities are most likely to be exploited in the wild, helping teams focus remediation efforts. Remediation uses code generation models to suggest patches. Continuous monitoring uses anomaly detection to identify exploitation attempts in runtime.

Key Technologies

• LLM-powered code review and vulnerability detection
• Exploit prediction models (EPSS, risk-based prioritization)
• AI-generated patch suggestions
• Runtime anomaly detection for exploitation
• EPSS (Exploit Prediction Scoring System)

Real-World Examples

GitHub Copilot and similar tools can identify common vulnerability patterns during code review. The FIRST EPSS model uses ML to predict exploitation probability, helping organizations prioritize CVE remediation. Snyk uses AI to suggest automated fixes for known vulnerabilities. Google Project Zero has experimented with LLM-assisted vulnerability research. Microsoft's Security Copilot assists security analysts with incident investigation.

How It Works

AI red teaming involves skilled testers systematically attempting to make an AI system behave in unintended ways. This includes trying to bypass safety guardrails (jailbreaking), extracting confidential training data, generating harmful content (CSAM, weapons instructions, malware), testing for biased or discriminatory outputs, and probing tool-calling capabilities for unauthorized actions. Red teams typically work from a defined scope and threat model, documenting findings in structured reports with severity ratings and remediation recommendations.

Key Technologies

• Jailbreak and guardrail bypass testing
• Training data extraction attempts
• Bias and discrimination probing
• Tool-calling and agent safety testing
• Automated red-teaming at scale (fuzzing for LLMs)

Real-World Examples

Anthropic published their model evaluation methodology, including red-team procedures. OpenAI engaged external red teams (including domain experts in biosecurity, cybersecurity, and political science) before releasing GPT-4. The White House organized a public AI red-teaming event at DEF CON 2023 with thousands of participants testing models from major AI labs. The UK AI Safety Institute conducts pre-release evaluations of frontier models.

How It Works

AI risk reporting involves documenting and communicating the known risks, limitations, and safety evaluations of AI systems. Model cards describe a model's intended use, performance characteristics, limitations, and ethical considerations. System cards extend this to the full deployment context. Incident reporting documents AI failures and near-misses for organizational learning. Public transparency reports disclose aggregate data about safety interventions, content moderation actions, and system reliability. Some organizations publish detailed risk assessments for frontier models before or after release.

Key Technologies

• Model cards (Google, 2019 proposal)
• System cards (deployment-level documentation)
• AI incident databases (AI Incident Database)
• Frontier model safety evaluations
• Transparency reports and responsible disclosure

Real-World Examples

Anthropic published detailed risk reports for Claude models, including capability evaluations and identified failure modes. OpenAI publishes system cards for major releases (GPT-4, GPT-4o). The AI Incident Database (maintained by the Partnership on AI) catalogs over 700 reported AI failures and incidents. The EU AI Act requires providers of high-risk AI systems to maintain documentation and logs accessible to regulators.