StruQ and SecAlign: Defending Against Prompt Injection Attacks

Prompt injection, where untrusted data contains an instruction that overrides the system's intended prompt, is listed as the #1 threat by OWASP for LLM-integrated applications. Production-level systems like Google Docs, Slack AI, and ChatGPT have been shown vulnerable. The authors propose two causes: lack of separation between prompt and data, and LLMs being trained to follow instructions anywhere in the input. To address these, they introduce a Secure Front-End that uses special tokens to delimit prompt and data, filtering out any delimiter in the data. StruQ (Structured Instruction Tuning) simulates prompt injections during training, teaching the LLM to ignore injected instructions in the data part. SecAlign (Special Preference Optimization) goes further by training on both desirable and undesirable responses, using preference optimization to create a large probability gap between them. Experiments on Llama3-8B-Instruct show that StruQ reduces the Attack Success Rate (ASR) to 45%, and SecAlign reduces it further to 8% against sophisticated attacks. SecAlign preserves utility as measured by AlpacaEval2, while StruQ decreases it by 4.5%. Both reduce optimization-free attack success rates to nearly 0%, and SecAlign reduces optimization-based attack ASR by a factor of over 4 compared to previous state-of-the-art.