How to Avoid Breaking LLMs While Protecting Data: Under the Hood of Guardrails Filter

The article discusses the development of a guardrails filter designed to protect personal data in interactions with large language models. While finding personal data using regular expressions is straightforward, the real challenge lies in integrating the filter between the application and the model without breaking the interaction. The filter must handle ordinary requests, streaming, tool calling, maintain message structure, and preserve metadata. To handle multiple instances of the same data type, the filter uses a mapping table, assigning unique placeholders like <PHONE_1> to each distinct value. Because LLMs are stateless, the filter must reprocess the entire conversation history with each request. In streaming mode, responses arrive in chunks, and placeholders may be split across chunks; the filter uses a buffer to accumulate enough data before attempting demasking. It also handles various fields in responses, including content, reasoning, and tool call arguments, ensuring data is correctly restored in all of them. The implementation grew to about 1,500 lines of code for streaming in Chat Completions alone. Additionally, supporting the Messages API required a separate implementation due to its event-based format, bringing the total test code to over 4,000 lines.