condense-json 1.0 Released: Efficient JSON Compression for Duplicated Strings
Simon Willison announced the 1.0 release of his condense-json library. The library compresses JSON by replacing duplicated strings with references to a replacements object, reducing storage space. It is used in SQLite logs for the LLM tool.
Simon Willison released version 1.0 of his condense-json library, which has been under development for a year and a half. The library provides functions condense_json and uncondense_json. condense_json takes an input JSON and a replacements object containing strings and substrings to replace; it scans the input for occurrences and replaces them with a special syntax like {"$r": ["string part", {"$": "1"}]} that references the replacements by key. uncondense_json reverses this effect. The goal is to ease storage of JSON with duplicated data from related structures. Willison uses this to save space in SQLite logs generated by his LLM tool, referencing PR #1586 for the latest iteration.
Source: Simon Willison —
original