ResearchOpen Source 🇺🇸 11.08.2026 04:02

SQLite compressed text-history prototypes

OpenAIOpenAI
Simon Willison shares his idea for storing revision histories in SQLite using compressed JSON arrays of all previous versions. He discusses the prototype with GPT-Live voice mode and then gets GPT-5.6 Sol Pro to build experimental prototypes, which show excellent compression results.
Simon Willison is interested in options for storing revision histories in relational databases. While on a dog walk, he had an idea: take the full text of every prior version in a big JSON array of strings and apply zlib or zstd compression to the whole thing. He discussed the prototype with the GPT-Live voice mode in the ChatGPT iPhone app, explaining that a history column on a table could be a blob storing a compressed JSON array of all previous documents, plus a separate JSON array of timestamps that doesn't need compression. He then typed a text prompt to GPT-5.6 Sol Pro to build experimental prototypes in Python. The prototypes worked well: 1,000 simulated revisions to a document resulted in 20.4 MB of raw revision text that compressed to 80.3 KB as Zstandard-compressed JSON array. To avoid the overhead of decompressing and recompressing the entire array on every edit, Sol suggested breaking the history up into multiple rows, each containing a maximum of either 128 revisions or 3MB of uncompressed JSON.
Abbreviations
JSON = JavaScript Object Notation — JavaScript Object Notation (формат обмена данными)
SQLite = SQLite (lightweight relational database) — SQLite (легковесная реляционная база данных)
BLOB = Binary Large Object — двоичный большой объект
Source: Simon Willison — original
Our earlier posts on this topic ↓
Fresh news