How LLMs Can Help Determine Data Lineage
Сбер
The article discusses using large language models (LLMs) to automatically extract Data Lineage from SQL code. It presents an approach based on GigaChat from Sber, achieving high F1 scores. An agent-based reflection loop optimizes prompts, and a web interface built with Streamlit visualizes the results.
The article addresses the challenge of maintaining accurate Data Lineage in complex data environments. The authors propose using LLMs to automatically parse SQL scripts and extract target and source tables. They selected 127 test scripts of varying complexity. Using GigaChat, they developed the SQLLineageExtractor class with LangChain. Initial results showed F1 scores of 0.7898 for basic GigaChat and 0.9218 for GigaChat 2 Pro. To improve prompt quality, they created an intelligent agent (GigaChatSQLLineageAgent) based on LangGraph that iteratively refines prompts using validation feedback, achieving F1 of 0.9336. A Streamlit web interface was built for single and batch SQL analysis, providing dependency visualization.
- Сокращения
- LLM = Large Language Model — большая языковая модель
- SQL = Structured Query Language — структурированный язык запросов
- ETL = Extract, Transform, Load — извлечение, преобразование, загрузка
- CTE = Common Table Expression — обобщённое табличное выражение
- F1 = F1 score — метрика F1
- JSON = JavaScript Object Notation — JavaScript Object Notation
Source: Habr — хаб ИИ —
original