From Dirty References to Golden Records: How AI/ML Saves MDM from Chaos
Manual cleaning of master data management (MDM) references is inefficient; the only sustainable path is AI/ML-based automation. Hybrid models combining classification, semantic search, and intelligent attribute extraction enable faster and more accurate attribute normalization, turning data chaos into unified golden records.
The article explores the causes of NSI degradation: parallel information systems, different data entry approaches, the human factor, incompleteness, and outdated standards. Simple string comparison algorithms (Damerau-Levenshtein distance, Jaro-Winkler) achieve acceptable quality in only 30–40% of cases, as they do not account for semantics, morphology, or context, potentially leading to false mergers of fundamentally different objects. The only reliable path is attribute normalization, which consists of five phases: audit and analysis, structuring and enrichment, cleaning with reference creation, publication and integration, and maintenance and improvement. Using the SOFROS AI/ML service as an example, it shows how machine learning and large language models (LLMs) automate classification, attribute extraction, and normalization, improving data quality and completeness.
Source: Habr — хаб ИИ —
original