How to Teach AI to Understand Charts: Analyzing the New ChartNet Dataset from MIT

Researchers from the Massachusetts Institute of Technology (MIT) and IBM Research have created ChartNet, a multimodal dataset for training vision-language models (VLMs) to understand charts. The main problem with existing VLMs is the lack of high-quality and diverse data: GPT-4o correctly answers only 46% of tasks from the OGE (Basic State Exam), Gemini 2.5 Flash 65%, and Qwen3.7 Plus 83%. ChartNet addresses this issue through synthetic generation based on code: from 150,000 TinyChart graphs, Python code is recovered via a VLM, and then an LLM performs augmentations—changing the visualization type, library (Matplotlib, Seaborn, Plotly, Vega-Altair, Pygal, Plotnine), data, and styling. The result is 1.7 million examples for four tasks: code recovery, table extraction, text description, and question answering with reasoning. Quality control includes filtering of infeasible code (77% success rate), automatic defect checking (defect rate reduced from 15% to 6%), a manually verified set of 100,000 examples, 30,000 real-world charts, and 7,600 examples for sensitive topics. Fine-tuning on ChartNet significantly improved metrics for open VLMs: compact models (2–7 billion parameters) began to compete with larger ones, with Granite-Vision-3.3-2B achieving the best result in five out of seven metrics, surpassing GPT-4o. Generalizability was confirmed on the ChartCap and ChartMimic-v2 benchmarks, with metrics increasing without exception. Limitations: possible generation errors due to the use of LLMs/VLMs, potential distribution mismatch of synthetic data with real data, and unclear applicability of the effect for strong models.