As the demand for timely and granular statistical data grows, national statistical institutes face increasing pressure to maintain high data quality while reducing respondent burden and working within limited resources. To address this, ISTAT launched an experimental initiative to enhance the efficiency and accuracy of product-based business surveys, starting with the Annual Survey on the Production of Manufactured Goods (PRODCOM), through AI-powered automation. The project explores how web scraping and artificial intelligence can be combined to automatically extract product information from company websites and assign the appropriate PRODCOM codes. Two distinct methodological approaches were developed: In the first approach, a custom Python tool autonomously visits company websites and, with the support of a large language model (LLM), extracts product names and descriptions, which are returned in a structured JSON format. These are then simplified by the LLM, converted into embeddings using OpenAI’s text-embedding model (together with the classification system descriptions), and stored in Qdrant vector database. Product-code matching is performed via cosine similarity queries. The second approach is fully open-source and uses a pipeline that crawls websites, extracts and chunks text, and applies a retrieval-augmented generation (RAG) system to generate product descriptions. Classification is then performed using two methods: (i) LabinLlama, a fine-tuned LLM classifier; (ii) a multi-task variational autoencoder (VAE) to extrapolate the latent semantic meaning, that shares with the appropriate PRODCOM definition to assign the closest code via cosine similarity. Both approaches rely on generic web-scraping combined with AI-driven classification. Early results show promising gains in automation and accuracy. While still in the experimental phase, this project showcases the potential of AI and online data in rethinking how statistical data is collected, classified, and used to foster the modernization of business surveys. Further developments are required to improve robustness, ensure scalability, and support integration into official statistical workflows.
ISTAT, Italian National Institute of Statistics, Italy
ISTAT, Italian National Institute of Statistics, Italy
ISTAT, Italian National Institute of Statistics, Italy
ISTAT, Italian National Institute of Statistics, Italy
ISTAT, Italian National Institute of Statistics, Italy
ISTAT, Italian National Institute of Statistics, Italy
Titolo del capitolo
From web to code: AI-driven product classification for smarter business surveys
Autori
Gabriele D'Amore, Donato Summa, Massimiliano Amarone, Luigi Briatico, Mirella Morrone, Francesco Scalfati
Lingua
Inglese
DOI
10.36253/979-12-215-1062-1.30
Opera sottoposta a peer review
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Titolo del libro
Data, Statistics and AI for Well-Being of People and Organisations
Curatori
Luigi Fabbris, Alberto Franci, Pietro Renzi
Opera sottoposta a peer review
Numero di pagine
420
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Editore
Firenze University Press
DOI
10.36253/979-12-215-1062-1
ISBN Print
979-12-215-1061-4
eISBN (pdf)
979-12-215-1062-1
eISBN (xml)
979-12-215-1063-8
Collana
Applied Statistics Series