Contenuto in:
Capitolo

From web to code: AI-driven product classification for smarter business surveys

  • Gabriele D'Amore
  • Donato Summa
  • Massimiliano Amarone
  • Luigi Briatico
  • Mirella Morrone
  • Francesco Scalfati

As the demand for timely and granular statistical data grows, national statistical institutes face increasing pressure to maintain high data quality while reducing respondent burden and working within limited resources. To address this, ISTAT launched an experimental initiative to enhance the efficiency and accuracy of product-based business surveys, starting with the Annual Survey on the Production of Manufactured Goods (PRODCOM), through AI-powered automation. The project explores how web scraping and artificial intelligence can be combined to automatically extract product information from company websites and assign the appropriate PRODCOM codes. Two distinct methodological approaches were developed: In the first approach, a custom Python tool autonomously visits company websites and, with the support of a large language model (LLM), extracts product names and descriptions, which are returned in a structured JSON format. These are then simplified by the LLM, converted into embeddings using OpenAI’s text-embedding model (together with the classification system descriptions), and stored in Qdrant vector database. Product-code matching is performed via cosine similarity queries. The second approach is fully open-source and uses a pipeline that crawls websites, extracts and chunks text, and applies a retrieval-augmented generation (RAG) system to generate product descriptions. Classification is then performed using two methods: (i) LabinLlama, a fine-tuned LLM classifier; (ii) a multi-task variational autoencoder (VAE) to extrapolate the latent semantic meaning, that shares with the appropriate PRODCOM definition to assign the closest code via cosine similarity. Both approaches rely on generic web-scraping combined with AI-driven classification. Early results show promising gains in automation and accuracy. While still in the experimental phase, this project showcases the potential of AI and online data in rethinking how statistical data is collected, classified, and used to foster the modernization of business surveys. Further developments are required to improve robustness, ensure scalability, and support integration into official statistical workflows.

  • Keywords:
  • automatic product classification,
  • web intelligence,
  • generative AI models (LLMs),
  • embeddings,
  • Deep Learning,
  • vector DB,
  • ProdCom,
+ Mostra di più

Gabriele D'Amore

ISTAT, Italian National Institute of Statistics, Italy

Donato Summa

ISTAT, Italian National Institute of Statistics, Italy

Massimiliano Amarone

ISTAT, Italian National Institute of Statistics, Italy

Luigi Briatico

ISTAT, Italian National Institute of Statistics, Italy

Mirella Morrone

ISTAT, Italian National Institute of Statistics, Italy

Francesco Scalfati

ISTAT, Italian National Institute of Statistics, Italy

  1. Kingma, D.P., Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
  2. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P. (2017). Focal loss for dense object detection, in Proceedings of the IEEE International Conference on Computer Vision, pp. 2980-2988. DOI: 10.1109/iccv.2017.324
  3. Rezende, D.J., Mohamed, S., Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models, in International Conference on Machine Learning, pp. 1278-1286.
  4. Salton, G., Wong, A., Yang, C.S. (1975). A vector space model for automatic indexing. Communications of the ACM, 18(11), pp. 613-620. DOI: 10.1145/361219.361220
  5. Schroff, F., Kalenichenko, D., Philbin, J. (2015). Facenet: A unified embedding for face recognition and clustering, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. DOI: 10.1109/cvpr.2015.7298682
PDF
  • Anno di pubblicazione: 2026
  • Pagine: 165-170

XML
  • Anno di pubblicazione: 2026

Informazioni sul capitolo

Titolo del capitolo

From web to code: AI-driven product classification for smarter business surveys

Autori

Gabriele D'Amore, Donato Summa, Massimiliano Amarone, Luigi Briatico, Mirella Morrone, Francesco Scalfati

Lingua

Inglese

DOI

10.36253/979-12-215-1062-1.30

Opera sottoposta a peer review

Anno di pubblicazione

2026

Copyright

© 2026 Author(s)

Licenza d'uso

CC BY 4.0

Licenza dei metadati

CC0 1.0

Informazioni bibliografiche

Titolo del libro

Data, Statistics and AI for Well-Being of People and Organisations

Curatori

Luigi Fabbris, Alberto Franci, Pietro Renzi

Opera sottoposta a peer review

Numero di pagine

420

Anno di pubblicazione

2026

Copyright

© 2026 Author(s)

Licenza d'uso

CC BY 4.0

Licenza dei metadati

CC0 1.0

Editore

Firenze University Press

DOI

10.36253/979-12-215-1062-1

ISBN Print

979-12-215-1061-4

eISBN (pdf)

979-12-215-1062-1

eISBN (xml)

979-12-215-1063-8

Collana

Applied Statistics Series

0

Download dei libri

0

Visualizzazioni

Salva la citazione

1.528

Libri in accesso aperto

in catalogo

3.744

Capitoli di Libri

6.674.567

Download dei libri

5.994

Autori

da 1219 Istituzioni e centri di ricerca

di 67 Nazioni

83

scientific boards

da 416 Istituzioni e centri di ricerca

di 46 Nazioni

1.439

I referee

da 400 Istituzioni e centri di ricerca

di 41 Nazioni