- Data, Statistics and AI for Well-Being of People and Organisations
- A cura di Luigi Fabbris, Alberto Franci, Pietro Renzi
The evolution of AI-driven textual analysis in Istat
- Elena Catanese
- Alessandra Righi
- © 2026 Author(s) |
- CC BY 4.0
- DOI: 10.36253/979-12-215-1062-1.24
Artificial intelligence has profoundly transformed textual analysis in recent years, introducing advanced methodologies for interpreting language and extracting meaningful statistical insights. Since 2018, Italy’s National Institute of Statistics has integrated AI-driven models to enhance its research capabilities, particularly in sentiment analysis and social media monitoring, given their real-time nature and lack of response burden. Sentiment-based indexes have been enriched through probabilistic models such as topic modeling via Latent Dirichlet Allocation, word embeddings, and neural networks, improving users' ability to interpret index dynamics. Initially reliant on rule-based approaches (e.g., lexicons) and clustering techniques, sentiment analysis has evolved with the adoption of BERT, enabling more nuanced contextual interpretation. However, official statistics face unique challenges, including biases inherent in social media data, necessitating robust data collection pipelines, monitoring systems, and clear imputation rules to ensure reliable datasets. Strict annotation protocols are particularly crucial in sensitive domains, such as gender-based violence studies and other social research tasks. AI-driven textual analysis has been applied across various domains, including the Social Media Economic Indicator, emotion detection in gender stereotypes, topic modeling of the Ukraine conflict, and discussions on gender-based violence. Modern large language models (LLMs) such as BERT, RoBERTa, and GPT variants learn contextual embeddings and are trained on vast corpora. Fine-tuned versions (e.g., BERT trained on sentiment data) significantly outperform lexicon-based methods in sentiment classification. Nevertheless, widespread adoption of LLMs remains constrained by high computational demands. Key challenges persist in methodological precision and resource optimization, with future advancements likely focusing on refining bias correction techniques and improving computational efficiency to ensure AI continues to play a pivotal role in statistical research.
- Keywords:
- Sentiment analysis,
- large language models,
- natural language processing,
ISTAT, Italian National Institute of Statistics, Italy - ORCID: 0009-0006-9616-3374
ISTAT, Italian National Institute of Statistics, Italy - ORCID: 0000-0001-7764-2261
- Bojanowski, P., Grave, E., Joulin, A., Mikolov, T. (2017). Enriching word vectors with sub-word information. Transactions of the Association for Computational Linguistics, 5, pp. 135–146. DOI: 10.1162/tacl_a_00051
- Bruno, M., Catanese, E., Ortame, F., Pugliese, F. (2025). Measuring social mood on economy during Covid times: A BiLSTM neural network approach, in P. Festa, D. Ferone, T. Pastore, O. Pisacane (eds.) Learning and Intelligent Optimization. LION 2024. Lecture Notes in Computer Science, vol 14990. Springer, Cham. https://doi.org/10.1007/978-3-031-75623-8_24. DOI: 10.1007/978-3-031-75623-8_24
- Castellucci, G., Croce, D., Basili, R. (2015). Acquiring a large-scale polarity lexicon through unsupervised distributional methods, in C. Biemann, S. Handschuh, A. Freitas, F. Meziane, E. Métais (eds), Natural Language Processing and Information Systems, 9103, pp. 73–86. DOI: 10.1007/978-3-319-19581-0_6
- Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440–8451. DOI: 10.18653/v1/2020.acl-main.747
- Blei, D.M., Ng, A.Y., Jordan, M.I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Resources, 3, pp. 993–1022. DOI: 10.7551/mitpress/1120.003.0082
- Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, pp. 4171–4186.
- Drus, Z., Khalid, H. (2019). Sentiment analysis in social media and its application: systematic literature review. Procedia Computer Science, 161, pp. 707–714. DOI: 10.1016/j.procs.2019.11.174
- Gerlitz, C., Helmond, A. (2016). The Like economy: Social buttons and the data-intensive web. New Media & Society, 15(8), pp. 1348-1365. DOI: 10.1177/1461444812472322
- Hochreiter, S. (1997). Long short-term memory. Neural Computation, 9(8), pp. 1735–1780. DOI: 10.1162/neco.1997.9.8.1735
- Nielsen, J. (2006). The 90-9-1 Rule for participation inequality in social media and online communities. NN/g. Available at https://www.nngroup.com/articles/participation-inequality/
- Schuster M., Paliwal, K.K. (1997). Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11), pp. 2673–2681. DOI: 10.1109/78.650093
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I. (2017). Attention is all you need. NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems, 30, pp. 6000–6010. DOI: 10.65215/ysbyhc05
- Wang, Z., Hale, S.A., Adelani, D.I., Grabowicz, P.A., Hartmann, T., Flöck, F., Jurgens, D (2019). Demographic inference and representative population estimates from multilingual social media data. The World Wide Web Conference, pp. 2056–2067. DOI: 10.1145/3308558.3313684
- Catanese, E., Scannapieco, M., Bruno, M., Valentino, L. (2022). Natural language processing in official statistics: The social mood on economy index experience. Statistical Journal of the IAOS, 38, pp. 1451 - 1459. DOI: 10.3233/sji-220062
Informazioni sul capitolo
Titolo del capitolo
The evolution of AI-driven textual analysis in Istat
Autori
Elena Catanese, Alessandra Righi
Lingua
Inglese
DOI
10.36253/979-12-215-1062-1.24
Opera sottoposta a peer review
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Informazioni bibliografiche
Titolo del libro
Data, Statistics and AI for Well-Being of People and Organisations
Curatori
Luigi Fabbris, Alberto Franci, Pietro Renzi
Opera sottoposta a peer review
Numero di pagine
420
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Editore
Firenze University Press
DOI
10.36253/979-12-215-1062-1
ISBN Print
979-12-215-1061-4
eISBN (pdf)
979-12-215-1062-1
eISBN (xml)
979-12-215-1063-8
Collana
Applied Statistics Series