- Data, Statistics and AI for Well-Being of People and Organisations
- A cura di Luigi Fabbris, Alberto Franci, Pietro Renzi
The role of large language models in generating sentiment signals: A comparative and effectiveness-based study
- Domenica Fioredistella Iezzi
- Roberto Monte
- © 2026 Author(s) |
- CC BY 4.0
- DOI: 10.36253/979-12-215-1062-1.41
Large Language Models (LLMs) have transformed the field of natural language processing (NLP), achieving state-of-the-art performance across a wide range of linguistic tasks, including sentiment analysis. Unlike traditional machine learning methods that rely on manual feature engineering and task-specific datasets, LLMs such as ChatGPT (based on the GPT architecture; OpenAI, 2023), LLaMA (Meta’s Large Language Model; Touvron et al., 2023), and BERT (Bidirectional Encoder Representations from Transformers; Devlin et al., 2019) are pretrained on vast textual corpora and can capture nuanced semantic and contextual relationships across domains. This makes them particularly well-suited for extracting sentiment signals from informal, noisy, and unstructured data, such as user-generated content on social media platforms. In this study, we investigate the sentiment detection capabilities of three leading LLMs by comparing the sentiment signals they generate from content posted on Twitter (now X) related to two automotive brands: Toyota Camry and FIAT 500 (L500). For the Toyota Camry, we analyse a dataset comprising approximately 163,000 tweets, posted between June 2009 and December 2020. For the FIAT 500, we examine 20,387 tweets collected over a period spanning from August 1, 2012, to December 31, 2018. Our goal is to systematically evaluate and compare the sentiment classifications produced by ChatGPT, LLaMA, and BERT across these corpora. Beyond assessing each model’s ability to capture sentiment polarity, granularity, and alignment with human intuition, we aim to explore the relationships and patterns emerging from the sentiment signals. This includes analysing how these signals vary across brands, periods, and contextual factors. Our focus is on the effectiveness of each model in capturing sentiment dynamics within real-world, domain-specific opinion data. The findings are intended to contribute to a deeper understanding of the comparative strengths and limitations of generative and transformer-based models in sentiment-oriented tasks and to inform future applications in social media monitoring, brand perception, and consumer behaviour analysis.
- Keywords:
- Large Language Models,
- ChatGpt,
- LLAMA,
- Bert,
University of Rome Tor Vergata, Italy - ORCID: 0000-0001-8041-2846
University of Rome Tor Vergata, Italy - ORCID: 0000-0003-4335-0388
- Balahur, A. (2013). Sentiment analysis in social media texts. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (pp. 120–128). Association for Computational Linguistics. DOI: 10.3115/v1/w14-2611
- Barbieri, F., Camacho-Collados, J., Espinosa-Anke, L., Neves, L. (2020). TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 1644–1650). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.148. DOI: 10.18653/v1/2020.findings-emnlp.148
- Bender, E. M., Gebru, T., McMillan-Major, A., Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623). Association for Computing Machinery. https://doi.org/10.1145/3442188.3445922. DOI: 10.1145/3442188.3445922
- Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., … Liang, P. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. https://doi.org/10.48550/arXiv.2108.07258. DOI: 10.48550/arXiv.2108.07258
- Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, pp. 1877–1901.
- Devlin, J., Chang, M.W., Lee, K., Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423. DOI: 10.18653/v1/N19-1423
- Iezzi, D.F., Monte, R. (2025). E-word of mouth in sales volume forecasting: Toyota Camry case study. Big Data Research, 41: 100542. https://doi.org/10.1016/j.bdr.2025.100542. DOI: 10.1016/j.bdr.2025.100542
- Jiang, A., Tay, Y., Liu, Q., Xu, Y., Zheng, B., and others. (2023). Mistral 7B. arXiv preprint arXiv:2310.06825. https://doi.org/10.48550/arXiv.2310.06825. DOI: 10.48550/arXiv.2310.06825
- Liu, B. (2012). Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 5(1), pp. 1–167. DOI: 10.1007/978-3-031-02145-9
- OpenAI. (2023). GPT-4 technical report. arXiv preprint arXiv:2303.08774. https://doi.org/10.48550/arXiv.2303.08774. DOI: 10.48550/arXiv.2303.08774
- Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. (2018). Improving language understanding by generative pre-training OpenAI. https://doi.org/10.48550/arXiv.1801.06146. DOI: 10.48550/arXiv.1801.06146
- Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., … Scialom, T. (2023). LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. https://doi.org/10.48550/arXiv.2302.13971. DOI: 10.48550/arXiv.2302.13971
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. Curran Associates. https://doi.org/10.48550/arXiv.1706.03762. DOI: 10.48550/arXiv.1706.03762
- Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., … Rush, A. M. (2020). Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 38–45). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-demos.6. DOI: 10.18653/v1/2020.emnlp-demos.6
- Zhang, T., Rao, Y., Xie, Y., Liu, Y. (2022). Sentiment analysis using pre-trained transformer models. Journal of Artificial Intelligence Research, 73, pp. 123–145. DOI: 10.1109/sceecs54111.2022.9741028
Informazioni sul capitolo
Titolo del capitolo
The role of large language models in generating sentiment signals: A comparative and effectiveness-based study
Autori
Domenica Fioredistella Iezzi, Roberto Monte
Lingua
Inglese
DOI
10.36253/979-12-215-1062-1.41
Opera sottoposta a peer review
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Informazioni bibliografiche
Titolo del libro
Data, Statistics and AI for Well-Being of People and Organisations
Curatori
Luigi Fabbris, Alberto Franci, Pietro Renzi
Opera sottoposta a peer review
Numero di pagine
420
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Editore
Firenze University Press
DOI
10.36253/979-12-215-1062-1
ISBN Print
979-12-215-1061-4
eISBN (pdf)
979-12-215-1062-1
eISBN (xml)
979-12-215-1063-8
Collana
Applied Statistics Series