- Data, Statistics and AI for Well-Being of People and Organisations
- A cura di Luigi Fabbris, Alberto Franci, Pietro Renzi
Human-centered multi-label annotation for made-in-Italy product reviews: Evaluating prompting strategies with multilingual LLMs
- Mahir Selek
- Nicolò Biasetton
- Luigi Salmaso
- © 2026 Author(s) |
- CC BY 4.0
- DOI: 10.36253/979-12-215-1062-1.66
Accomplishing human-aligned multi-label annotation in text classification remains a substantial challenge in natural language processing (NLP), particularly in specialized domains. This study addresses this gap by evaluating large language models (LLMs) for multilabel classification of Amazon customer reviews in Italian language performed on the dimensions of Product Attributes (PA) and Customer Values (CV). These two dimensions, validated in customer analysis literature, are central to understanding customer behaviour and perception to guide product development, refine market segmentation, and improve customer overall experience. The ability to extract structured insights from unstructured feedback supports scalable, data-driven decision-making. However, achieving accurate and consistent annotation across large datasets requires methods that align closely with human reasoning. With this purpose, based on a critical review of existing literature, this work examines many techniques—such as prompting and augmentation strategies—for their effectiveness in achieving cautious and high-precision annotations within a specialized vertical domain. Three foundation models with multilingual abilities—Mistral-large, LLaMA 3.3, and Anthropic Claude Haiku 3.5— are evaluated for performance under firmly controlled prompting arrangements. Approaches including few-shot learning, retrieval-augmented generation (RAG) and BM25 (Best Matching 25)-based context ranking are consistently examined to determine contributions of these techniques to annotation quality. Among these, few-shot prompting, enhanced by carefully engineered high-precision prompts, reveals high-level alignment with human annotations, especially when implemented using the Mistral model. Evaluation against a human-annotated gold standard signifies that the Mistral model with few-shot learning achieves the highest alignment with human annotations for both PA and CV, outperforming zero-shot and RAG-based approaches. These findings emphasise the importance of precise prompt engineering and controlled design in achieving human-close annotation uniqueness, offering practical guidance for deploying LLMs in critical annotation workflows while contributing a replicable, human-centered evaluation methodology.
- Keywords:
- Natural Language Processing (NLP),
- Topic Modeling,
- User-Generated Reviews,
University of Padua, Italy - ORCID: 0009-0008-1831-3204
University of Padua, Italy - ORCID: 0000-0002-1123-9899
University of Padua, Italy - ORCID: 0000-0001-6501-1585
- Chirkova, N., Rau, D., Déjean, H., Formal, T., Clinchant, S., Nikoulina, V. (2024). Retrieval-augmented generation in multilingual settings. arXiv preprint arXiv: 2407.01463.
- Dogruoz, A.S., Sitaram, S., Bullock, B., Toribio, A.J. (2021). A survey of code-switching: Linguistic and social perspectives for language technologies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1654–1666. DOI: 10.18653/v1/2021.acl-long.131
- Herrera, F., Charte, F., Rivera, A.J., Del Jesus, M.J. (2016). Multilabel Classification. Springer. DOI: 10.1007/978-3-319-41111-8_2
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459–9474. DOI: 10.18653/v1/2021.naacl-main.200
- Li, G., Yang, L., Zhang, B., Li, X., Chen, F. (2021). How do environmental values impact green product purchase intention? The moderating role of green trust. Environmental Science and Pollution Research, 28: pp. 46020–46034. DOI: 10.1007/s11356-021-13946-y
- Liu, Y., Fabbri, A.R., Liu, P., Zhao, Y., Nan, L., Han, R., Han, S., Joty, S., Wu, C.S., Xiong, C., Radev, D. (2022). Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. arXiv preprint arXiv:2212.07981, 2022. DOI: 10.18653/v1/2023.acl-long.228
- Marchal, M., Scholman, M., Yung, F., Demberg, V. (2022). Establishing annotation quality in multi-label annotations. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 3659–3668.
- Perez, E., Kiela, D., Cho., K. (2021). True few-shot learning with language models. Advances in Neural Information Processing Systems, 34: pp. 11054–11070.
- Radiya-Dixit, E., Wang, X. (2020). How fine can fine-tuning be? learning efficient language models. In International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, pp. 2435-2443.
- Sheth, J. N., Newman, B.I., Gross., B.L. (1991). Why we buy what we buy: A theory of consumption values. Journal of Business Research, 22(2): pp. 159–170. DOI: 10.1016/0148-2963(91)90050-8
- Törnberg., P. (2024). Best practices for text annotation with large language models. arXiv preprint arXiv: 2402.05129 DOI: 10.4135/9781529683707
Informazioni sul capitolo
Titolo del capitolo
Human-centered multi-label annotation for made-in-Italy product reviews: Evaluating prompting strategies with multilingual LLMs
Autori
Mahir Selek, Nicolò Biasetton, Luigi Salmaso
Lingua
Inglese
DOI
10.36253/979-12-215-1062-1.66
Opera sottoposta a peer review
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Informazioni bibliografiche
Titolo del libro
Data, Statistics and AI for Well-Being of People and Organisations
Curatori
Luigi Fabbris, Alberto Franci, Pietro Renzi
Opera sottoposta a peer review
Numero di pagine
420
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Editore
Firenze University Press
DOI
10.36253/979-12-215-1062-1
ISBN Print
979-12-215-1061-4
eISBN (pdf)
979-12-215-1062-1
eISBN (xml)
979-12-215-1063-8
Collana
Applied Statistics Series