- Data, Statistics and AI for Well-Being of People and Organisations
- A cura di Luigi Fabbris, Alberto Franci, Pietro Renzi
Modelling the human factor: Perspectivism and latent trait influence in annotated language data
- Alex Cucco
- Lara Fontanella
- Berta Chulvi
- © 2026 Author(s) |
- CC BY 4.0
- DOI: 10.36253/979-12-215-1062-1.29
The annotation of textual data is an essential phase in the development of machine learning systems, especially in natural language processing. Conventionally, annotation has sought to reduce inconsistencies among annotators to approximate a singular "ground truth," perceiving labeling as an aptitude task with a correct answer. Perspectivist approaches increasingly contest this assumption, particularly when the problem addressed is the subject of a strong social debate. In such a case, annotators face an opinion task, whose main characteristic is not the subjectivity but the fact that, looking at the way that society considers them, it seems that a correct answer does not exist. In this context, the descriptive framework for data annotation acknowledges disagreement as valuable, viewing it as indicative of the variety in human interpretation. This approach facilitates the modeling of annotator subjectivity and the capture of diverse beliefs. Previous studies indicate that disagreement often correlates with demographic and community variables; however, recent research reveals that non-demographic factors, including ideological beliefs and attitudes, may exert an even more significant influence. However, a characteristic of human beings is that attitudes do not always predict behavior or not directly predict behavior. Our study examines the relationship between attitudes and annotating behavior. Utilizing a collection of sexist jokes annotated by 76 individuals, each of whom completed a brief Hostile Neosexism scale, we investigate the impact of personal attitudes on annotation choices. We utilize IRT models to estimate latent features from the neosexism questionnaire (attitudes) and the annotation replies (behavior), and we model the correlation between the attitudes and behavior. Our findings indicate that attitudes, although not consistently predictive of action, are a pertinent element in an opinion annotation task. This understanding can enhance annotation design and annotator recruitment, and promoting more sophisticated and socially conscious data collection procedures within the descriptive framework.
- Keywords:
- Perspectivism,
- NLP,
- Latent Traits,
University of Chieti-Pescara G. d'Annunzio, Italy
University of Chieti-Pescara G. d'Annunzio, Italy - ORCID: 0000-0002-5441-0035
University of València, Spain - ORCID: 0000-0003-1169-0978
- Arthur, R. (2025). Correlation and autocorrelation of data on complex networks. EPJ Data Science, 14(1), 6. DOI: 10.1140/epjds/s13688-025-00525-1
- Basile, V., Cabitza, F., Campagner, A., Fell, M. (2021). Toward a perspectivist turn in ground truthing for predictive computing. arXiv:2109.04270. DOI: 10.1609/aaai.v37i6.25840
- Blodgett, S.L. (2021). Sociolinguistically driven approaches for just natural language processing. Doctoral dissertation, UMass Amherst.
- Chulvi, B., Fontanella, L., Labadie-Tamayo, R., Rosso, P. (2023). Social or individual disagreement? Perspectivism in the annotation of sexist jokes. In Proceedings of the CEUR Workshop (Vol. 3494). CEUR.
- Cucco, A., Del Gobbo, E., Fontanella, L., Fontanella, S., Ippoliti, L. (2025). Covering the online spectrum of opinion in social context: The benefit of network node sampling through an Italian case study. In Proceedings of the International Conference on Computational Science (pp. 60–67). Springer DOI: 10.1007/978-3-031-97554-7_5
- Frenda, S., Abercrombie, G., Basile, V., Pedrani, A., Panizzon, R., Cignarella, A. T., Marco, C., Bernardi, D. (2025). Perspectivist approaches to natural language processing: A survey. Language Resources and Evaluation, 59(2), pp. 1719–1746. DOI: 10.1007/s10579-024-09766-4
- Gordon, M. L., Zhou, K., Patel, K., Hashimoto, T., Bernstein, M. S. (2021). The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–14).
- Labadie-Tamayo, R., Chulvi, B., Rosso, P. (2023). Everybody hurts, sometimes: Overview of HUrtful HUmour at IberLEF 2023: Detection of humour spreading prejudice in Twitter. Procesamiento del Lenguaje Natural (SEPLN), 71
- Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Retrieved from https://CRAN.R-project.org/package=lavaan. R package version 0.6-15. DOI: 10.18637/jss.v048.i02
- Wirth, R.J., Edwards, M.C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12(1), pp. 58–79 DOI: 10.1037/1082-989x.12.1.58.supp
- Sap, M., Swayamdipta, S., Vianna, L., Zhou, X., Choi, Y., Smith, N. A. (2022). Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 5884–5906). Association for Computational Linguistics, Seattle (WA). DOI: 10.18653/v1/2022.naacl-main.431
Informazioni sul capitolo
Titolo del capitolo
Modelling the human factor: Perspectivism and latent trait influence in annotated language data
Autori
Alex Cucco, Lara Fontanella, Berta Chulvi
Lingua
Inglese
DOI
10.36253/979-12-215-1062-1.29
Opera sottoposta a peer review
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Informazioni bibliografiche
Titolo del libro
Data, Statistics and AI for Well-Being of People and Organisations
Curatori
Luigi Fabbris, Alberto Franci, Pietro Renzi
Opera sottoposta a peer review
Numero di pagine
420
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Editore
Firenze University Press
DOI
10.36253/979-12-215-1062-1
ISBN Print
979-12-215-1061-4
eISBN (pdf)
979-12-215-1062-1
eISBN (xml)
979-12-215-1063-8
Collana
Applied Statistics Series