- Wikidata e la ricerca. Condividere esperienze / Wikidata and Research. Sharing Experiences
- A cura di Elena Marangoni, Camillo Carlo Pellizzari di San Girolamo
HQWiki: A Universal Pipeline for High-Quality Monolingual Dataset Creation
- Iaroslav Chelombitko
- Aleksey Komissarov
- © 2026 Author(s) |
- CC BY 4.0
- DOI: 10.36253/979-12-215-1010-2.44
Creating high-quality monolingual datasets poses significant challenges due to cross-language contamination, metadata noise, and inconsistent quality control, especially for less-resourced languages. We present HQWiki, a pipeline for extracting and validating clean monolingual content from Wikipedia. The framework introduces universal approaches combining character set validation, context-aware filtering, and adaptive language detection. Evaluation across multiple language families demonstrated significant noise reduction while maintaining high accuracy in language identification. The modular architecture allows researchers to adapt filtering criteria for specific requirements, providing a foundation for systematic dataset creation across different language families.
- Keywords:
- dataset creation,
- monolingual corpora,
- Wikipedia processing,
- language identification,
- less-resourced languages,
Neapolis University Pafos, Cyprus - ORCID: 0009-0003-6843-0453
aglabx, Cyprus
- Chelombitko, Iaroslav, and Aleksey Komissarov. 2024. “Specialized Monolingual BPE Tokenizers for Uralic Languages Representation in Large Language Models.” In Proceedings of the 8th International Workshop on Computational Linguistics for Uralic Languages, 115–25. Stroudsburg, PA: Association for Computational Linguistics. https://aclanthology.org/2024.iwclul-1.11/
- Chelombitko, Iaroslav, Egor Safronov, and Aleksey Komissarov. 2024. “Qtok: A Comprehensive Framework for Evaluating Multilingual Tokenizer Quality in Large Language Models.” arXiv preprint arXiv:2410.12989. https://arxiv.org/abs/2410.12989
Informazioni sul capitolo
Titolo del capitolo
HQWiki: A Universal Pipeline for High-Quality Monolingual Dataset Creation
Autori
Iaroslav Chelombitko, Aleksey Komissarov
Lingua
Inglese
DOI
10.36253/979-12-215-1010-2.44
Opera sottoposta a peer review
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Informazioni bibliografiche
Titolo del libro
Wikidata e la ricerca. Condividere esperienze / Wikidata and Research. Sharing Experiences
Sottotitolo del libro
Atti del Convegno, Firenze, 5-6 giugno 2025 / Conference proceedings, Florence, 5-6 June 2025
Curatori
Elena Marangoni, Camillo Carlo Pellizzari di San Girolamo
Opera sottoposta a peer review
Numero di pagine
378
Anno di pubblicazione
2026
Copyright
© 2026 Author(s)
Licenza d'uso
Licenza dei metadati
Editore
Firenze University Press
DOI
10.36253/979-12-215-1010-2
ISBN Print
979-12-215-1009-6
eISBN (pdf)
979-12-215-1010-2
Collana
Studi e saggi
ISSN della collana
2704-6478
e-ISSN della collana
2704-5919