- Wikidata e la ricerca. Condividere esperienze / Wikidata and Research. Sharing Experiences
- Edited by Elena Marangoni, Camillo Carlo Pellizzari di San Girolamo
HQWiki: A Universal Pipeline for High-Quality Monolingual Dataset Creation
- Iaroslav Chelombitko
- Aleksey Komissarov
- © 2026 Author(s) |
- CC BY 4.0
- DOI: 10.36253/979-12-215-1010-2.44
Creating high-quality monolingual datasets poses significant challenges due to cross-language contamination, metadata noise, and inconsistent quality control, especially for less-resourced languages. We present HQWiki, a pipeline for extracting and validating clean monolingual content from Wikipedia. The framework introduces universal approaches combining character set validation, context-aware filtering, and adaptive language detection. Evaluation across multiple language families demonstrated significant noise reduction while maintaining high accuracy in language identification. The modular architecture allows researchers to adapt filtering criteria for specific requirements, providing a foundation for systematic dataset creation across different language families.
- Keywords:
- dataset creation,
- monolingual corpora,
- Wikipedia processing,
- language identification,
- less-resourced languages,
Neapolis University Pafos, Cyprus - ORCID: 0009-0003-6843-0453
aglabx, Cyprus
- Chelombitko, Iaroslav, and Aleksey Komissarov. 2024. “Specialized Monolingual BPE Tokenizers for Uralic Languages Representation in Large Language Models.” In Proceedings of the 8th International Workshop on Computational Linguistics for Uralic Languages, 115–25. Stroudsburg, PA: Association for Computational Linguistics. https://aclanthology.org/2024.iwclul-1.11/
- Chelombitko, Iaroslav, Egor Safronov, and Aleksey Komissarov. 2024. “Qtok: A Comprehensive Framework for Evaluating Multilingual Tokenizer Quality in Large Language Models.” arXiv preprint arXiv:2410.12989. https://arxiv.org/abs/2410.12989
Chapter Information
Chapter Title
HQWiki: A Universal Pipeline for High-Quality Monolingual Dataset Creation
Authors
Iaroslav Chelombitko, Aleksey Komissarov
Language
English
DOI
10.36253/979-12-215-1010-2.44
Peer Reviewed
Publication Year
2026
Copyright Information
© 2026 Author(s)
Content License
Metadata License
Bibliographic Information
Book Title
Wikidata e la ricerca. Condividere esperienze / Wikidata and Research. Sharing Experiences
Book Subtitle
Atti del Convegno, Firenze, 5-6 giugno 2025 / Conference proceedings, Florence, 5-6 June 2025
Editors
Elena Marangoni, Camillo Carlo Pellizzari di San Girolamo
Peer Reviewed
Number of Pages
378
Publication Year
2026
Copyright Information
© 2026 Author(s)
Content License
Metadata License
Publisher Name
Firenze University Press
DOI
10.36253/979-12-215-1010-2
ISBN Print
979-12-215-1009-6
eISBN (pdf)
979-12-215-1010-2
Series Title
Studi e saggi
Series ISSN
2704-6478
Series E-ISSN
2704-5919