[go: up one dir, main page]

RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian

Adrian Cosma, Ioan-Bogdan Iordache, Paolo Rosso


Abstract
Recently, large language models (LLMs) have become increasingly powerful and have become capable of solving a plethora of tasks through proper instructions in natural language. However, the vast majority of testing suites assume that the instructions are written in English, the de facto prompting language. Code intelligence and problem solving still remain a difficult task, even for the most advanced LLMs. Currently, there are no datasets to measure the generalization power for code-generation models in a language other than English. In this work, we present RoCode, a competitive programming dataset, consisting of 2,642 problems written in Romanian, 11k solutions in C, C++ and Python and comprehensive testing suites for each problem. The purpose of RoCode is to provide a benchmark for evaluating the code intelligence of language models trained on Romanian / multilingual text as well as a fine-tuning set for pretrained Romanian models. Through our results and review of related works, we argue for the need to develop code models for languages other than English.
Anthology ID:
2024.lrec-main.1236
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
14173–14185
Language:
URL:
https://aclanthology.org/2024.lrec-main.1236
DOI:
Bibkey:
Cite (ACL):
Adrian Cosma, Ioan-Bogdan Iordache, and Paolo Rosso. 2024. RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14173–14185, Torino, Italia. ELRA and ICCL.
Cite (Informal):
RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian (Cosma et al., LREC-COLING 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.lrec-main.1236.pdf