Abstract
We present experiments in predicting a discourse sense for explicit inter-sentential discourse relations in Czech and English, using embedding and deep learning (fine-tuned BERT) to predict the senses, and annotation projection from English to Czech to increase the size of training data for Czech. We show that strengthening information about the explicit connective improves the results and, on the other hand, that annotation projection from an annotation system with a different set of senses may not help if used in a straightforward way, which we discuss in detail.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
For Czech, we also experimented with the DeepPavlov model trained for four Slavic languages (https://huggingface.co/DeepPavlov/bert-base-bg-cs-pl-ru-cased), with the same results.
- 3.
Meaning “without the additional training data”.
- 4.
We use the term “explicit discourse relation” in a broad sense, i.e. a discourse relation explicitly marked by a connective or an alternative lexicalization in the terms used in the PDTB, or by both primary and secondary connectives in the terms used in the PDT.
- 5.
Please note again that we understand a connective here as a broad class that includes alternative lexicalizations.
References
Devlin J, Chang MW, Lee K, Toutanova K (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, pp 4171–4186
Hajič J, Hajicová E, Panevová J, Sgall P, Bojar O, Cinková S, Fucíková E, Mikulová M, Pajas P, Popelka J et al (2012) Announcing prague Czech–English dependency treebank 2.0. In: LREC, pp 3153–3160
Hajič J, Hajiová E, Panevová J, Sgall P, Cinková S, Fucíková E, Mikulová M, Pajas P, Popelka J, Semecký J, Šindlerová J, Štěpánek J, Toman J, Urešová Z, Žabokrtský Z (2012) Prague Czech-English dependency treebank 2.0. data/software, linguistic data consortium. University of Pennsylvania, Philadelphia. LDC2012T08
Hajič J, Bejček E, Bémová A, Buráňová E, Hajičová E, Havelka J, Homola P, Kárník J, Kettnerová V, Klyueva N, Kolářová V, Kučová L, Lopatková M, Mikulová M, Mírovský J, Nedoluzhko A, Pajas P, Panevová J, Poláková L, Rysová M, Sgall P, Spoustová J, Straňák P, Synková P, Ševčíková M, Štěpánek J, Urešová Z, Hladká BV, Zeman D, Zikánová Š, Žabokrtský Z (2018) Prague dependency treebank 3.5. data/software, Univerzita Karlova, MFF, ÚFAL, Prague, Czech Republic
Lin Z, Ng HT, Kan MY (2014) A PDTB-styled end-to-end discourse parser. Nat Lang Eng 20(2):151–184
Marcus MP, Santorini B, Marcinkiewicz MA (1995) Treebank-2. data/software, linguistic data consortium. University of Pennsylvania, Philadelphia. LDC95T7
Poláková L, Jínová P, Zikánová Š, Hajičová E, Mírovský J, Nedoluzhko A, Rysová M, Pavlíková V, Zdeňková J, Pergler J, Ocelák R (2012) Prague discourse treebank 1.0. data/software, ÚFAL MFF UK, Prague, Czech Republic
Poláková L, Mírovský J, Nedoluzhko A, Jínová P, Zikánová Š, Hajičová E (2013) Introducing the prague discourse treebank 1.0. In: Proceedings of the 6th international joint conference on natural language processing. Asian Federation of Natural Language Processing, Asian Federation of Natural Language Processing, Nagoya, Japan, pp 91–99
Prasad R, Lee A, Dinesh N, Miltsakaki E, Campion G, Joshi A, Webber B (2008) Penn discourse treebank version 2.0. data/software, linguistic data consortium. University of Pennsylvania, Philadelphia. LDC2008T05
Prasad R, Webber B, Lee A, Joshi A (2019) Penn discourse treebank version 3.0. data/software, linguistic data consortium. University of Pennsylvania, Philadelphia. LDC2019T05
Shi W, Demberg V (2019) Next sentence prediction helps implicit discourse relation classification within and across domains. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 5794–5800
Varia S, Hidey C, Chakrabarty T (2019) Discourse relation prediction: revisiting word pairs with convolutional networks. In: Proceedings of the 20th annual SIGdial meeting on discourse and dialogue, pp 442–452
Weiss G, Bajec M (2018) Sense classification of shallow discourse relations with focused rnns. PloS One 13(10):e0206,057
Xue N, Ng HT, Pradhan S, Rutherford A, Webber B, Wang C, Wang H (2016) CoNLL 2016 shared task on multilingual shallow discourse parsing. In: Proceedings of the CoNLL-16 shared task, pp 1–19
Acknowledgements
The authors gratefully acknowledge support from the Grant Agency of the Czech Republic (project GA19-03490S). The research reported in the present contribution has been using language resources developed, stored and distributed by the LINDAT/CLARIAH-CZ project of the Ministry of Education, Youth and Sports of the Czech Republic (LM2018101).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Mírovský, J., Poláková, L. (2022). Sense Prediction for Explicit Discourse Relations with BERT. In: Yang, XS., Sherratt, S., Dey, N., Joshi, A. (eds) Proceedings of Sixth International Congress on Information and Communication Technology. Lecture Notes in Networks and Systems, vol 216. Springer, Singapore. https://doi.org/10.1007/978-981-16-1781-2_72
Download citation
DOI: https://doi.org/10.1007/978-981-16-1781-2_72
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-1780-5
Online ISBN: 978-981-16-1781-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)