<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="gao-etal-2018-automated">
    <titleInfo>
        <title>Automated Content Analysis: A Case Study of Computer Science Student Summaries</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yanjun</namePart>
        <namePart type="family">Gao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Patricia</namePart>
        <namePart type="given">M</namePart>
        <namePart type="family">Davies</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rebecca</namePart>
        <namePart type="given">J</namePart>
        <namePart type="family">Passonneau</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2018-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Joel</namePart>
            <namePart type="family">Tetreault</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jill</namePart>
            <namePart type="family">Burstein</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ekaterina</namePart>
            <namePart type="family">Kochmar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Claudia</namePart>
            <namePart type="family">Leacock</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Helen</namePart>
            <namePart type="family">Yannakoudakis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">New Orleans, Louisiana</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Technology is transforming Higher Education learning and teaching. This paper reports on a project to examine how and why automated content analysis could be used to assess precis writing by university students. We examine the case of one hundred and twenty-two summaries written by computer science freshmen. The texts, which had been hand scored using a teacher-designed rubric, were autoscored using the Natural Language Processing software, PyrEval. Pearson’s correlation coefficient and Spearman rank correlation were used to analyze the relationship between the teacher score and the PyrEval score for each summary. Three content models automatically constructed by PyrEval from different sets of human reference summaries led to consistent correlations, showing that the approach is reliable. Also observed was that, in cases where the focus of student assessment centers on formative feedback, categorizing the PyrEval scores by examining the average and standard deviations could lead to novel interpretations of their relationships. It is suggested that this project has implications for the ways in which automated content analysis could be used to help university students improve their summarization skills.</abstract>
    <identifier type="citekey">gao-etal-2018-automated</identifier>
    <identifier type="doi">10.18653/v1/W18-0531</identifier>
    <location>
        <url>https://aclanthology.org/W18-0531</url>
    </location>
    <part>
        <date>2018-06</date>
        <extent unit="page">
            <start>264</start>
            <end>272</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Automated Content Analysis: A Case Study of Computer Science Student Summaries
%A Gao, Yanjun
%A Davies, Patricia M.
%A Passonneau, Rebecca J.
%Y Tetreault, Joel
%Y Burstein, Jill
%Y Kochmar, Ekaterina
%Y Leacock, Claudia
%Y Yannakoudakis, Helen
%S Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications
%D 2018
%8 June
%I Association for Computational Linguistics
%C New Orleans, Louisiana
%F gao-etal-2018-automated
%X Technology is transforming Higher Education learning and teaching. This paper reports on a project to examine how and why automated content analysis could be used to assess precis writing by university students. We examine the case of one hundred and twenty-two summaries written by computer science freshmen. The texts, which had been hand scored using a teacher-designed rubric, were autoscored using the Natural Language Processing software, PyrEval. Pearson’s correlation coefficient and Spearman rank correlation were used to analyze the relationship between the teacher score and the PyrEval score for each summary. Three content models automatically constructed by PyrEval from different sets of human reference summaries led to consistent correlations, showing that the approach is reliable. Also observed was that, in cases where the focus of student assessment centers on formative feedback, categorizing the PyrEval scores by examining the average and standard deviations could lead to novel interpretations of their relationships. It is suggested that this project has implications for the ways in which automated content analysis could be used to help university students improve their summarization skills.
%R 10.18653/v1/W18-0531
%U https://aclanthology.org/W18-0531
%U https://doi.org/10.18653/v1/W18-0531
%P 264-272

Markdown (Informal)

[Automated Content Analysis: A Case Study of Computer Science Student Summaries](https://aclanthology.org/W18-0531) (Gao et al., BEA 2018)

• Automated Content Analysis: A Case Study of Computer Science Student Summaries (Gao et al., BEA 2018)

ACL

• Yanjun Gao, Patricia M. Davies, and Rebecca J. Passonneau. 2018. Automated Content Analysis: A Case Study of Computer Science Student Summaries. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 264–272, New Orleans, Louisiana. Association for Computational Linguistics.