<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="hasegawa-etal-2023-duet">
    <titleInfo>
        <title>DueT: Image-Text Contrastive Transfer Learning with Dual-adapter Tuning</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Taku</namePart>
        <namePart type="family">Hasegawa</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kyosuke</namePart>
        <namePart type="family">Nishida</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Koki</namePart>
        <namePart type="family">Maeda</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kuniko</namePart>
        <namePart type="family">Saito</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>This paper presents DueT, a novel transfer learning method for vision and language models built by contrastive learning. In DueT, adapters are inserted into the image and text encoders, which have been initialized using models pre-trained on uni-modal corpora and then frozen. By training only these adapters, DueT enables efficient learning with a reduced number of trainable parameters. Moreover, unlike traditional adapters, those in DueT are equipped with a gating mechanism, enabling effective transfer and connection of knowledge acquired from pre-trained uni-modal encoders while preventing catastrophic forgetting. We report that DueT outperformed simple fine-tuning, the conventional method fixing only the image encoder and training only the text encoder, and the LoRA-based adapter method in accuracy and parameter efficiency for 0-shot image and text retrieval in both English and Japanese domains.</abstract>
    <identifier type="citekey">hasegawa-etal-2023-duet</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.839</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.839</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>13607</start>
            <end>13624</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T DueT: Image-Text Contrastive Transfer Learning with Dual-adapter Tuning
%A Hasegawa, Taku
%A Nishida, Kyosuke
%A Maeda, Koki
%A Saito, Kuniko
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F hasegawa-etal-2023-duet
%X This paper presents DueT, a novel transfer learning method for vision and language models built by contrastive learning. In DueT, adapters are inserted into the image and text encoders, which have been initialized using models pre-trained on uni-modal corpora and then frozen. By training only these adapters, DueT enables efficient learning with a reduced number of trainable parameters. Moreover, unlike traditional adapters, those in DueT are equipped with a gating mechanism, enabling effective transfer and connection of knowledge acquired from pre-trained uni-modal encoders while preventing catastrophic forgetting. We report that DueT outperformed simple fine-tuning, the conventional method fixing only the image encoder and training only the text encoder, and the LoRA-based adapter method in accuracy and parameter efficiency for 0-shot image and text retrieval in both English and Japanese domains.
%R 10.18653/v1/2023.emnlp-main.839
%U https://aclanthology.org/2023.emnlp-main.839
%U https://doi.org/10.18653/v1/2023.emnlp-main.839
%P 13607-13624

Markdown (Informal)

[DueT: Image-Text Contrastive Transfer Learning with Dual-adapter Tuning](https://aclanthology.org/2023.emnlp-main.839) (Hasegawa et al., EMNLP 2023)

• DueT: Image-Text Contrastive Transfer Learning with Dual-adapter Tuning (Hasegawa et al., EMNLP 2023)

ACL

• Taku Hasegawa, Kyosuke Nishida, Koki Maeda, and Kuniko Saito. 2023. DueT: Image-Text Contrastive Transfer Learning with Dual-adapter Tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13607–13624, Singapore. Association for Computational Linguistics.