Abstract
Most existing Continuous Sign Language Recognition (CSLR) benchmarks have fixed backgrounds and are filmed in studios with a static monochromatic background. However, signing is not limited only to studios in the real world. In order to analyze the robustness of CSLR models under background shifts, we first evaluate existing state-of-the-art CSLR models on diverse backgrounds. To synthesize the sign videos with a variety of backgrounds, we propose an algorithm to automatically generate a benchmark dataset utilizing existing CSLR benchmarks. Our newly constructed benchmark dataset simulates diverse scenes to simulate a real-world environment. Interestingly, we observe even the most recent CSLR method cannot recognize glosses well on our new dataset with changed backgrounds. In this regard, we also propose a simple yet effective training scheme including (1) background randomization and (2) feature disentanglement for CSLR models. The experimental results on our dataset demonstrate that our method generalizes well to other unseen background data with minimal additional training images.Video
Citation
@inproceedings{Jang_2022_BMVC,
author = {Youngjoon Jang and Youngtaek Oh and Jae Won Cho and Dong-Jin Kim and Joon Son Chung and In So Kweon},
title = {Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0322.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by