UniFormerV2

This repo is the official implementation of "UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer". By Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang and Yu Qiao.

Update

11/14/2023

Thanks for Innat'help @innat. Now our models also support Keras! 😄

07/14/2023

UniFormerV2 has been accepted by ICCV2023! 🎉

02/13/2023

UniFormerV2 has been integrated into MMAction2. Training code will be provided soon! 😄

11/20/2022

We give a video demo in hugging face. Have a try! 😄

11/19/2022

We give a blog in Chinese Zhihu.

11/18/2022

All the code, models and configs are provided. Don't hesitate to open an issue if you have any problem! 🙋🏻

Introduction

In UniFormerV2, we propose a generic paradigm to build a powerful family of video networks, by arming the pre-trained ViTs with efficient UniFormer designs. It inherits the concise style of the UniFormer block. But it contains brand- new local and global relation aggregators, which allow for preferable accuracy-computation balance by seamlessly integrating advantages from both ViTs and UniFormer. It gets the state-of-the-art recognition performance on 8 popular video benchmarks, including scene-related Kinetics-400/600/700 and Moments in Time, temporal-related Something-Something V1/V2, untrimmed ActivityNet and HACS. In particular, it is the first model to achieve 90% top-1 accuracy on Kinetics-400.

Model Zoo

All the models can be found in MODEL_ZOO.

Instructions

See INSTRUCTIONS for more details about:

• Environment installation
• Dataset preparation
• Training and validation

Cite Uniformer

If you find this repository useful, please use the following BibTeX entry for citation.

@misc{li2022uniformerv2,
      title={UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer}, 
      author={Kunchang Li and Yali Wang and Yinan He and Yizhuo Li and Yi Wang and Limin Wang and Yu Qiao},
      year={2022},
      eprint={2211.09552},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

License

This project is released under the MIT license. Please see the LICENSE file for more information.

Acknowledgement

This repository is built based on UniFormer and SlowFast repository.