This is an attempt to reproduce the results and findings of LaViSE: Explaining Deep Convolutional Neural Networks via Latent Visual-Semantic Filter Attention.
The environment necessary to run the code can be created with the environment_lisa.yml file when running on lisa cluster and the environment_windesk when running locally on windows.
To install the data, you can run the install_data.sh script. Alternatively, you can install them manually by following the instructions on Visual Genome Python Driver and COCO API WRAPPER. The wrappers and api are already installed in the environment.
To format the Visual Genome object, run the notebook.
There are two job scripts available to run the code. The train_explainer.job file trains with a resnet18 model with Visual Genome as reference dataset. The infer_filter.job file contains several experiments that test different parameters, most of which have been commented out.
Alternatively, you can manually run
python train_explainer.py --refer <reference_dataset> --epochs <number_of_epochs> --anno_rate <annotation> --name <run_name>
on your desktop to train the explainer, and
python infer_filter.py --refer <reference_dataset> --anno_rate <annotation> --name <run_name>
to visualize the filters. Note that both parts should have matching hyperparameters.
This is the official repository for paper "Explaining Deep Convolutional Neural Networks via Unsupervised Visual-Semantic Filter Attention" to appear in CVPR 2022.
Authors: Yu Yang, Seungbae Kim, Jungseock Joo
- Please follow the instructions in the COCO API README and here to download and setup the COCO data.
- Please follow the instructions in the README of the python wrapper for the Visual Genome API and here.
- We load the pretrained GloVe word embeddings directly from the torchtext library.
- The list of entities used to discover new concepts is provided in
data/entities.txt.
Required packages can be found in requirements.txt.
Train an explainer with
python train_explainer.py
Explain a target filter of any model with
python infer_filter.py
More features will be added soon! 🍻
@inproceedings{yang2022explaining,
author = {Yang, Yu and Kim, Seungbae and Joo, Jungseock},
title = {Explaining Deep Convolutional Neural Networks via Unsupervised Visual-Semantic Filter Attention},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2022},
}