The goal of this repository is to enable real time super resolution for upsampling low resolution videos. Currently, the design follows the SR-GAN architecture. For speed, the upsampling is done through pixel shuffle.
The training setup looks like the following diagram:
The following runtimes/fps are obtained by averaging runtimes over 800 frames. Measured on MPS (MacBook M1 Pro GPU).
| Input Image Size | Output Size | Time (s) | FPS |
|---|---|---|---|
| 90x160 | 360x640 (360p) | 0.01 | 82 |
| 180x320 | 720x1080 (720p) | 0.04 | 27 |
We see it's possible to upsample to 720p at around 30fps.
This was tested on Python 3.10. To install the required packages, use the provided Pipfile:
pip install pipenv --upgrade
pipenv install --system --deployA pretrained generator model on the DIV2k dataset is provided in the 'models' directory. It uses 8 residual blocks, with 64 filters in every layer of the generator.
To try out the provided pretrained model on your own images, run the following:
python inference.py --image_dir 'path/to/your/image/directory' --output_dir 'path/to/save/super/resolution/images'To train, simply edit the config file in the folder configs/config.yaml with your settings, and then launch the training with:
python train.pyYou can also change the config parameters from the command line. The following will run training with a batch_size of 32, a generator with 12 residual blocks, and a path to the image directory /path/to/image/dataset.
python train.py data.image_dir="/path/to/image/dataset" training.batch_size=32 generator.n_layers=12
This is powered by hydra, which means all the parameters in the config are editable via the CLI.
Model checkpoints and training summaries are saved in tensorboard. To monitor training progress, open up tensorboard by pointing it to the outputs directory that will be created when you start training.
Following are some results from the provided trained model. Left shows the low res image, after 4x bicubic upsampling. Middle is the output of the model. Right is the actual high resolution image.
The following shows images upsampled 4x by bicubic interpolation, the pretrained model from this repository and the original high resolution image as a comparison
If you have ideas on improving model performance, adding metrics, or any other changes, please make a pull request or open an issue. I'd be happy to accept any contributions.