Deep Learning Super Sampling

Video game graphics are constantly evolving, becoming more and more computationally intensive. Complex particle physics, real time lightning using ray-tracing effects, photo realistic textures and materials, all contribute to the improvements of graphics. Although these improvements are impressive, the processing power requirements from graphical processing units (GPUs) are becoming so high, that hardware limitations are becoming an issue. GPUs have plateaued by reaching atomic level transistor sizes, meaning that to further increase the power, the size and power consumption must increase at an exponential level.

Image quality has improved by solving issues such as the aliasing problem. Jagged and pixelated edges from continuous smooth curves and lines can drastically reduce the image quality. Traditionally, this was solved by rendering the image at a higher resolution than the one being displayed, then scaling it back down leaving us with extra pixels which are used for different color calculations. This is an extremely intensive process that drastically decreases the performance of video games. To overcome this issue, new methods of improving performance have been devised, using new technologies such as deep learning. 

DLSS (Deep Learning Super Sampling), is an NVIDIA artificial intelligence assisted broadcasting feature, which with the help of dedicated AI processors (Tensor cores) from NVIDIA GPUs, render fewer pixels and use AI to construct sharp and higher resolution images.

Trained by a NVIDIA supercomputer, the AI network receives two primary inputs:

1. Low resolution, aliased images rendered by the game engine

2. Low resolution, motion vectors from the same images -- also generated by the game engine

The mentioned AI network is a special type called a convolutional autoencoder, which uses the motion vectors from the previous frame to predict what the next frame will look like. This process is called ‘temporal feedback’. In other words the AI takes the low resolution current frame, and the high resolution previous frame, to determine how to generate a higher quality current frame.

To train such a network, NVIDIA takes the output image and compares it to an offline rendered, ultra-quality 16K reference image, and the difference is communicated back into the network so that it can improve. This process is repeated tens of thousands of times. This however requires special processing units found on NVIDIA GPUs, which offer 110 teraflops of dedicated AI horsepower. 

With the help of DLSS, the video game industry managed to considerably increase the quality of the images, whilst no longer worrying about huge performance losses from traditional supersampling algorithms. Further developments in the supersampling field can be expected from other manufacturers such as AMD, although still far behind what NVIDIA has managed to achieve.

References: 

[1] Watson, Alexander. "Deep learning techniques for super-resolution in video games." arXiv preprint arXiv:2012.09810 (2020).

[2] https://www.nvidia.com/en-us/data-center/tensor-cores/

[3] https://en.wikipedia.org/wiki/Supersampling

[4] https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-2-0-a-big-leap-in-ai-rendering/