neural-super-sampling

• New architecture and trained weights. • Higher image quality compared to NSS v0.1, with comparable runtime performance. • Three quality modes (High, Mid, and Low) let you balance image quality and performance. • Updated example scenarios using fragment shaders. Neural Super Sampling (NSS) is an innovative, efficient network for temporal super sampling on mobile devices. Content rendered at a low resolution can be upscaled to a higher resolution, resulting in up to 50% GPU savings.

With our retraining tools content creators and game studios can build derivatives of the model suited to artwork style and performance requirements.

🎥 Neural Super Sampling Demo

Your browser does not support the video tag.

Model Details

Neural Super Sampling (NSS) is a parameter prediction model for real-time temporal super sampling developed by Arm, optimized for execution on Neural Accelerators (NX) in mobile GPUs. It enables high-resolution rendering at a lower compute cost by reconstructing high-quality output frames from low-resolution temporal inputs. NSS is particularly suited for mobile gaming, XR, and other power-constrained graphics use cases.

NSS v1 provides three quality modes that allow you to balance image quality and runtime performance:

• high: highest runtime cost but best-quality option. Checks the current frame and depth/motion detail more thoroughly, uses a larger image filter for clean-up, and samples previous frames more accurately.
• low: lowest runtime cost but lowest-quality option. Uses lighter current-frame and depth/motion checks, and samples previous frames less accurately. May show more flicker or motion artifacts around fine detail and moving objects.
• mid: balanced option. Similar to low but samples previous frames more accurately, like high.

For a more detailed explanation of these quality modes, refer to the Neural Graphics Model Gym documentation.

NSS is under active development with regular updates planned. As we increase the size and diversity of the training dataset we expect to see significant quality improvements. Follow Arm to stay up to date on the latest releases.

The model is released under Arm's permissive AI Model Community License, designed to foster innovation in the graphics industry and provide differentiation to content creators. It allows NSS to be retrained on datasets captured from your own content.

A plugin for capturing data in Unreal Engine is available to capture your own content for use in (re)training or evaluation in the Neural Graphics Model Gym.

Integration

NSS can be directly integrated into graphics pipelines using ML Extensions for Vulkan.

The recommended way of integrating the model into a graphics pipeline is by using the VGF Library from the ML SDK for Vulkan.

Integration in Game Engines

NSS has been integrated into Unreal Engine via the NSS Plugin for Unreal Engine. See the Learning Path for step-by-step instructions on how to use NSS in Unreal Engine.

It is also available to use as part of the Neural Graphics SDK for Game Engines. This is designed to help integrate NSS across a wide variety of game engines.

Getting Started

This repository contains pre-trained weights and compiled NSS models in VGF format, ready for integration with Vulkan applications. These VGF models are unshaped, so will require developers to shape-specialize them before use. An example of how to do this can be seen in the Neural Graphics SDK for Game Engines.

See the included ML SDK for Vulkan scenarios for the simplest way to evaluate the model. The scenarios include the necessary pre- and post-processing fragment shaders along with a single frame worth of input data.

They demonstrate full execution of NSS on a Vulkan compute-capable system. The Emulation Layer can be used to implement ML Extensions for Vulkan where it is not supported by the native Vulkan driver.

Three different scenarios for high, mid, and low quality modes of NSS are provided. For demonstration purposes these work on a fixed size input, performing upsampling from 540p to 1080p.

1. Clone the Repository

These instructions will work on both Windows and Linux. On Windows, we strongly recommend the use of PowerShell over cmd.exe ("Command Prompt").

1. Ensure that Git LFS is installed. See the project's website for more details.

2. Clone the NSS model repository from Hugging Face:

git clone https://huggingface.co/Arm/neural-super-sampling
cd neural-super-sampling
git lfs pull

2. Install the Scenario Runner and Emulation Layers

1. Follow the steps in the Python documentation's "Create a new virtual environment" guide. We will assume below that you installed this environment into a .venv subdirectory.

2. Follow the steps in the Python documentation's ["Activate a virtual environment"](https://packaging.py