DL(S)SS

This post contains detailed information about stereoscopic DLSS, or as I like to call it, DL(S)SS.

For about 10 years now, basically all games have been relying on some form of temporal antialiasing, both to resolve jagged edges and to smooth over ugly artifacts left in the image by optimization techniques like checkerboard rendering. Temporal antialiasing was first implemented in a vendor-agnostic way, generically called TAA, but since then all video card makers have come up with their own custom implementations.

Intel has XeSS; AMD has FSR. However, the highest quality by far in the PC space is provided by NVIDIA's Deep Learning Super Sampling, or DLSS.

(Granted, maybe now NVIDIA has actually gone too far with DLSS 5, but that's a discussion for another day!)

Unfortunately, neither TAA nor its specialized forms, like DLSS, work correctly in VR without engine modifications, because they rely on precise motion vectors to match the pixels rendered in the current frame to the ones accumulated in previous frames. And the motion vectors generated by the engine do not account for the sharp motion imparted to the camera when it is displaced horizontally, so that the world can be rendered alternately from the left-eye and right-eye positions.

The errors are particularly evident around the borders of nearby objects or NPCs, because the view from the two eyes can either reveal or hide different background areas, which makes TAA matching fail and results in visible shimmering halos.

Around the end of 2024, I had an idea for an algorithm that could perhaps reconstruct the correct stereoscopic motion vectors (or at least a decent approximation of them) for any game, with no need to reverse-engineer the byzantine ways in which the vectors are computed by each different engine, but instead using only the information that the engine already provides to NVIDIA's DLSS libraries.

I patiently set down to work on it, which took me way longer than I had anticipated. Many failed attempts, more sleepless nights than I care to admit, a number of apparently promising intuitions that turned out to be completely wrong. And a plethora of details to take care of, dozens of unexpected corner cases to smooth out.

But in the end the result was actually better than I had dared hope for. I thought there would be some incremental progress in image quality, that could maybe make the defects a little less annoying; as a matter of fact instead, sometimes the upscaling is almost as good as native, and I was genuinely surprised at how much missing information it's actually possible to reconstruct without coaxing the game engine into doing any extra work.

Let's go into some details of what this DLSS fix (which I like to call DL(S)SS, short for Deep Learning Stereoscopic Super Sampling) can and cannot do.

Starting with the really bad stuff: just like AER v2, this is NVIDIA only. If you have an AMD card, you won't be able to take advantage of this technique with my framework. I apologize to Team Red, but I'm only one person. There are some third-party mods that make it possible to emulate DLSS on AMD cards using FSR, but it is completely uncharted territory and you will have to experiment on your own. Also, though I have no affiliation whatsoever with NVIDIA and receive no money from them, I think that their cards are simply the best choice for VR. And basically everyone on the web seems to agree that DLSS is far superior to FSR in terms of image quality.

Next, the still-bad-but-not-quite-as-much stuff: this only works with DLSS. Games that only support TAA but not DLSS will not see any improvements. That is because, unlike DLSS, ordinary in-engine TAA doesn't have any standardized way of conveying the motion information, so it's not possible for me to make a fix that will work across different engines.

Moving on to the so-so news. The technology is based on a series of approximations, heuristics and a real-time reconstruction of missing data, so it is nowhere perfect. In many cases it works amazing well. But there are other situations where it fails. In particular, distant moving objects will tend to show longer trails than they do in 2D. There will be some shimmering left, especially on sharp contours. Some games may also give less satisfactory results, like for instance titles based on the UE4 engine.

Also, having DLSS work (almost) correctly doesn't mean that we no longer need to use AER. The requirement for the game to render two stereoscopic images instead of just one remains. So the motion artifacts of AER v2 (or the judder of legacy AER) will still be there, completely unchanged.

But there's good news too! First of all, this tech is essentially free, in the sense that it has no measurable performance hit compared to normal (non stereoscopic) DLSS. And even though there might be moments in the game where it doesn't work perfectly, it is always a big improvement in quality versus the old unfixed TAA or DLSS, especially on large nearby objects like NPC faces and hair.

Moreover, the fix is more or less "universal", in the sense that it is possible for me to implement it on new games with just a reasonable amount of customization. Talking days instead of weeks. Already in December 2024, the tech was made immediately available for all the titles in our list that offer support for DLSS (including Elden Ring, which has no native support but can do DLSS via a third-party mod). Of course you also get a checkbox to disable it in case you find any portions of the game on which it doesn't work satisfactorily.

And finally, the super-good news: the quality improvement that you can squeeze out with DL(S)SS is so vast that you can now actually use DLSS not just on the Quality setting, but also on Balanced and all the way down to Performance, despite the huge magnification that VR applies to everything. Together with the FOV optimizations, this means that for the very first time it's becoming possible to play a game in VR with essentially the same image quality that you would get in 2D.

I think the importance of this cannot be overstated. Now maybe the studios and publishers will finally accept that they can offer official VR ports of their games without huge investments, while at the same time keeping the graphics quality at AAA levels.