Let's do the Space Warp again!

Hey friends, I'm sorry if lately things have been a bit quiet here on the release front ;-)

As you know, I've been working on a few new titles. One of them, which might or might not have swinging from skyscrapers as its awesome map traversal mechanic, gave me the insane urge to return to the drawing table and see whether Alternate Eye Rendering could be made even better. If you already know all about AER and/or want to cut to the chase, please skip to the "What's cooking" section below; otherwise keep reading for some background information that you might find interesting.

My current AER implementation

Just a bit of context for new Patrons and people who just played my mods without giving too much thought to the technical aspects. I mostly specialize on making modern, open-world titles playable in VR from start to finish, with full immersion in the 3D virtual environment. That means that on your typical 90 Hz headset, since both eyes need to be drawn at each headset refresh, the game engine will need to pump out 180 fps. That is already beyond the realm of possibility for many current titles unless you have a beast of a PC, and yet it gets even worse, because in VR you need much higher resolutions, due to the greater field of view and the fact that the pixel structure is hugely magnified by the lenses in front of your eyes.

To quantify, games begin to look good in VR starting from 2160x2160 pixels (I typically employ square-ish aspect ratios, because for most headsets the horizontal and vertical FOV are comparable, and rendering a 16:9 image would just mean wasting previous GPU cycles on drawing invisible bars to the left and right of your field of view). And to get a nice, sharp image, you'll probably want to go up to 2736x2736 or even higher. The total number of pixels in a 2736x2736 image is comparable to that of a 4K image, about 8 megapixels.

So, to achieve "native" VR quality, you would need to make the game render at 180 fps with a resolution comparable to 4K. Most modern open-world titles will never reach that kind of performance on current consumer hardware (although the RTX 4090 coupled with a last-gen CPU might change that, but for now only a small minority of my Patrons has access to that kind of GPU power).

For the above reasons, I employ Alternate Eye Rendering in all my mods. At every game engine tick, only one eye is actually rendered, while the other is reprojected from its last image. The next frame will draw the other eye (that's where the "alternate" comes from) and reproject the previous image, so for example: frame 1 = render L; frame 2 = render R and reproject L from 1; frame 3 = again render L and reproject R from 2; and so on.

Such a stratagem halves the performance requirements imposed on the game engine, bringing it down to a more reasonable (but still quite hard to meet with older cards and newer games) 90 fps at 4K-ish resolution. The rest I do with some additional tricks, which allow you to play smoothly in VR even if the game cannot hit 90 fps reliably.

Suggested frame rates for AER

Speaking of not hitting 90 fps, many of you have been wondering about the "ideal" frame rates that they need to target to play in VR with my mods. That's somewhat subjective, so there is no clear-cut answer, but assuming that your headset refreshes at 90 Hz, I can give you a few figures that reflect my personal preferences. Note that the default refresh rate of the Quest 2 is 72 Hz, but at least for my mods it is strongly recommended to set it at 90 Hz.

Please keep in mind that these numbers are meant to be compared to the internal fps counter that the mod provides: if you use third-party utilities that measure the VR frame rate, they might not understand the mechanism I use and wrongly show you that you're running at a constant 90 fps, while in fact you're not. So always use the mod overlay to properly measure frame rate.

If you have really powerful hardware, and/or the game engine is old enough that you can reach at least 140 fps, maybe with occasional dips down to 120 fps, then go ahead and choose "R.E.A.L." from the "Render Mode" menu in the overlay, and enjoy an experience which is basically identical to that of a native VR title.

Otherwise (Render Mode setting to its default, i.e., "Stereo (AER)") the fps ranges are as follows:

• solid 90 fps: great subjective quality, with barely perceptible ghosting that can be made to look almost like a traditional motion blur if your headset can be set to 120 Hz refresh without making your system unstable (it's possible with the Valve Index for instance);

• at least 70-80 fps: very good subjective quality, with some ghosting particularly while strafing and/or when NPCs very close to you move from side to side. This is the "sweet spot" range that I usually recommend: it can be reached with most titles and most GPUs by dialing the in-game resolution up or down according to how much the game is taxing your system;

• at least 50-60 fps: acceptable subjective quality, with some judder and ghosting that becomes more visible and starts to get annoying for some people. If the game only occasionally dips to this frame rate range, that might be OK, but I would not recommend doing an entire playthrough in this conditions. It might be preferable to lower the resolution or the graphics options, and/or to use DLSS, until you get up to at least 70 fps;

• less than 50 fps, with dips under half the headset refresh frequency (so 45 Hz for a 90 Hz VR unit): that's a range you never want to be in. You will experience judder, jittering, bad tracking, plenty of ghosting, and/or an unrestrainable urge to go to Reddit/YouTube comment sections and publicly announce that my mods are crappy ^_^ While that might be true, you definitely don't want to play in those conditions. Run RealConfig again, don't push the game settings up (for example I get a lot of people trying and failing to play with ray tracing on: that will only cause you grief! Leave the settings as RealConfig presets them), and perhaps lower the resolution and/or use DLSS so that you get away from the 40-50 fps "danger zone".

What's cooking

While the image ghosting associated with Alternate Eye Rendering is definitely tolerable, and most people learn to "unsee" it after just a few minutes of gameplay (especially when following the common-sense recommendations that you can find in the previous "Suggested frame rates for AER" section of this post), it would be awesome to be able to get rid of the ghosting for good.

That's what I've been working on lately. I'm not 100% sure that the effort will succeed, and it's probably going to take at least another month of work before even a raw prototype can be released, but I'm beginning to feel confident enough that at least some improvement can be obtained, so I took some time to make a video that can show you more or less what I'm talking about.

Please note that the video is completely simulated. While I made it to resemble as closely as possible the final effect that you should get in your headset, the nature of the "ghosting" caused by AER is such that it cannot be captured on video. It's purely a subjective phenomenon, essentially an optical illusion that your brain creates when the image for one eye flashes twice on the low-persistence headset display. Our vision system evolved with natural images where motion is continuous and there is no stroboscopic effect, so when confronted with AER it wrongly decodes the double-strobed reprojected images as though the border of objects were fuzzy and duplicated.

On the left side of the video, I faked that effect by creating a low-frame-rate render where each frame is the blending of two successive images. Again, it's a simulation, but at least to me it looks close enough to the effect you get with AER, especially when your fps are a bit on the low side. On the right portion of the screen I made an offline recreation of what should be possible if I succeed with this crazy project, which I rather imaginatively called "AER v2" ;-)

You will see that it's by no means perfect: if you pause the video or slow it down a lot, especially when crazy stuff is going on rather close to the camera, the objects in the scene will look badly distorted. However (such is the magic of the human vision system), when played at normal speed, the distorted images will blend together in a way that makes everything look sharper and more natural!

What to expect

If I succeed, my mods will be essentially free from ghosting, as you can see from the simulated rendering in the video. This will automatically backport and apply to all of my Patreon conversions, not just future titles!

However, due to technical reasons, this will most likely only work with NVIDIA cards. There is a small chance that I could extend it to AMD GPUs, but it's not under my control and I would need some "insider" help (if any you knows anyone working in a technical capacity at AMD, or if any AMD developer liaison happens to be reading this, please get in touch with me!)

Also, the quality might not be the same that you can see from the video, at least initially. It might turn out to be even better in time, or fail miserably... It's not a solved problem in literature, so think of this as a research effort: it can be awesome, but no guarantees.

This was a long post, but a status update was kind of overdue :-) Happy Halloween!!!

Alternate Eye Rendering revisited (YouTube)