Léna Piquet
Refracting Pixels
Taking at look at how games and engines render refractive surfaces
Taking at look at how games and engines render refractive surfaces
December 03, 2024
What started this article is that I wanted to support refraction in my realtime game engine (which you can learn more about here). I didn't find a lot of clear documentation on the subject, so I went to look into a small list of games to see how they were handling it.
Additionally I had questions on when refraction should done compared to regular transparency rendering, which usually involves rendering objects in a specific order (to avoid sorting issues).
I also discovered a few tricks while doing all of this, so hopefully there is some interesting knowledge here even if the subject is already familiar.
Disclaimer: in this article I'm going to use some keywords in a recurring manner, but I'm not sure if they really are commonly used. So apologies in advance if my terminology is confusing.
When I'm talking about refraction, it is about the physical phenomenon of light passing from a medium into another one.
Quoting wikipedia:
Refraction of light is the most commonly observed phenomenon, but other waves such as sound waves and water waves also experience refraction. How much a wave is refracted is determined by the change in wave speed and the initial direction of wave propagation relative to the direction of change in speed.
Source
The refractive index is the ratio of the apparent speed of light in the air or vacuum to the speed in the medium. The refractive index determines how much the path of light is bent, or refracted, when entering a material.
Source
One obvious example of this is between air and water:
(A glass of water with a pencil showing the refraction effect.)
Another example where we can see refraction in real life is via mirages:
A mirage is a naturally-occurring optical phenomenon in which light rays bend via refraction to produce a displaced image of distant objects or the sky.
Source
(Photo of a desert with mountains on the horizon being reflected down. Photo from Unplash)
In real-time rendering it is common to see refraction through various effects, such heat in the air, water, or frosted glass/window.
Here is an example:
(A refractive sphere made in Unreal Engine 4.)
Ideally, you would refract rays used to render a scene, like in real life with a camera and a lens. However that requires using raytracing (or pathtracing), which is still an expensive solution today. So instead we rely on other tricks.
The trick in question is to distort the UVs of the image we display on the screen. This can be done with a normal map, like with the sphere we saw just above:
(The normal map used on the sphere.)
An article from GPU Gems 2 implement this method by using a copy of the scene color buffer and then distort it by warping the UV coordinates when sampling the image.
Note: there is one fundamental example that I won't cover in this article and it is actually water.
The main reason is that water is often a separate system in a game engine, just like landscapes, and therefore they have their own rendering system.
Here my focus has been on simpler objects instead (FX, glass, small props, etc.) since this is a less discussed topic.
To more easily answer my questions I did frame captures of several games in RenderDoc to see how they manage refraction. Initially this was to get a rough idea of how the technique would work, but I discovered quite a few nuances that are worth sharing.
Note that I focused on "old" games because I wanted to see how it was done in forward renderers since it would better match my own engine.
From these frame captures I noticed there were 3 big schools:
Fundamentally, the first method suits best a forward renderer (since we render final objects one by one and only once) while the second one is more aligned with the logic of a deferred renderer (where object properties are in separate buffers and then composited at the end).
This method has the benefit of accumulating refraction and/or other transparent objects in the scene since a new copy is done each time. So a dust cloud or a water leak will appear behind a frosted glass. This is obviously done at the cost of doing a copy and switching between different framebuffers.
Half-Life 2 refraction is quite straightforward to understand:
The important point to note here is that the steps above are repeated before each draw that will be refracting pixels. The liquid filled tubes in Dr Kleiner's lab are good example of this.
The inner tube is rendered first (with a copy just before as explained), then the transparent outer layer that has specular reflections from the nearby cubemap is added just afterward. This is for one tube, so we repeat for the second.
(Step by step of how the tubes are rendered.)
And here is what the tubes look like in movement:
(The tubes filled with liquid in the lab.)
There is no advanced filtering, which means that opaque objects in the foreground can bleed into the refracted pixels since they have already been drawn. Here is an example with a speech bubble:
(The rotating speech bubble bleeds into the glass behind it.)
Another example is with particles that simulate heat/gas distortion:
(He is taking a nap, don't mind him.)
In this case the foreground with the fence doesn't seem to appear on the gas particle because the blur is strong enough to hide it. The depth test mask out pixels from the fences, making the particles appear behind it.
(Pixels contributing to the final image.)
F.E.A.R. uses the same process as Half-Life 2. Early in the game you can find a fire inside a barrel with very aggressive distortion going on:
(This is realtime, I didn't accelerate the angry fire.)
You can notice how the pistol sometimes get duplicated, FEAR refraction suffers from the same issue as in Half-Life 2. Note also that in this case the distortion is applied via a quad facing the player's camera (which has a panning normal map over it):
(Before vs after the distortion is applied.)
The fire is rendered first then followed by the distortion, to ensure the heat also applies to the fire.
F.E.A.R. Project Origin (the following game) also uses the same principle. For example the following double door features frosted glass:
(Double door with frosted glass.)
The left and right side are rendered as separate objects, so each side triggers a copy of the current scene color buffer before it gets used to render the refraction.
Fun fact: the fog is rendered as a screen space pass on the whole screen but after the refracted glass on the doors have been rendered. I found that a bit curious.
Crysis 3 also works in a similar manner to Half-Life 2 but with a nuance.
(A gas leak particle combining regular transparent textures with distortion.)
In Crysis 3 the current scene color is copied with a scissor rect enabled at the same time to limit the area drawn. The main goal is to save performance since the game can use Multi-Sample Anti-Aliasing (MSAA). It makes sense to try to skip copying (and resolving) as many pixels as possible, especially at high resolution.
The scissor bounds have some padding compared to actual area drawn by the particle, likely to avoid issues if the sampling goes a lot beyond the initial pixel position.
(The copy of the scene color in Crysis 3.)
Notice the area on the right is black: this is just because I used the "clear before draw" option in RenderDoc. It's just here to show where the scissor stopped. In practice the buffer reference the pixels from the previous frame.
(The drawing of the particle system, illustrated with its wireframe.)
(The actual pixels drawn by the particle, partially cut off by the depth test.)
Like in Half-Life 2, this operation is done for each object that applies refraction on the image. This is also mixed in the middle of the rendering of other transparent objects. With everything sorted from back to front.
With this method, particles and other effect render additively into the same buffer (as vectors, similar to a normal map). Then this buffer is used during a single pass to distort the UV coordinates of the image on screen.
The benefit is that the distortion can be rendered/accumulated as a separate pass, it doesn't hijack the regular rendering. The main drawback however is that you lose layering, meaning that the refraction is always in front of transparent objects for example (or behind, depending when it happens). It cannot be interleaved.
This method usually provides good enough results for general distortion effects.
I wanted to look into Clive Barker's Jericho because it is kind of an old game now, which I knew was running on its own engine. While the main rendering is done in a forward manner, the refraction on screen is done via an intermediate buffer.
Distortion is accumulated into a BGRA8_UNORM texture, in this case via specific particle systems currently in view. Those particles are rendered after transparent objects. So the fire and smoke particle a drawn first before switching to the accumulation buffer.
Then the scene color is distorted by the buffer and the result stored into a new texture.
The buffer has a black color as its default value. Particles write vectors in the RGB channel for the direction, while the Alpha channel stores the opacity, likely used to scale the distortion intensity.
The game anti-aliasing is classic MSAA, so both the scene color buffer and the accumulation buffer use multiple samples and get resolved at the end of their rendering pass.
Here is an example:
(The distortion in movement.)
(The fire particle before and after distortion.)
(The accumulation/distortion buffer, heavily contrasted here to better view it.)
Batman Arkham Knight is a very interesting game to look at. First of all because it is build on an heavily modified Unreal Engine 3 version, so that made me curious to see how it would behave differently from UE3.
In Batman refraction is also done via an accumulation buffer that is BGRA8_UNORM, like in Jericho. The difference here however is about when the refraction is applied on screen. In this case it happens before rendering transparent objects.
For example inside the GCPD building there are is a room where you can align multiple windows and a frosted glass door:
(The final result once the transparent windows are rendered.)
(Before and after the accumulation buffer is applied.)
(The content of the accumulation buffer, RGB on the left, alpha on the right.)
It's not easy to see, but the refractive windows are actually all here, not just the door:
(The wireframe of a window further away behind the door.)
The rendering order doesn't matter, since the drawing is additive. In this case for example the frosted door refraction is the first element drawn and the background windows are drawn afterward.
The transparent meshes, in particular those used to get the specular reflections and tinting of the windows, are drawn in the next pass. This means that only the opaque geometry is distorted by the buffer.
If the specular reflections need to be distorted as well, this can be done directly inside the shader that draw the meshes via a normal map to change the reflection vector.
Note: Batman doesn't seem to do any kind of filtering to avoid bleeding as well. In some cases this can lead to pretty obvious artifacts, like on this other door:
By curiosity I also looked at the Unreal Engine 3, via the Unreal Development Toolkit (UDK) to see if it was working in a similar manner. Turns out it is the case as well.
Here is an example with an explosion effect from the Unreal Tournament game/demo:
(Before and after the distortion is applied.)
(The content of the accumulation buffer.)
(The final result with transparent particles drawn afterward.)
Dishonored 2 runs on a custom engine, the Void Engine (which is heavily based on id Tech 5).
In this game they also use the concept of an accumulation buffer, but instead of doing it into a dedicated texture the distortion vectors are added into the velocity buffer used for the motion blur. That buffer uses the RGBA16_FLOAT format and compared to other game it is also at half resolution.
Since motion blur is simply a directional blur along a given vector (by accumulation of multiple samples per pixel) it leads to very smooth distortions, like so:
(A particle system showing heat over an oven.)
(An upward effect from a magic power.)
And here is what the velocity buffer looks like with both the power effect (on the left) and the oven heat (on the right):
(Velocity buffer from Dishonored 2, with the magic power and the oven.)
Quantum Break heavily features refraction and distortions to showcase the time travel effects. It also uses a dedicated buffer to accumulate vectors. That buffer uses the RG16_FLOAT format.
(Before refraction is drawn, only particle system sampling the scene are present.)
(After refraction is rendered based on the accumulation buffer.)
(The accumulation buffer.)
Note: Quantum Break also uses tesselation and vertex deformation on the scene meshes to simulate wave and ripples across the world, this is not handled only via screen space refraction. The end result is achieved by combining a lot of effects. In the example above this can be seen on the door and wall being weirdly deformed.
I highly recommend checking out the article on FxGuide about it.
Something else that is interesting to note is how refraction near the border of the screen is managed. Instead of clamping the UV coordinates (as it often happens in other games) they mirror them instead. The game relies heavily of broken glass and mirror reflections, so that fits it really well:
(Example of refraction with mirrored UVs at the border of the screen.)
Note: another possible fallback instead of mirroring UVs is to go sample the nearby reflection cubemap. This way you can go read beyond the pixel on screen. It might be tricky to blend between the two without obvious seams/artifacts.
So far we saw both ways of doing refractions, but why don't we try a middle ground ?
Why not doing both methods at the same time ?
In Unreal Engine 3 (as explained above) things were straightforward, but with Unreal Engine 5 it's another story. UE4 and now UE5 are mainly deferred renderers with a forward rendering pass for transparency.
Refraction is possible in two different ways:
Here is an example from the game Injustice, which was using UE3, but showcases (I believe) the use of the scene color node in materials to do cool effects:
(Excerpt from the reel by William Kladis)
Unreal also allows to render transparent objects into a separate buffer (called Separate Translucency) and that buffer gets composited after refraction is rendered/applied. Therefore it gives more control on when the refraction should happen in the rendering pipeline.
Then all of these buffers get mingled into the Temporal Super Resolution (TSR, aka the upscale and temporal anti-aliasing) pass in some magic way.
Here is an example in practice, with a refractive sphere:
(The accumulation buffer.)
(The scene color being refracted by the accumulation buffer.)
(The refracted buffer getting merged back into the scene color.)
(The final result with the object from the Separate Translucency pass composited over.)
Not sure why this refraction requires these 3 passes however (I didn't had the energy to dive into the source code to get an answer).
Doom features two ways of doing refraction. Because the engine is a (clustered) forward render, it can use layered refraction (like in Half-Life 2). This is done for the frosted glass which also relies on blurring the background element instead of doing a simple copy:
(Step by step on how to do a frosted glass in Doom.)
Note: the blur uses a gaussian kernel to approximate a GGX distribution. More details in the great Siggraph presentation.
The video just above shows the steps done as follow:
Then the glass mesh is rendered with the downscaled (and blurred) buffers from 1 to 5 as input.
(Before and after the glass refraction is drawn.)
You can notice how the shotgun extremity in the foreground is slightly bleeding into the glass.
For more basic effects, Doom also uses an accumulation buffer:
(The heat from the lava creating distortions in the air.)
That buffer is in the format RGBA8_UNORM, which is used during the tonemap pass (which combine bloom and applies color grading) to distort the UVs in screen space. This means that this distortion happen at the end after everything has been drawn on screen.
(Before and after the refraction is applied.)
(The accumulation buffer.)
I mentioned it a few times, but there is an important point to note in all of this. Depending on when the refraction is rendered, you may or may not have an anti-aliased image to distort.
If the distortion happens late in the pipeline (like in the tonemap pass in DOOM) then you can get an anti-aliased image to distort (whatever the method employed). The constraint is that refraction sits on top of everything.
If the distortion happens in the middle of the rendering process however, then you either have to use MSAA in some ways with the accumulation buffer or just live with the aliasing. TAA could also get rid of this aliasing afterward, but given the scene color isn't quite matching the geometry anymore (depth buffer) then you need to be sure to output proper motions vectors and you may not fix the aliasing or end-up dealing with incorrect blur.
So what am I going to use in my engine ?
It will be the Half-Life 2/Crysis 3 method, that seems to be matching the best how my engine already work. I even got a quick hack working which I used for the header image:
(A glass panel in front of a character showing a frosted glass with a pyramid pattern.)
