Motion Blur for Half-Life Video Recording with Vulkan

Video of a Half-Life movement map run, recorded with motion blur.

Half-Life is an award-winning first-person shooter from 1998. One of the many things setting it apart is a fluid player movement system. You can gradually build up speed in the air by turning left and right, and if you jump as soon as you hit the ground, you keep all of that speed.

These techniques, called strafing and bunny-hopping, unlocked a whole new dimension to the game, and spawned a big community of mappers and players. There are hundreds of custom maps with challenging obstacles designed to test one’s movement skills, and thousands of players competing to finish them as fast as possible.

The main hubs for these maps and records are Xtreme-Jumps1 and Cosy-Climbing. They are centered around Counter-Strike 1.6, which has similar movement mechanics to Half-Life, but paced slower with a jump-speed limit and some tweaks to the acceleration.

People generally record their runs to in-game demo files. They are very lightweight, making them easy to store and share. Players can then capture demos to videos. There are YouTube channels uploading videos of the best, or notable runs and edits, like the Xtreme-Jumps channel.

Since this gameplay style is focused on movement, it has been customary to use motion blur for video recordings. When done right, it can make the video look smoother and nicer to watch; here’s such an edit (volume warning) for example. Though, ultimately, motion blur is a subjective preference, and plenty of people will tell you how it is completely unwatchable. Part of this, I suspect, comes from frequent cases of motion blur done wrong. I shudder at the vision of someone dropping a 60 frames-per-second (FPS) video into a 30 FPS Vegas project, and leaving frame blending on its default enabled setting.

The simplest way to get motion blur is to capture the video at a higher frame rate, then blend together multiple frames for every output video frame. And when dealing with a closed-source game engine, there actually isn’t much else you can do. Thankfully, Half-Life demos play back smoothly at any frame rate, regardless of the FPS the player had when running2, so this approach works perfectly.

Let’s explore how several frames combine into the final frame. The following interactive widget shows the final frame at the top, and a set of sixty consecutive sub-frames, or samples, below. Green outlines show which of the samples are blended together to form the final frame. Even though motion blur is best experienced in motion, this still-frame demo should make it clear how it works under the hood.

Loading...

Your browser does not seem to support HTML canvas.

Samples per Second: 1800

Exposure: 0.5

There are two sliders corresponding to two of the most important parameters for controlling the motion blur.

The top one controls the video capturing FPS, usually called “samples per second”, or “SPS” for short, in the context of frame blending. A low SPS value will produce clearly visible copies of the same objects, making it easy to tell apart the individual frames in the final composite. A high SPS value on the other hand will make the composite smooth and nice to look at in the final video.

The bottom slider controls the amount of blur. It is called “exposure” because it is analogous to exposure in photography and filming. Higher exposure means more sub-frames are blended together. Compare low exposure giving a mostly-clear still frame and high exposure resulting in a lot of blur as all samples are included.

Real-world exposure lets the camera sensor take in light continuously while the shutter is open, but game video capturing is discrete and can only use the samples that were recorded. This is why low SPS makes individual sub-frames visible—they are the only samples we have, with no in-between contents available.

So, high SPS is extremely important for good motion blur. When the player is moving or spinning fast, you need a lot of samples to blend together a smooth-looking frame without obvious edges. This means recording the video at a very high FPS. For Half-Life movement-oriented maps, you want to go up to at least around 3600 FPS to get a decent result regardless of what’s happening on the screen. If you can only capture the video at 60 real-time FPS, it will take you sixty times longer to capture the demo at 3600 FPS compared to its real-time duration. That is, an hour of recording per every minute of the demo. Basically, you really want your recording to go as fast as your computer allows, without being limited to real-time.

This problem was one of the main drivers for me to explore and make my own Half-Life video recording tools, first hl-capture, then bxt-rs. Frame blending is included in the widely-used HLAE tool, which has been around for many years. Unfortunately, HLAE had extremely slow recording speed back then (it is considerably better nowadays), making it unreasonable to use high FPS for motion blur. This is why so many of the older movement videos have clearly visible frame blending artifacts.

In the previous post about bxt-rs video recording, I wrote about how these kinds of tools can capture video much faster than real-time. Now let’s see how I added motion blur to the process.

Frame blending is really just per-pixel averaging of multiple frames. If you know anything about how GPUs work, you can immediately see that this is the perfect task to do on a GPU: a ton of mutually-independent computations that can be done in parallel. Better still, the input frames are already on the GPU: the game has just rendered them. This means that we can avoid the costly data transfers between the GPU video memory and the main memory, making the whole process practically instantaneous. In fact, the interactive widget above is basically doing the same computation in real-time, as you’re dragging the sliders.

In bxt-rs I copy the frame rendered by the game into a texture shared between OpenGL and Vulkan. The Vulkan part then converts the pixel values from the RGB format to YUV 4:2:0 used in video files, downloads the result into the main memory, and hands it over to FFmpeg for video encoding.

The frame blending step will live in Vulkan, between receiving a new frame from OpenGL and pixel format conversion. We only need to output and encode the final blended frame, so intermediate sub-frames can stay in RGB.

We don’t actually need all sixty or so sub-frames at once in video memory. We can allocate one buffer for the output frame (the sampling buffer), then accumulate the intermediate frames into it one by one as they come in. Averaging pixel values means summing them up and dividing by the total count, which is equivalent to dividing every value by the total count first, and summing them together afterwards. Since we know the total count in advance (it’s SPS multiplied by exposure), we can do this division and addition as the frames come in, and get the correct averaged result in the end.

I chose the 16-bit RGB format for the sampling buffer. This allows up to 257 sub-frames with no precision loss due to quantization3, which is more than enough. By default, bxt-rs records at 7200 FPS, which means up to 120 sub-frames with an output FPS of 60, or up to 240 sub-frames with an output FPS of 30—both below 257.

let create_info = vk::ImageCreateInfo { // Use the 16-bit RGB format in UNORM mode. format: vk::Format::R16G16B16A16_UNORM, usage: // For updating during the sampling stage. | vk::ImageUsageFlags::STORAGE // For clearing. | vk::ImageUsageFlags::TRANSFER_DST // For reading during YUV conversion. | vk::ImageUsageFlags::SAMPLED, // ... }; let image_sample = device.create_image(&create_info, None)?;

UNORM lets me treat pixel values as floating point numbers from 0 to 1 in the shader, without having to round them manually.

The shader itself is pretty simple. Its inputs are the new sub-frame from the game, the sampling buffer and the weight factor. The code loads a pixel from the new sub-frame, multiplies it by the weight, and adds it into the sampling buffer.

#version 450

// Copied from the color conversion shader where it gave the best performance. #define WORKGROUP_SIZE 4 layout (local_size_x = WORKGROUP_SIZE, local_size_y = WORKGROUP_SIZE, local_size_z = 1) in;

// Frames coming from the game are rgba8. layout (binding = 0, rgba8) uniform readonly image2D image_frame; // Our intermediate buffer is rgba16. layout (binding = 1, rgba16) uniform image2D image_sample;

// Weight is set for every frame at runtime. layout (push_constant) uniform push_constants { float weight; };

void main() { ivec2 size = imageSize(image_frame); uint width = uint(size.x), height = uint(size.y);

uint x = gl_GlobalInvocationID.x, y = gl_GlobalInvocationID.y; if (x >= width || y >= height) return;

ivec2 coords = ivec2(x, y);

vec4 frameColor = imageLoad(image_frame, coords); vec4 sampleColor = imageLoad(image_sample, coords);

// Accumulate the frame into the intermediate buffer. vec4 newColor = sampleColor + frameColor * weight; imageStore(image_sample, coords, newColor); }

Using multiplication instead of division is just a common thing to do. A usual justification is that it’s faster this way. I’m not sure how much this is still true, especially for shader code4, but anyhow.

I can use image2D rather than sampler2D for the input image, since in this shader I read every pixel independently of the rest. This lets me avoid making the whole Vulkan combined image sampler required for a sampler2D.

The weight can vary between frames, so I pass it as a Vulkan push constant. This is because exposure can cut a sub-frame in half, causing its weight to be reduced. That might seem like an edge case which can be solved by rounding the exposure to always cover full sub-frames. You can indeed do that for capturing demos. However, bxt-rs also supports capturing TAS playback where the game’s FPS cannot be changed, so the frame blending code must work with any timing thrown at it.

As you may find in the previous post, the video recording loop without frame blending looks somewhat like this:

Run game tick

Run game tickDraw frame

Draw frame Capture frame

Capture frame Swap buffers

Swap buffers Copy to intermediate

Copy to...Convert colors

Convert colors Transfer to CPU

Transfer to CPU Send to FFmpeg

Send to FFmpeg Second Thread

Second Thread

Every frame is copied to an intermediate buffer, because we don’t know its duration until the next frame. Then, bxt-rs converts its colors from RGB to YUV 4:2:0 on the GPU, transfers the result to the main memory, and sends it to FFmpeg for encoding.

With frame blending, the steps are a bit different:

Run game tick

Run game tickDraw frame

Draw frame Capture sub-frame

Capture sub-frame Swap buffers

Swap buffers Copy to intermediate

Copy to intermediate Convert colors

Convert colors Transfer to CPU

Transfer to CPU Send to FFmpeg

Send to FFmpeg Second Thread

Second ThreadYes

YesIs frame complete?

Is frame complete? Accumulate

Accumulate

After obtaining the frame duration, bxt-rs computes the weight and accumulates the sub-frame into the sampling buffer. For most frames this is where the process ends. The recorder keeps track of accumulated frame time. As soon as we’ve accumulated a full output frame worth of sub-frames, it’s sent through the usual steps of color conversion and FFmpeg encoding. The sampling buffer is then cleared to zeroes, ready to accumulate for the next video frame.

There are a few edge cases to take care of. First, as I mentioned before, the sub-frame can be covered