A Comprehensive Introduction and Overview of Gaussian Splats

AlphaPixel is always looking towards the next cutting edge of computer graphics and performance computing. In 2023, a new technique for creating realistic scenes from sparse photos debuted, called Gaussian Splats.

Welcome to our more than comprehensive guide to Gaussian Splats. If you’re reading this it’s probably because you have heard of Gaussian Splats and have a LOT of questions. What are Gaussian Splats? What are Gaussian Splats used for? How do you create Gaussian Splats? How do you view Gaussian Splats? What are Spherical Harmonics and how are they used by Gaussian Splats? What are the advantages and disadvantages of Gaussian Splats compared to polygonal photogrammetry and radiance fields like NeRFs? Who actually was Carl Friedrich Gauss and why do we care? Ok, you probably didn’t ask that last one.

We’re going to start at the beginning, and explain as thoroughly as we can, without too much heavy math or tech. Let’s decompose the words, “Gaussian Splat” and unpack what they mean together.

Splats have been a rendering technique for years, mostly in the realm of texturing. Texture splatting is a method of blending two different material appearances together based upon a third, blending mask texture.

By Simeon87 – Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=4371291

Ok, so what’s Gaussian got to do with it (with apologies to Tina Turner)?

Wikipedia again explains (with a lot of math) that it’s a function that has little value at the edges, and a rounded peak of value in the middle. In 2D it creates a nice round blended spot of value in the middle of the domain of influence.

By Kopak999 – Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=97678049

So, taken together, a Gaussian Splat is a thing that blends textures together using a Gaussian function to modulate where the texture is (and isn’t). Which is a pretty good description of the point-blob bits that make up a Gaussian Splats scene. A single 3D Gaussian is a spheroid-shaped blob (it can be stretched different amounts on its X, Y and Z axes) with texturing that is most opaque in the center and fades out towards the edges of the Gaussian. Interestingly, the texture is a view-dependent Spherical Harmonic, which means it can have a different appearance depending on which direction it’s observed from.

A 3d ellipsoid splat textured with monocolor Gaussian texture blend falloff.

By the author, Chris Hanson.

So, we’re done? You understand everything about Gaussian Splats now? Oh, yeah, there’s more. Like, how did we get here?

Gaussian splats are a new way of rendering what has been known previously as a Radiance Field. Same as it ever was. So, let’s talk about Radiance Fields first.

A radiance field is a representation of the way light interacts with a scene, describing how light radiates from every point in a 3D space. It captures the color and intensity of light as a function of position and direction, essentially detailing how light is emitted, reflected, or transmitted through different points in the scene.

In a radiance field, each point in space has associated light information that depends on the viewing direction. This allows for rendering realistic images by calculating the light that would be perceived by an observer from various viewpoints. The radiance field is used to generate photorealistic images by accurately simulating the lighting, shading, and colors of objects within a scene, considering how light sources and materials interact with each other.

You COULD create a radiance field synthetically, by tracing rays in a scene. But one of the most exciting aspects of Radiance Fields is creating them empirically, by feeding in real-world captured data. Analysis algorithms can infer the structure and content of the full radiance field from sparse slices of how the radiance was captured in the real world. The term Radiance Field doesn’t really specify anything about how the Radiance Field is created from sparse empirical data, nor does it mandate how you’d render the Radiance Field to create novel views that weren’t represented in the original data.

A naive representation of a radiance field might be a dense, fine, voxel structure where each voxel contains a lightprobe-like image showing what light emits from that spot. Of course this would not be an efficient representation.

A 1998 lightprobe image of all the light falling into one spot in Grace Cathedral, San Francisco.

By Paul Debevec ( https://www.pauldebevec.com/Probes/ )

Prior to NeRFs, radiance field rendering was performed by constructing multiple local radiance fields and then blending nearby radiance fields to synthesize novel views.

A 2019-vintage pre-NeRF Radiance Field render (LLFF, https://bmild.github.io/llff/)

From https://www.matthewtancik.com/nerf

Coming on the scene around 2020, Neural Radiance Fields (Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng) are an implementation of Radiance Fields using Deep Neural Networks to render. The process of generating the NeRF is the process of training the Neural Network. It can be fairly compute-intensive to train, and there is no “one right answer”. You can keep optimizing and training and looking for a better solution that produces a better error metric than what you already have, but at some point you need to call it good enough. Backpropagation is used to push improvements back into the network.

Wikipedia explains “The network predicts a volume density and view-dependent emitted radiance given the spatial location (x, y, z) and viewing direction in Euler angles (θ, Φ) of the camera. By sampling many points along camera rays, traditional volume rendering techniques can produce an image.” So, basically, you can input an XYZ location and orientation of a ray, and the network says “about this much of that color light will come to you along that ray”, much like a conventional non-neural reverse ray-tracer or ray-marcher.

However, rather than using conventional trigonometry to solve for the light output of a given ray, you must run a deep neural network. So, this technique can be quite slow, even with fast GPU hardware. A variety of techniques have been implemented since 2020 to optimize the problem and make the rendering process faster, and also to improve/speed the training process, but NeRFs are still fairly slow. They also, like other generative neural network techniques, can “hallucinate” data – if real data isn’t available for a given view, some fictitious (plausible or not) data may be inferred by the network. This can be both good and bad – it can help fill in for missing inputs, but it also can create artifacts and anomalies.

The secret sauce that turned Local Radiance Fields into NeRFs is the thought that the concept of a holistic unified radiance field is pretty hairy. Instead of trying to solve it by devising a method for merging multiple Local Light Fields, we can construct a differentiable model with an error metric. Then we can throw a lot of compute power at the problem to let machine learning stumble forward with some gradient descent guidance until it finds and acceptable single solution.

The 2020 NeRF paper Github code repository is here.

Ray Tracing a Neural Radiance Field of a Lego building block toy bulldozer.

From Representing Scenes as Neural Radiance Fields for View Synthesis

Photogrammetry is a related technique that predates NeRFs and Gaussian Splats. Like NeRFs and Gaussian Splats, it’s the process of taking multiple views from camera images, and reconstructing a scene representation. It does this by correlating camera locations and poses, identifying common visible features and reverse projecting pixel coordinates out into real-world space to see what spot a feature would need to be in to be visible in the pixel locations it was observed at.

The photogrammetry process was developed long ago but came into its own for map-making from aerial photos in the WW1 era. Many United States Geologic Survey contour maps were produced by well-dressed men standing at photogrammetry desks, adjusting the focus and alignment of two airphoto prints until only a thin area of elevation was aligned and focused, and then tracing the contour of that elevation around the map. See DEVELOPMENT OF PHOTOGRAMMETRY IN THE U. S. GEOLOGICAL SURVEY, page 7.

1932 model Wilson photoalidade, used to trace photogrammetry contour lines. USGS

In the modern, digital age, computer photogrammetry identifies common features scenewide, using a feature-description-vector method like SIFT or SURF, then compares features visible in every image to find common features. These common features are combined with prior knowledge of the camera+lens (often from the EXIF data in digital photos, combined with a camera+lens reference database) to compute where the camera was located when each photo was taken, and then where each identified feature was in front of the cameras. A “sparse point cloud” is created, consisting of the many thousands of locations in the scene that could be matched up in enough camera’s views to predict where that point was in the real world. Those points are also each assigned a single color (based on statistical analysis of all the colors each seemed to show in various input images). This process is now often referred to as Structure-from-Motion (SfM), as it is often done on frames from video from moving cameras, and it infers the structure of the scene geometry from the multiple-viewpoint side-effects of the camera in motion.

Once a sparse point cloud is created, a “densification” process is performed that tries to infer additional matching colored points between the sparse points. A densified point cloud can be output at this stage, and further work can be performed to group points in a common plane together into facets that are then output as triangular polygons, each with a small patch of raster texturing covering the polygon. This is called triangulation and texturing.

These polygons are generally opaque and hard-edged and are only created where the algorithms have a high probability of believing a piece of solid matter existed in the real world at that location. If polygons are created at less-high-probability locations, the scene tends to fill up with jagged, spiky, unwanted junk polygons that get in the way and look ugly. The downside is that sparse geometry, like organic structures such as vegetation or hair, and synthetic surfaces like metal and glass, don’t survive well through the photogrammetry and triangulation process.

Polygonal Photogrammetry Output. Note issues with thin or transparent objects.

By the author, Chris Hanson

A dense point cloud of a forest, viewed from above.

By the author, Chris Hanson.

Photogrammetry CAN produce VERY good results under the right, limited conditions. There are a great number of 3D assets in the geospatial world (like 3d buildings, and 3d models in game toolkit asset stores) that are carefully produced with photogrammetry and have excellent quality.

As Lightning McQueen is fond of saying, Speed.

NeRFs are rendered by executing a deep neural network. While this process IS accelerated by GPUs, it’s still pretty slow. Though optimizations were being invented, in 2023 some guys at INRIA (National Institute for Research in Digital Science and Technology) in France came up with a novel approach (the INRIA SIGGRAPH 2023 paper “3D Gaussian Splatting for Real-Time Radiance Field Rendering”) for computing and storing a Radiance Field without the whole deep neural network. Their idea was to use Gradient Descent optimization, just like is used in training a Neural Network, but instead they would use a common and fast method (rasterized Gaussian Splats) of putting the radiance into pixels.

To perform the view-dependent appearance trick, they would texture each of these splats with a Spherical Harmonic representation of the radiance (co