Capcom RE:Dox, open source serialization/deserialization for game engines

1. What is RE:Dox?

RE:Dox is a high-performance, token-based structured data engine for .NET, developed as one of the core components of 'REX' Technology for CAPCOM's next-generation game engine.

RE:Dox is not just a JSON serializer. It parses structured data into a compact, fixed-size token DOM / IR, then uses that structural index for reading, editing, format conversion, serialization, deserialization, and automatic parallel deserialization.

JSON / JSON5 / CBOR / MessagePack / TOML / XML / HTML / CSV / INI / DOX ↓ Compact token DOM / IR ↓ Reader / Writer / Serializer / Deserializer ↓ .NET objects, JSON, CBOR, MessagePack, TOML, XML, HTML, DOX, ...

The same token representation acts as both a compact parsed document model and a common structural layer shared by serializers, converters, and supported formats.

RE:Dox combines three goals that usually conflict:

dotnet add package CAPCOM.REDox

Serialize, deserialize, parse, and edit through the same API surface:

using REDox.Json;

var player = new Player { Name = "Leon", Level = 42, Items = ["Handgun", "Green Herb"] };

// Serialize / deserialize var json = JsonSerializer.Serialize(player); var restored = JsonSerializer.Deserialize<Player>(json);

// Parse into a token DOM and edit it in place using var doc = JsonDocument.Parse(json);

var root = doc.RootElement.AsObject(); root["Name"] = "Claire"; // replace root.Add("Hp", 100); // add root.Remove("Level"); // remove

var items = root["Items"].AsArray(); items.Add("First Aid Spray"); // append items.Insert(0, "Knife"); // insert items.RemoveAt(1); // remove by index

var edited = doc.RootElement.ToJsonString();

public sealed class Player { public string? Name { get; set; } public int Level { get; set; } public string[] Items { get; set; } = []; }

4. Performance

Environment: BenchmarkDotNet, .NET 10 (X64 RyuJIT x86-64-v3), AMD Ryzen Threadripper PRO 5975WX, Windows 11.

These ratios describe the benchmark datasets shown below; they are not universal performance guarantees.

dotnet run -c Release --project benchmarks/REDox.Json.Benchmarks

Benchmark results depend on data shape, target type, runtime, CPU, and serializer options. Always benchmark with your own workload.

A 64-bit dual-mode token at the core

Every value is described by a fixed-size 64-bit token. An extension bit selects how the payload is interpreted:

extension bit = 0 → payload interpretation is delegated to the document/format layer (source-backed, format-specific payload; many formats, one token shape)

extension bit = 1 → payload interpretation is fixed by RE:Dox, and control tokens enable format-agnostic editing (insert / remove / replace)

This allows one token structure to represent both a source-backed view of format data and a RE:Dox-owned editable DOM. Where the document/format layer can retain the original payload, unchanged values can continue to reuse source slices instead of being eagerly materialized.

A token can represent null, boolean, integer, floating point, string, binary, timestamp, big number, array, map/object, trivia (comment/whitespace), or format-specific extension data. It may store inline data, a source offset/length, a container count, link information, or an extension id.

Reading and writing have different information requirements, so RE:Dox uses different paths for each direction:

Serialize : DataWriter, one pass, written directly to the output buffer (information is known → no look-ahead, no intermediate DOM)

Deserialize : DataReader over the token DOM, random access by token id (information is unknown → look-ahead, out-of-order, context lookup)

On the write path the value is already known, so RE:Dox writes straight to the buffer using bulk (WriteValues) and fused-property (WriteProperty*) helpers — no intermediate object tree is built.

On the read path the structure is unknown, so RE:Dox first builds a compact token DOM. Because the structure is then fully addressable, RE:Dox can:

Tokens can store offsets and lengths into the original source buffer. Strings, numbers, timestamps, and binary values are decoded only when requested. Elements are lightweight handles over a document and token id, while object and array views are materialized only when needed.

A tape DOM is normally read-only because its logical structure is represented by a flat token sequence. RE:Dox adds indirection at container views: DArray, DObject, and DMap keep value slots that can be re-linked, a free list reuses emptied token slots, and extension/control tokens carry edit state.

This preserves the cache-friendly characteristics of a sequential token representation while providing node-like operations such as insert, remove, and replace without rebuilding the entire document for each edit.

Converters (DataConverter<T>) are written against the format-agnostic DataReader / DataWriter abstractions rather than a specific format implementation.

DataConverter<T> │ DataReader / Writer / |
JSON CBOR MessagePack ...

This allows converter logic to be shared by formats that participate in the common reader/writer model. Instead of duplicating equivalent conversion logic for each type-format pair, RE:Dox can reuse one converter implementation across those formats — also providing a foundation for future AOT / source-generator support.

NuGet package IDs use the CAPCOM.REDox.* prefix. C# namespaces use REDox.*.

Preview packages use NuGet pre-release versions such as 0.1.0-preview.1; the package name itself does not include -preview. Preview components may change API behavior before stable release.

dotnet add package CAPCOM.REDox dotnet add package CAPCOM.REDox.Cbor dotnet add package CAPCOM.REDox.MessagePack dotnet add package CAPCOM.REDox.Dynamic dotnet add package CAPCOM.REDox.Serialization.DataContractJson dotnet add package CAPCOM.REDox.Ini

Install preview packages with --prerelease:

dotnet add package CAPCOM.REDox.Serialization.SystemTextJson --prerelease dotnet add package CAPCOM.REDox.Serialization.NewtonsoftJson --prerelease dotnet add package CAPCOM.REDox.Toml --prerelease dotnet add package CAPCOM.REDox.Xml --prerelease dotnet add package CAPCOM.REDox.Html --prerelease dotnet add package CAPCOM.REDox.Csv --prerelease

CBOR

using REDox.Cbor;

byte[] cbor = CborSerializer.Serialize(new Player { Name = "Jill", Level = 30, Items = ["Lockpick"] });

var player = CborSerializer.Deserialize<Player>(cbor);

// Parse into the token DOM and re-encode using var doc = CborDocument.Parse(cbor); byte[] encoded = CborDocument.Encode(doc.RootElement);

MessagePack

using REDox.MessagePack;

byte[] msgpack = MessagePackSerializer.Serialize(new Player { Name = "Chris", Level = 35, Items = ["Knife", "First Aid Spray"] });

var player = MessagePackSerializer.Deserialize<Player>(msgpack);

using var doc = MessagePackDocument.Parse(msgpack); byte[] encoded = MessagePackDocument.Encode(doc.RootElement);

Dynamic

using REDox; using REDox.Dynamic; using REDox.Json;

using var doc = JsonDocument.Parse("""{"name":"Leon","level":40,"stats":{"alive":true}}""");

var player = doc.RootElement.AsDynamic()!; var name = (string)player.name; var level = (int)player.level; var alive = (bool)player["stats"]["alive"];

// Build a new object dynamically var obj = new DObject().AsDynamic(); obj.Name = "Ada"; obj.Items = new[] { "Hookshot" };

string json = JsonSerializer.Serialize(obj); // {"Name":"Ada","Items":["Hookshot"]}

Cross-format conversion

Formats that share the common token IR can be converted through the document model:

using REDox.Cbor; using REDox.Json;

using var doc = JsonDocument.Parse(json); byte[] cbor = CborDocument.Encode(doc.RootElement);

JSON5 → token IR → JSON JSON → token IR → CBOR CBOR → token IR → MessagePack TOML → token IR → JSON XML → token IR → JSON-like structural representation

Formats with different native data models may use a structural projection rather than a lossless semantic round-trip.

Automatic parallel deserialization

The deserializer inspects each array's element count and token range before materializing objects. When an array is large enough, element token ids are collected and each element is deserialized into its corresponding index in parallel. Small arrays stay sequential to avoid parallel scheduling overhead.

using REDox; using REDox.Json; using REDox.Serialization;

var settings = new DoxSerializerSettings { ParallelOptions = new ParallelDeserializeOptions { ParallelDeserializeEnabled = true, MinimumNumberOfElements = 1024, MinimumNumberOfTokens = 4096 } };

var players = JsonSerializer.Deserialize<Player[]>(json, settings);

JSON5 with comment and trivia preservation

using REDox.Json;

var json5 = """ { // Player name name: 'Leon',

// Inventory items: [ 'Handgun', 'Green Herb', ], } """;

using var doc = Json5Document.Parse(json5, options: new Json5DocumentOptions { PreserveTrivia = true, EnableValueValidation = true });

var root = doc.RootElement.AsObject(); root["name"] = "Claire";

var output = Json5Document.EncodeToString(doc.RootElement, new Json5WriteOptions { PreserveTrivia = true });

Async streaming of NDJSON / JSON sequences

JsonSequence reads a stream of JSON values (NDJSON / JSON Lines, or a large top-level array) asynchronously, one value at a time, without loading the whole input into memory. For NDJSON, enable UseNewlineDelimitedFormat in JsonDocumentOptions.

using REDox.Json;

var options = new JsonDocumentOptions { UseNewlineDelimitedFormat = true };

await using var stream = File.OpenRead("players.ndjson");

// Deserialize each line into a .NET object await foreach (var player in JsonSequence.DeserializeAsync<Player>(stream, options: options, cancellationToken: ct)) { Console.WriteLine(player?.Name); }

To work with the token DOM directly, use ParseAsync. Without UseNewlineDelimitedFormat, a top-level JSON array is streamed element by element. The yielded DElement is backed by a reused document and is valid only until the next iteration; call Clone() or convert it with To<T>() if it must be retained.

using REDox; using REDox.Json;

await using var stream = File.OpenRead("events.json"); // [ { "level": "info", ... }, { "level": "error", ... }, ... ]

var errors = new List<DElement>();

await foreach (var element in JsonSequence.ParseAsync(stream, cancellationToken: ct)) { if (element.TryGetProperty("level", out var level) && level.GetString() == "error") { errors.Add(element.Clone()); } }

DOX format and deep clone

DOX is a binary layout designed to mirror the in-memory token array closely. Constructing a document from DOX requires little structural reconstruction, which also makes DOX suitable for serialization-based deep clone: the result is an independent, editable copy rather than a shared read-only view.

using REDox;

byte[] dox = DoxSerializer.Serialize(player); var clone = DoxSerializer.Deserialize<Player>(dox);

Compatibility layers

Compatibility packages allow existing serializer options to be adapted to RE:Dox.

using REDox.Serialization.SystemTextJson;

var stjSettings = new SystemTextJsonSerializerSettings(new System.Text.Json.JsonSerializerOptions { PropertyNamingPolicy = System.Text.Json.JsonNamingPolicy.CamelCase });

Newtonsoft.Json

using Newtonsoft.Json; using REDox.Serialization.NewtonsoftJson;

var nsjSettings = new NewtonsoftJsonSerializerSettings(new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore });

DataContractJsonSerializer

using REDox.Serialization.DataContractJson;

var dcjSettings = new DataContractJsonSerializerSettings( new System.Runtime.Serialization.Json.DataContractJsonSerializerSettings());

Requirements

.NET 10 or later.

How to build

1. Clone with submodules

Test and benchmark data (simdjson-data, JSONTestSuite, json5-tests, toml-test) live under external/ as git submodules.

git clone --recurse-submodules https://github.com/CAPCOM-TD-OSS/REDox.git cd REDox

If you already cloned without submodules:

git submodule update --