Show HN: A Go JSON library faster than Sonic, with encoding/JSON/v2 support

Fast JSON encoder/decoder compatible with encoding/json for Go

Drop-in replacement of encoding/json. Values are decoded, and their errors are reported, as encoding/json of Go 1.27 (built on encoding/json/v2) does, on every Go version. Only the syntax errors of decoding and the reading of a stream token by token (Token, More, InputOffset, and Decode between calls of Token) follow the encoding/json the program is built with. Values are encoded as the encoding/json the program is built with encodes them, except for some behaviors of Go 1.27 that apply on every version, such as the limit of 10000 nested levels, map keys of pointer types, and the syntax errors of MarshalJSON output, Compact and Indent. The MarshalJSONTo and UnmarshalJSONFrom methods are called before MarshalJSON and UnmarshalJSON, as encoding/json of Go 1.27 calls them: the ones which take the Encoder and the Decoder of github.com/goccy/go-json/jsontext first, on every Go version, and the ones which take those of encoding/json/jsontext where the program is built with Go 1.27. As encoding/json does, they are given an encoder and a decoder with the options of encoding/json (DefaultOptionsV1), and the decoder with the ones of DisallowUnknownFields and UseNumber of a Decoder, so that MarshalEncode and UnmarshalDecode within the methods encode and decode as this package does; UseNumber is given only to the methods of github.com/goccy/go-json/jsontext, as encoding/json exports no option of it.

Drop-in replacement of encoding/json/v2 and encoding/json/jsontext by github.com/goccy/go-json/v2 and github.com/goccy/go-json/jsontext, with the API and behavior of those packages of Go 1.27 on Go 1.23 and later, with no need of GOEXPERIMENT=jsonv2. Their types are their own, not aliases of the ones of the standard library: replace both imports together, and declare MarshalJSONTo and UnmarshalJSONFrom methods with the Encoder and Decoder of github.com/goccy/go-json/jsontext, since methods taking the ones of encoding/json/jsontext are not called.

Fast ( See Benchmark section )

MarshalOf and UnmarshalOf, which take the value by its type and save the allocations Marshal and Unmarshal need for an interface{} argument

Flexible customization with options

Coloring the encoded string

Can propagate context.Context to MarshalJSON or UnmarshalJSON

Can dynamically filter the fields of the structure type-safely

Installation

go get github.com/goccy/go-json

How to use

Replace import statement from encoding/json to github.com/goccy/go-json

-import "encoding/json" +import "github.com/goccy/go-json"

Code which uses encoding/json/v2 or encoding/json/jsontext replaces them likewise

-import "encoding/json/v2" +import "github.com/goccy/go-json/v2" -import "encoding/json/jsontext" +import "github.com/goccy/go-json/jsontext"

JSON library comparison

name encoder decoder compatible with encoding/json

encoding/json yes yes N/A

json-iterator/go yes yes partial

easyjson yes yes no

gojay yes yes no

segmentio/encoding/json yes yes partial

jettison yes no no

simdjson-go no yes no

bytedance/sonic yes yes partial

goccy/go-json yes yes yes

json-iterator/go isn't compatible with encoding/json in many ways (e.g. json-iterator/go#229 ), but it hasn't been supported for a long time.

segmentio/encoding/json is well supported for encoders, but some are not supported for decoder APIs such as Token ( streaming decode )

bytedance/sonic is compatible with encoding/json in its ConfigStd configuration; by default it doesn't escape HTML, sort the keys of maps or validate strings. It decodes and encodes by native code: SIMD code and, on amd64, code generated at run time. The benchmarks of this repository compare go-json with sonic in both configurations ( see below ).

Other libraries

jingo

I tried the benchmark but it didn't work. Also, it seems to panic when it receives an unexpected value because there is no error handling...

Benchmarking gave very slow results. It seems that it is assumed that the user will use the buffer pool properly. Also, development seems to have already stopped

The JSON libraries of Go are measured doing the same work on GitHub Actions, on amd64 and arm64, and the results are published at https://goccy.github.io/go-json/, measured again whenever go-json, the version of a library or the report changes. The page has every payload, the encode and decode of each library, the allocations, and the results without a live heap.

A comparison is fair only between libraries doing the same work, so the results are shown by category of behavior: the behavior of encoding/json, the behavior of encoding/json/v2, the behavior of encoding/json without HTML escaping, key sorting and string copying, and every library at its fastest. In each category, every library is configured by its options to behave as the category requires, as far as its options allow, and as fast as they allow. Every run checks the behavior of each library on small probes before it measures it, and a library which behaves differently is still shown, marked, with what differs. The result files are attested by GitHub Artifact Attestations: gh attestation verify tells that they were produced by the workflow of this repository.

$ make bench-report

To compare go-json with bytedance/sonic side by side, benchmark by benchmark:

$ make bench-compare-encode $ make bench-compare-decode

BENCH_LIVE_HEAP_MB=64 make bench-compare-decode runs the decode benchmarks with 64 MB of live heap, as a real program has: the GC then runs less often, as it does in such a program.

go-json-fuzz is the repository for fuzzing tests. If you run the test in this repository and find a bug, please commit to corpus to go-json-fuzz and report the issue to go-json.

go-json is very fast in both encoding and decoding compared to other libraries. It's easier to implement by using automatic code generation for performance or by using a dedicated interface, but go-json dares to stick to compatibility with encoding/json and is the simple interface. Despite this, we are developing with the aim of being the fastest library.

Here, we explain the various speed-up techniques implemented by go-json.

The techniques listed here are the ones used by most of the libraries listed above.

Since the only value required for the result of json.Marshal(interface{}) ([]byte, error) is []byte, the only value that must be allocated during encoding is the return value []byte .

Also, as the number of allocations increases, the performance will be affected, so the number of allocations should be kept as low as possible when creating []byte.

Therefore, there is a technique to reduce the number of times a new buffer must be allocated by reusing the buffer used for the previous encoding by using sync.Pool.

Finally, you allocate a buffer that is as long as the resulting buffer and copy the contents into it, you only need to allocate the buffer once in theory.

type buffer struct { data []byte }

var bufPool = sync.Pool{ New: func() any { return &buffer{data: make([]byte, 0, 1024)} }, }

buf := bufPool.Get().(*buffer) data := encode(buf.data) // reuse buf.data

newBuf := make([]byte, len(data)) copy(newBuf, buf)

buf.data = data bufPool.Put(buf)

Elimination of reflection

As you know, the reflection operation is very slow.

Therefore, using the fact that the address position where the type information is stored is fixed for each binary ( we call this typeptr ), we can use the address in the type information to call a pre-built optimized process.

For example, you can get the address to the type information from interface{} as follows and you can use that information to call a process that does not have reflection.

To process without reflection, pass a pointer (unsafe.Pointer) to the value is stored.

type emptyInterface struct { typ unsafe.Pointer ptr unsafe.Pointer }

var typeToEncoder = map[uintptr]func(unsafe.Pointer)([]byte, error){}

func Marshal(v any) ([]byte, error) { iface := (*emptyInterface)(unsafe.Pointer(&v) typeptr := uintptr(iface.typ) if enc, exists := typeToEncoder[typeptr]; exists { return enc(iface.ptr) } ... }

※ In reality, typeToEncoder can be referenced by multiple goroutines, so exclusive control is required.

Encoder

Encode a value without copying it to the heap by MarshalOf

json.Marshal receives an interface{} value. A value which is not a pointer is copied to the heap when it is converted to an interface{} value, so json.Marshal(v) allocates the copy of v for every call, in addition to the result.

json.MarshalOf[T] receives the value by its type. It copies the value to a value in the heap which is reused, so the result is the only allocation.

b, err := json.MarshalOf(v)

Which one to use depends on what is passed:

MarshalNoEscape, which left the value on the stack, is deprecated: the encoder refers to the value by its address, and the address gets invalid when the stack of the goroutine is moved. It is now the same as Marshal.

encoding/json writes the fields of a struct in the order of the struct, and so does go-json by default. A JSON object doesn't define the order of its keys, so when the order doesn't matter to the reader of the JSON, the option json.OptimizeFieldOrder() lets the encoder order the fields as it encodes them fastest:

b, err := json.MarshalWithOption(v, json.OptimizeFieldOrder())

The fields of the same kind ( int, uint, float64, string, bool ) are put together, in the order of the first field of each kind, and are encoded without a dispatch of the VM between them ( see below ).

A field of the struct's own type ( the next node of a list ), if there is one, is put last, so that a list of values is encoded in one frame of the VM instead of one frame for each value.

The keys are the same, only their order differs. The opcodes of a type are compiled for the option apart from the ones in the order of the struct, so the two can be used together.

I explained that you can use typeptr to call a pre-built process from type information.

In other libraries, this dedicated process is processed by making it an function calling like anonymous function, but function calls are inherently slow processes and should be avoided as much as possible.

Therefore, go-json adopted the Instruction-based execution processing system, which is also used to implement virtual machines for programming language.

If it is the first type to encode, create the opcode ( instruction ) sequence required for encoding. From the second time onward, use typeptr to get the cached pre-built opcode sequence and encode it based on it. An example of the opcode sequence is shown below.

json.Marshal(struct{ X int json:"x" Y string json:"y" }{X: 1, Y: "hello"})

When encoding a structure like the one above, create a sequence of opcodes like this:

• opStructFieldHead ( { )
• opStructFieldInt ( "x": 1, )
• opStructFieldString ( "y": "hello" )
• opStructEnd ( } )
• opEnd

※ When processing each operation, write the letters on the right.

In addition, each opcode is managed by the following structure ( Pseudo code ).

type opType int const ( opStructFieldHead opType = iota opStructFieldInt opStructFieldStirng opStructEnd opEnd ) type opcode struct { op opType key []byte next *opcode }

The process of encoding using the opcode sequence is roughly implemented as follows.

func encode(code *opcode, b []byte, p unsafe.Pointer) ([]byte, error) { for { switch code.op { case opStructFieldHead: b = append(b, '{') code = code.next case opStructFieldInt: b = append(b, code.key...) b = appendInt((*int)(unsafe.Pointer(uintptr(p)+code.offset))) code = code.next case opStructFieldString: b = append(b, code.key...) b = appendString((*string)(unsafe.Pointer(uintptr(p)+code.offset))) code = code.next case opStructEnd: b = append(b, '}') code = code.next case opEnd: goto END } } END: return b, nil }

In this way, the huge switch-case is used to encode by manipulating the linked