msgpack encoding for Go

Introduction

github.com/vmihailenco/msgpack is a fast and efficient implementation of MessagePack encoding for Go programming language.

Installation

msgpack supports 2 last Go versions and requires support for Go modules. So make sure to initialize a Go module:

go mod init github.com/my/repo

And then install msgpack/v5 (note v5 in the import; omitting it is a popular mistake):

go get github.com/vmihailenco/msgpack/v5

Quickstart

import "github.com/vmihailenco/msgpack/v5"

type Item struct {
    Foo string
}

func main() {
    b, err := msgpack.Marshal(&Item{Foo: "bar"})
    if err != nil {
        panic(err)
    }

    var item Item
    err = msgpack.Unmarshal(b, &item)
    if err != nil {
        panic(err)
    }
    fmt.Println(item.Foo)
    // Output: bar
}

Struct tags

msgpack supports following struct field tags:

• msgpack:"-" - ignores the field.
• msgpack:"myname" - overwrites the default field name.
• msgpack:"alias:old_name" - adds an alias for the field. Use it to decode data that contains the field under the old name (alias).
• msgpack:",omitempty" - omits the field from the encoding if the field value is empty.
• msgpack:",as_array" - uses msgpack array encoding to encode the struct.
• msgpack:",inline" - inlines the struct.
• msgpack:",noinline" - prevents inlining the struct.

Custom encoding/decoding

msgpack provides two ways to overwrite the default encoding. The first one uses msgpack.CustomEncoder/CustomDecoder interfaces and is preferred:

type Item struct {
    Foo string
}

var _ msgpack.CustomEncoder = (*Item)(nil)

func (i *Item) EncodeMsgpack(enc *msgpack.Encoder) error {
    return enc.EncodeString(i.Foo)
}

var _ msgpack.CustomDecoder = (*Item)(nil)

func (i *Item) DecodeMsgpack(dec *msgpack.Decoder) error {
    s, err := dec.DecodeString()
    if err != nil {
        return err
    }
    i.Foo = s
    return nil
}

The second one uses well-known bytes marshalers:

type Item struct {
    Foo string
}

var _ msgpack.Marshaler = (*Item)(nil)

func (i *Item) MarshalMsgpack() ([]byte, error) {
    return msgpack.Marshal(i.Foo)
}

var _ msgpack.Unmarshaler = (*Item)(nil)

func (i *Item) UnmarshalMsgpack(b []byte) error {
    return msgpack.Unmarshal(b, &i.Foo)
}

Omitting empty fields

To omit a field when empty:

type Item struct {
    Foo string `msgpack:",omitempty"`
}

To omit all empty fields in a struct:

type Item struct {
    _msgpack struct{} `msgpack:",omitempty"`
}

Encoding structs as msgpack arrays

To save some bytes and encode a struct as a msgpack array:

type Item struct {
    _msgpack struct{} `msgpack:",as_array"`

    Foo string
}

// Produces `["bar"]`.
msgpack.Marshal(&Item{Foo: "bar"})

Alternatively this can be enabled on the encoder level for all structs:

enc := msgpack.NewEncoder(&buf)
enc.UseArrayEncodedStructs(true)

// Produces `["bar"]`.
enc.Encode(&Item{Foo: "bar"})

Decoding raw data with RawMessage

To encode/decode a raw MessagePack value, use msgpack.RawMessage:

type Item struct {
    Foo msgpack.RawMessage
}

b, err := msgpack.Marshal("some string")
if err != nil {
    panic(err)
}

item := &Item{
    Foo: msgpack.RawMessage(b),
}

Sorting map keys

To produce a stable encoding for Go maps that don't preserve keys order:

var buf bytes.Buffer
enc := msgpack.NewEncoder(&buf)
enc.SetSortMapKeys(true)

err := enc.Encode(map[string]interface{}{
    "hello": "world",
    "foo":   "bar",
})
if err != nil {
    panic(err)
}

Decoding maps into an interface{}

Decoding maps into an interface{} requires special care because msgpack maps don't have types. Instead type is defined on each separate key and value. Which means that we must decode into map[interface{}]interface{} to cover all possibilities.

But to mimic encoding/json behavior this library decodes msgpack maps as map[string]interface{}. Meaning that with default settings you can't decode maps with non-string keys.

So to decode into a map[interface{}]interface{} instead of a map[string]interface{}, use DecodeUntypedMap:

dec := msgpack.NewDecoder(&buf)
dec.SetMapDecoder(func(dec *msgpack.Decoder) (interface{}, error) {
    return dec.DecodeUntypedMap()
})

var m interface{}
err := dec.Decode(&m)
if err != nil {
    panic(err)
}
fmt.Println(m.(map[interface{}]interface{}))

If you know that a map has a fixed type for the keys and values, for example, map[int]string, use DecodeTypedMap:

dec := msgpack.NewDecoder(&buf)
dec.SetMapDecoder(func(dec *msgpack.Decoder) (interface{}, error) {
    return dec.DecodeTypedMap()
})

Please note that this only applies if you don't know the exact type of the map and decode into an interface{}. Otherwise you can decode directly into a Go map:

var m map[int]string
msgpack.Unmarshal(b, &m)

Using JSON struct tags

msgpack can be used as a drop-in replacement for JSON encoding and utilize existing JSON struct tags:

type Item struct {
    Foo string `json:"foo"`
}

var buf bytes.Buffer
enc := msgpack.NewEncoder(&buf)
enc.SetCustomStructTag("json")

// Produces `{"foo": "bar"}`.
enc.Encode(&Item{Foo: "bar"})

MessagePack extensions

MessagePack type extension can be registered with msgpack.RegisterExt. For example, to create your own time encoding:

func init() {
    msgpack.RegisterExt(1, (*MyTime)(nil))
}

type MyTime struct {
    time.Time
}

var (
    _ msgpack.Marshaler   = (*MyTime)(nil)
    _ msgpack.Unmarshaler = (*MyTime)(nil)
)

func (tm *MyTime) MarshalMsgpack() ([]byte, error) {
    b := make([]byte, 8)
    binary.BigEndian.PutUint32(b, uint32(tm.Unix()))
    binary.BigEndian.PutUint32(b[4:], uint32(tm.Nanosecond()))
    return b, nil
}

func (tm *MyTime) UnmarshalMsgpack(b []byte) error {
    if len(b) != 8 {
        return fmt.Errorf("invalid data length: got %d, wanted 8", len(b))
    }
    sec := binary.BigEndian.Uint32(b)
    usec := binary.BigEndian.Uint32(b[4:])
    tm.Time = time.Unix(int64(sec), int64(usec))
    return nil
}

Uptrace

Uptrace is a OpenTelemetry APM powered by OpenTelemetry and ClickHouse. It allows you to identify and fix bugs in production faster knowing what conditions lead to which errors.

You can get started with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.