msgpack encoding for Go

Introduction

github.com/vmihailenco/msgpackopen in new window is a fast and efficient implementation of MessagePackopen in new window encoding for Go programming language.

Installation

msgpack supports 2 last Go versions and requires support for Go modulesopen in new window. So make sure to initialize a Go module:

go mod init github.com/my/repo

And then install msgpack/v5 (note v5 in the import; omitting it is a popular mistake):

go get github.com/vmihailenco/msgpack/v5

Quickstart

import "github.com/vmihailenco/msgpack/v5"

type Item struct {
    Foo string
}

func main() {
    b, err := msgpack.Marshal(&Item{Foo: "bar"})
    if err != nil {
        panic(err)
    }

    var item Item
    err = msgpack.Unmarshal(b, &item)
    if err != nil {
        panic(err)
    }
    fmt.Println(item.Foo)
    // Output: bar
}

Struct tags

msgpack supports following struct field tags:

  • msgpack:"-" - ignores the field.
  • msgpack:"myname" - overwrites the default field name.
  • msgpack:"alias:old_name" - adds an alias for the field. Use it to decode data that contains the field under the old name (alias).
  • msgpack:",omitempty" - omits the field from the encoding if the field value is empty.
  • msgpack:",as_array" - uses msgpack array encoding to encode the struct.
  • msgpack:",inline" - inlines the struct.
  • msgpack:",noinline" - prevents inlining the struct.

Custom encoding/decoding

msgpack provides two ways to overwrite the default encoding. The first one uses msgpack.CustomEncoder/CustomDecoder interfaces and is preferred:

type Item struct {
    Foo string
}

var _ msgpack.CustomEncoder = (*Item)(nil)

func (i *Item) EncodeMsgpack(enc *msgpack.Encoder) error {
    return enc.EncodeString(i.Foo)
}

var _ msgpack.CustomDecoder = (*Item)(nil)

func (i *Item) DecodeMsgpack(dec *msgpack.Decoder) error {
    s, err := dec.DecodeString()
    if err != nil {
        return err
    }
    i.Foo = s
    return nil
}

The second one uses well-known bytes marshalers:

type Item struct {
    Foo string
}

var _ msgpack.Marshaler = (*Item)(nil)

func (i *Item) MarshalMsgpack() ([]byte, error) {
    return msgpack.Marshal(i.Foo)
}

var _ msgpack.Unmarshaler = (*Item)(nil)

func (i *Item) UnmarshalMsgpack(b []byte) error {
    return msgpack.Unmarshal(b, &i.Foo)
}

Omitting empty fields

To omit a field when empty:

type Item struct {
    Foo string `msgpack:",omitempty"`
}

To omit all empty fields in a struct:

type Item struct {
    _msgpack struct{} `msgpack:",omitempty"`
}

Encoding structs as msgpack arrays

To save some bytes and encode a struct as a msgpack array:

type Item struct {
    _msgpack struct{} `msgpack:",as_array"`

    Foo string
}

// Produces `["bar"]`.
msgpack.Marshal(&Item{Foo: "bar"})

Alternatively this can be enabled on the encoder level for all structs:

enc := msgpack.NewEncoder(&buf)
enc.UseArrayEncodedStructs(true)

// Produces `["bar"]`.
enc.Encode(&Item{Foo: "bar"})

Decoding raw data with RawMessage

To encode/decode a raw MessagePack value, use msgpack.RawMessage:

type Item struct {
    Foo msgpack.RawMessage
}

b, err := msgpack.Marshal("some string")
if err != nil {
    panic(err)
}

item := &Item{
    Foo: msgpack.RawMessage(b),
}

Sorting map keys

To produce a stable encoding for Go maps that don't preserve keys order:

var buf bytes.Buffer
enc := msgpack.NewEncoder(&buf)
enc.SetSortMapKeys(true)

err := enc.Encode(map[string]interface{}{
    "hello": "world",
    "foo":   "bar",
})
if err != nil {
    panic(err)
}

Decoding maps into an interface{}

Decoding maps into an interface{} requires special care because msgpack maps don't have types. Instead type is defined on each separate key and value. Which means that we must decode into map[interface{}]interface{} to cover all possibilities.

But to mimic encoding/json behavior this library decodes msgpack maps as map[string]interface{}. Meaning that with default settings you can't decode maps with non-string keys.

So to decode into a map[interface{}]interface{} instead of a map[string]interface{}, use DecodeUntypedMap:

dec := msgpack.NewDecoder(&buf)
dec.SetMapDecoder(func(dec *msgpack.Decoder) (interface{}, error) {
    return dec.DecodeUntypedMap()
})

var m interface{}
err := dec.Decode(&m)
if err != nil {
    panic(err)
}
fmt.Println(m.(map[interface{}]interface{}))

If you know that a map has a fixed type for the keys and values, for example, map[int]string, use DecodeTypedMap:

dec := msgpack.NewDecoder(&buf)
dec.SetMapDecoder(func(dec *msgpack.Decoder) (interface{}, error) {
    return dec.DecodeTypedMap()
})

Please note that this only applies if you don't know the exact type of the map and decode into an interface{}. Otherwise you can decode directly into a Go map:

var m map[int]string
msgpack.Unmarshal(b, &m)

Using JSON struct tags

msgpack can be used as a drop-in replacement for JSON encoding and utilize existing JSON struct tags:

type Item struct {
    Foo string `json:"foo"`
}

var buf bytes.Buffer
enc := msgpack.NewEncoder(&buf)
enc.SetCustomStructTag("json")

// Produces `{"foo": "bar"}`.
enc.Encode(&Item{Foo: "bar"})

MessagePack extensions

MessagePack type extensionopen in new window can be registered with msgpack.RegisterExt. For example, to create your own time encoding:

func init() {
    msgpack.RegisterExt(1, (*MyTime)(nil))
}

type MyTime struct {
    time.Time
}

var (
    _ msgpack.Marshaler   = (*MyTime)(nil)
    _ msgpack.Unmarshaler = (*MyTime)(nil)
)

func (tm *MyTime) MarshalMsgpack() ([]byte, error) {
    b := make([]byte, 8)
    binary.BigEndian.PutUint32(b, uint32(tm.Unix()))
    binary.BigEndian.PutUint32(b[4:], uint32(tm.Nanosecond()))
    return b, nil
}

func (tm *MyTime) UnmarshalMsgpack(b []byte) error {
    if len(b) != 8 {
        return fmt.Errorf("invalid data length: got %d, wanted 8", len(b))
    }
    sec := binary.BigEndian.Uint32(b)
    usec := binary.BigEndian.Uint32(b[4:])
    tm.Time = time.Unix(int64(sec), int64(usec))
    return nil
}
Get insights and updates in your inbox: