This repository contains utilities for serializing/deserializing C++ objects for DUNE DAQ. Serialization allows objects to be sent across the network or persisted to disk.

Quick start

An appropriately-defined C++ type (see below) can be serialized/deserialized as follows:

MyClass m;
m.some_member=3;
// ... set other parts of m...
dunedaq::serialization::SerializationType stype=dunedaq::serialization::kMsgPack; // MsgPack is only supported option for now
std::vector<uint8_t> bytes=dunedaq::serialization::serialize(m, stype);
 
// ...elsewhere, after receiving the serialized object:
MyClass m_recv=dunedaq::serialization::deserialize<MyClass>(bytes);

Making types serializable

Classes can be made serializable by adding convertor functions for msgpack.

The easiest way to make your class (de)serializable is with the DUNE_DAQ_SERIALIZE() and DUNE_DAQ_SERIALIZABLE() convenience macros provided in Serialization.hpp:

// A type that's made serializable "intrusively", ie, by changing the type itself
namespace dunedaq {
namespace ns {
struct MyTypeIntrusive
{
  int some_int;
  std::string some_string;
  std::vector<double> some_vector;
 
  DUNE_DAQ_SERIALIZE(MyTypeIntrusive, some_int, some_string, some_vector);
};
} // namespace ns
 
DUNE_DAQ_SERIALIZABLE(ns::MyTypeIntrusive, "MyTypeIntrusive"); // should be in the dunedaq namespace
} // namespace dunedaq

You may not be able to change the type itself, either because you don't have access to it, or because it lives in a package that cannot depend on the serialization package. In this case, you can use the DUNE_DAQ_SERIALIZE_NON_INTRUSIVE() macro instead. In the global namespace, call DUNE_DAQ_SERIALIZE_NON_INTRUSIVE() with first argument the namespace of your class, second argument the class name, and the rest of the arguments listing the member variables. For example:

// A type that's made serializable "non-intrusively", ie, without changing the type
namespace ns {
struct MyType
{
  int some_int;
  std::string some_string;
  std::vector<double> some_vector;
};
} // namespace ns
 
DUNE_DAQ_SERIALIZE_NON_INTRUSIVE(ns, MyType, some_int, some_string, some_vector);

A complete example, showing both intrusive and non-intrusive strategies, can be found in theSerialization_test unit test.

It is also a good idea to define a unit test for each message type being serialized and deserialized, to ensure that all macro calls are complete (i.e. not missing member variables as parameters). See the dfmessages tests for good examples.

Full instructions for serializing arbitrary types with msgpack are available here. These include instructions for (de)serializing classes that are not default-constructible. Several custom serializers have been implemented, such as Fragment_serialization.hpp in dfmessages. Note that custom seralizers may need to use macros or functions specific to serialization libraries. This is the only place where seralizers should be referenced outside of this library!

Serialization Libraries

Currently, only MsgPack is supported. Additional serialization methods can be implmemented using the instructions in Serialization.hpp. The design of Serialization.hpp was created with only a few serialization types envisioned. If multiple serialization libraries need to be added, the design of this package may need to be revisited.

Choice of serialization methods: there are many, many libraries and formats for serialization/deserialization, with a range of tradeoffs. For the binary format, we wanted a library that allows serialization of arbitrary types, rather than requiring types to be specified in a library-specific manner or DSL (this rules out, eg, protobuf). We may have to revisit that requirement if we find that msgpack does not meet performance requirements. However, via the macros DUNE_DAQ_SERIALIZE and DUNE_DAQ_SERIALIZE_NON_INSTRUSIVE, it is possible to inject code into the message types if needed by the serialization library (as is done for MsgPack and was done for nlohmann::json).

Serialization libraries are tested in Serialization_test, by adding to the BOOST_DATA_TEST_CASE macro calls.