Bit Fields, Byte Order and Serialization

In order to store data most efficiently, the C language has supported bit fields since its early days. While saving a few bytes of memory isn’t as critical today, bit fields remain widely used in scenarios like network packets. Endianness adds complexity to bit field handling – especially since network packets are typically big-endian, while most modern architectures are little-endian. This article explores these problems and their solutions, including my reflection-based serialization project.

Memory layout of bit fields

The memory layout of bit fields is implementation-defined. In a typical little-endian environment, bit fields start from the lower bits of the lower byte and extend toward higher bits and bytes. In a typical big-endian environment, bit fields start from the higher bits of the lower byte and extend toward lower bits and higher bytes.

Let’s consider a practical scenario. Suppose we want to use a 32-bit integer to store a date. How should we achieve this? A simple approach is to store the number of days from a fixed point of time (e.g. 1 January 1900). We can calculate the number of years that can be expressed as follows:

(1)

However, with this approach, extracting specific year, month, and day information becomes very cumbersome. A simpler way is to store the year, month, and day as bit fields. We can define the following struct, using only 32 bits:

  struct Date {
    int      year  : 23;
    unsigned month : 4;
    unsigned day   : 5;
  };

Our intention is to use a 23-bit signed integer for the year (ranging from -4,194,304 to 4,194,303), a 4-bit unsigned integer for the month (0–15, covering legal values 1–12), and a 5-bit unsigned integer for the day (0–31, covering legal values 1–31). This representation is similarly compact, with a slightly narrower range, but it’s quite sufficient and much more convenient for many common usages (excepting interval calculation).

If you want to store data in a file or send it over a network, directly sending in-memory data is potentially problematic. Big-endian and little-endian environments have different memory layouts for such bit fields.

Little-endian environments store them as follows (the memory layout on mainstream processors):

Big-endian environments store them differently (the memory layout expected by network packets):

As we can see, these two approaches differ significantly. If we want to serialize in the big-endian order (which is the standard in the networking world), we have two possible solutions:

1. When bit fields don’t cross byte boundaries, we can design separate structs for big-endian and little-endian systems, using conditional compilation to select the appropriate definition.
2. When bit fields cross byte boundaries, the above approach alone isn’t sufficient. We need to define different structs (with reversed bit-field ordering for big-endian versus little-endian) and perform byte-order conversion during serialization and deserialization (using functions like htonl).

Examples of the simple approach

Here are some actual definitions from Linux.

A simple case (single byte, only requiring order reversal):

  struct iphdr
    {
  #if __BYTE_ORDER == __LITTLE_ENDIAN
      unsigned int ihl:4;
      unsigned int version:4;
  #elif __BYTE_ORDER == __BIG_ENDIAN
      unsigned int version:4;
      unsigned int ihl:4;
  #else
  # error	"Please fix <bits/endian.h>"
  #endif
      // …
    };

Listing 1 is a more complex case (multiple bytes, but not crossing byte boundaries).

struct tcphdr
  {
    __extension__ union
    {
      // …
      struct
      {
        uint16_t source;
        uint16_t dest;
        uint32_t seq;
        uint32_t ack_seq;
# if __BYTE_ORDER == __LITTLE_ENDIAN
        uint16_t res1:4;
        uint16_t doff:4;
        uint16_t fin:1;
        uint16_t syn:1;
        uint16_t rst:1;
        uint16_t psh:1;
        uint16_t ack:1;
        uint16_t urg:1;
        uint16_t res2:2;
# elif __BYTE_ORDER == __BIG_ENDIAN
        uint16_t doff:4;
        uint16_t res1:4;
        uint16_t res2:2;
        uint16_t urg:1;
        uint16_t ack:1;
        uint16_t psh:1;
        uint16_t rst:1;
        uint16_t syn:1;
        uint16_t fin:1;
# else
#  error "Adjust your <bits/endian.h> defines"
# endif
        // …
      };
    };
};

As we can see, the field ordering here is quite distinctive. This arrangement ensures these fields have a consistent memory layout on both little-endian and big-endian systems.

Example of the ‘standard’ approach

Since its bit fields cross byte boundaries, merely modifying the field order will not do for our Date struct as shown above. The conventional approach is to use the code in Listing 2 for serialization.

#ifdef _WIN32
#include <winsock2.h>  // htonl/...
#else
#include <arpa/inet.h> // htonl/...
#endif
struct Date {
  union {
    struct {
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
      unsigned   day : 5;
      unsigned month : 4;
      int       year : 23;
#else
      int       year : 23;
      unsigned month : 4;
      unsigned   day : 5;
#endif
    };
    unsigned year_month_day;
  };
};
// …
Date date;
// …
date.year_month_day = htonl(date.year_month_day);
// date is ready for transmission

Under this ‘standard’ approach, bit fields that can be assembled into a single integer must be placed in a struct, which is then wrapped in a union. This allows us to directly access the integer for byte-order conversion later. Of course, we need to determine whether the platform is little-endian or big-endian to choose the appropriate struct definition.

Since macros for endianness detection aren’t standardized, such code isn’t truly cross-platform. However, the code above works correctly on mainstream compilers like GCC, Clang, and MSVC. While GCC and Clang recognize the special macros __BYTE_ORDER__ and __ORDER_LITTLE_ENDIAN__, MSVC recognizes neither. Therefore, MSVC defaults to the first case (#if 0 == 0), which conveniently matches Windows’ use of little-endian order.

This approach remains cumbersome, requiring manual maintenance of two code branches and attention to htonl-like function calls – exactly once for serialization or deserialization, no more and no less! Experience from real projects shows this method is error-prone, and issues like missed or duplicate byte-order conversions are common.

It is even worse than that. While this approach is common in C, and all mainstream C++ compilers continue to allow such code to work, it is technically undefined behaviour in C++. The orthodox way is to use bit_cast or memcpy, which would make the code even more complicated.

Serialization features in Mozi

At the 2023 C++ Summit (China), I presented static reflection and demonstrated the Mozi open source project [mozi] that utilized manual reflection techniques. Using macros and templates, this project provides basic reflection functionality in C++17, even though C++ does not yet support static reflection natively.

This year I found more time to implement serialization and deserialization for network messaging in Mozi. Now, we only need to define a reflected struct to enable fully automated serialization and deserialization – users don’t need to manually write field-specific handling code or perform byte-order conversions. For example, for a Date struct (without using bit fields for now), we can serialize and deserialize it as shown in Listing 3.

#include <assert.h>
#include <stdint.h>
#include <mozi/bit_fields.hpp>
#include <mozi/net_pack.hpp>
#include <mozi/print.hpp>
#include <mozi/serialization.hpp>
#include <mozi/struct_reflection.hpp>
DEFINE_STRUCT(
  Date,
  (int16_t)year,
  (uint8_t)month,
  (uint8_t)day
);
DECLARE_EQUAL_COMPARISON(Date);
int main()
{
  Date date{2024, 8, 19};
  mozi::println(date);
  mozi::serialize_t result;
  mozi::net_pack::serialize(date, result);
  mozi::println(result);
  mozi::deserialize_t input{result};
  Date date2{};
  auto ec =
    mozi::net_pack::deserialize(date2, input);
  assert(ec ==
         mozi::deserialize_result::success);
  assert(input.empty());
  assert(date == date2);
}

This program will output the following result:

  {
    year: 2024,
    month: 8,
    day: 19
  }
  { 7, 232, 8, 19 }

Here are some important details:

• Reflected structs don’t provide comparison operations by default to avoid unnecessary ‘unused function’ warnings. However, you can easily enable comparison operations using macros like DECLARE_COMPARISON or DECLARE_EQUAL_COMPARISON. These operations perform member-wise comparisons.
• As a reflected object, date can be output directly to cout using mozi::print/println. Due to special handling in the code, uint8_t (i.e. unsigned char) is output as an integer rather than as a character, as is the usual case when using <<.
• The code uses the serialization result as input for deserialization, where input is a span of bytes. When deserialization completes successfully, the following conditions should be met:
  • ec indicates success
  • input` is empty (indicating all input has been consumed)
  • date2 equals date

I would like to mention that the implementation doesn’t use non-standard functions like htons. Instead, it uses handwritten platform-independent function templates. These templates are friendly for compile-time programming, and they can be translated into optimal assembly instructions during serialization (under GCC and Clang compilers at least), or even eliminated entirely on big-endian systems. You can check the results in the following link: https://godbolt.org/z/f1Gn8Mcx1

(The deserialization logic is similar, but the compiler wasn’t able to generate similarly highly-optimized code, possibly due to alignment.)

The serialization target type

For flexibility and safety, the serialization target is a vector. However, for targets with known lengths, we should be able to avoid heap memory allocation entirely. Therefore, in environments that support polymorphic allocators, the default serialization target type is set to std::pmr::vector<std::byte> (which can be overridden by setting the macro MOZI_SERIALIZATION_USES_PMR to 0 or 1). Using allocators provided by C++17, we can avoid the heap allocations easily in such circumstances. Here’s an example:

  std::byte buffer[50];
  std::pmr::monotonic_buffer_resource
    res(buffer, sizeof buffer);
  std::pmr::polymorphic_allocator<std::byte>
    a(&res);
  mozi::serialize_t result(a);
  result.reserve(50);
  // Heap memory will now be allocated only
  // if the size exceeds 50 bytes
  mozi::net_pack::serialize(date, result);
  // Use result as you like in current scope

The bit_field type

Reflected structs don’t directly support bit fields; but they don’t have to. Instead, we can define a special class template bit_field that represents bit fields and automatically converts objects to the appropriate memory layout during serialization.

Objects of this type use the most compact integer type (uint8_t, uint16_t, or uint32_t) to store their data. The type supports construction and assignment from integers, as well as on-demand conversion to appropriate integer types. Using objects of this type feels similar to using regular integers, but like bit fields, the data is limited to a specified number of bits, with values being truncated if they exceed this limit.

Here’s an example demonstrating its basic usage:

  bit_field<4> f{13};  // Construct from integer
  cout << f << '\n';   // 13 (automatically
                       // converts to unsigned)
  f = 17;              // Can be assigned to
   << f << '\n';       // 1 (due to truncation)

The previous example showed the most common case – an unsigned bit_field. Since bit fields can also be signed (like our earlier year bit field), bit_field uses a second template parameter to specify whether it is signed (unsigned by default). Using SFINAE, I’ve constrained unsigned bit_fields to be convertible with unsigned, while signed bit_fields are convertible with (signed) int.

Here’s some code demonstrating the subtle differences between signed and unsigned bit_fields:

  bit_field<4> f1{13};
  cout << f1 << '\n';  // 13
  bit_field<4, bit_field_signed> f2{13};
   << f2 << '\n';  // -3 (due to truncation)
  f1 = -1;             // Triggers warning with
                       // -Wsign-conversion
  cout << f1 << '\n';  // 15
  f2 = -1;             // OK
  cout << f2 << '\n';  // -1

Bit-field containers

Just as we needed to encapsulate bit fields in a struct for byte-order conversion earlier, we need to explicitly place multiple bit fields into a bit-field container to enable proper byte-order conversion. The serialization process explicitly checks that the total number of bits is 8, 16, or 32 – otherwise, we get a compilation error.

In practice, we can simply change the Date definition to:

  DEFINE_BIT_FIELDS_CONTAINER(
    Date,
   (bit_field<23, bit_field_signed>)year,
    (bit_field<4>)month,
    (bit_field<5>)day
  );

We do not need to change anything else in the code, and it will produce new output:

  {
    year: 2024,
    month: 8,
    day: 19
  }
  { 0, 15, 209, 19 }

Listing 4 is the complete working code for experimentation and reference.

#include <assert.h>
#include <mozi/bit_fields.hpp>
#include <mozi/net_pack.hpp>
#include <mozi/print.hpp>
#include <mozi/serialization.hpp>
#include <mozi/struct_reflection.hpp>
using mozi::bit_field;
using mozi::bit_field_signed;
DEFINE_BIT_FIELDS_CONTAINER(
  Date,
  (bit_field<23, bit_field_signed>)year,
  (bit_field<4>)month,
  (bit_field<5>)day
);
DECLARE_EQUAL_COMPARISON(Date);
int main()
{
  Date date{2024, 8, 19};
  mozi::println(date);
  mozi::serialize_t result;
  mozi::net_pack::serialize(date, result);
  mozi::println(result);
  mozi::deserialize_t input{result};
  Date date2{};
  auto ec =
    mozi::net_pack::deserialize(date2, input);
  assert(ec ==
         mozi::deserialize_result::success);
  assert(input.empty());
  assert(date == date2);
}

If we change ‘23’ to ‘22’, we get a compilation error immediately (see Figure 1).

…
…/mozi/net_pack_bit_fields.hpp:41:5: fatal error: static_assert failed due to requirement 'size_bits == 8 || size_bits == 16 || size_bits == 32' "A bit-fields container must have 8, 16, or 32 bits"
    static_assert(size_bits == 8 || size_bits == 16 || size_bits == 32,
    ^             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
…

In other words, if padding is needed, my current approach requires explicitly writing out the padding rather than letting the compiler handle it automatically. I believe this approach better ensures serialization consistency.

It might be worth noting that, unlike the built-in bit fields (especially those on big-endian architectures), the in-memory layout of Date is now different from the serialization result. The serialization result is like the true (big-endian) bit fields, but in memory year, month, and day are represented as integral values, which can be accessed faster than bit fields. So we get the benefits of simplicity and performance, at the cost of a few more bytes of memory use.

Nested struct handling

The processing of reflected structs, whether for output or serialization, is recursive. For objects without variable-length data (which lacks an intuitive/direct handling method and isn’t supported in the current net_pack scheme), we can now simply nest and use them. For example:

  DEFINE_STRUCT(
    Data,
    (std::array<char, 8>)name,
    (uint16_t)age,
    (Date)last_update
  );
  // … 
  Data data{{"John"}, 17, {2024, 8, 19}};
  mozi::println(data);
  mozi::serialize_t result;
  mozi::net_pack::serialize(data, result);
  mozi::println(result);

Here’s the output we get (using -DMOZI_PRINT_USE_FMTLIB flag for prettier formatting with the {fmt} library [fmt]):

  {
    name: { 'J', 'o', 'h', 'n', '\x00', '\x00', 
            '\x00', '\x00' },
    age: 17,
    last_update: {
        year: 2024,
        month: 8,
        day: 19
    }
  }
  { 74, 111, 104, 110, 0, 0, 0, 0, 0, 17, 0, 15, 
    209, 19 }

Quite convenient, isn’t it?

Extensible serialization schemes

The net_pack serialization demonstrated above is stateless and simple, suitable for basic network messaging scenarios. Mozi supports more sophisticated serialization schemes, including:

• Extending existing serialization schemes via explicit specialization to support your custom data types
• Creating new serialization schemes that can work alongside existing ones (next scheme in list is used only when previous ones don’t support a type)
• Using state data during serialization/deserialization to track counts, nesting levels, and suchlike

For more details on these advanced features, please refer to the test code in the Mozi project.

I hope you find my work and approach useful, and can apply them in your software projects. If you find any issues in the Mozi project, please don’t hesitate to report them. And patches are even more welcome!

References

[fmt] https://github.com/fmtlib/fmt

[mozi] https://github.com/adah1972/mozi

Wu Yongwei Having been a programmer and software architect, Yongwei is currently a consultant and trainer on modern C++. He has nearly 30 years’ experience in systems programming and architecture in C and C++. His focus is on the C++ language, software architecture, performance tuning, design patterns, and code reuse. He has a programming page at http://wyw.dcweb.cn/.