Load-Resilient C++ Servers

A real-time server can be stable under moderate loads but fail when requests suddenly arrive faster than the server can process them. Incoming connections keep on coming, queues fill up, delays rise, and memory usage increases. The process may even stay alive, but the quality of service becomes much worse from the user’s point of view. The issue is often not just a lack of computational resources, but the lack of a clear strategy for coping with overload. This also results in higher latency, disruption of services, losses and a tarnished corporate reputation due to loss of customers.

This article discusses a simple and elegant architecture of a C++ server capable of dealing with load spikes. This server uses a non-blocking I/O thread, bounded queues, worker threads, and backpressure control. Instead of trying to increase the throughput to the maximum possible level, the architecture aims to provide predictable latencies, bounded resource consumption, clear ownership of the network I/O, and graceful degradation in case of overload.

Introduction

Consider a RESTful API that consistently demonstrates reasonable response times and resource utilization. One day, however, due to a software release, an automated retry storm or an accidental increase of traffic volume, the rate of incoming requests temporarily surpasses the processing capacity of the server. Connections are still accepted, queues start growing, response times skyrocket from milliseconds to seconds, and memory starts being used up. Even though the process is apparently alive and well, the users are experiencing increasing latency and failure of their requests.

The problem is seldom about insufficient processing capacity; most of the time it is about the inability to have a coherent approach to handling requests once the system is overwhelmed. Servers accepting requests indefinitely will shift the overload from the network level to the level of the application memory, increasing queueing delay, tail latency, and recovery difficulties. An overloaded system might look like a working one even if it does not deliver acceptable service anymore.

Sometimes, a very simple architecture might solve the problem quite effectively. This architecture would consist of a single non-blocking I/O thread with select() [Linux-7], a bounded queue for incoming requests, a worker thread pool for CPU-heavy or blocking application logic, and an output queue going back from the workers to the I/O thread. It allows a good separation of concerns while still maintaining simplicity. Instead of maximizing the throughput, the goal is to maintain predictable latency, bounded memory usage and graceful degradation in case the load is higher than the processing capacity of the server.

Despite the common perception that select() is obsolete, its limitations are often not well understood. Instead of imposing some limitations on the request rate, the limitation of select() lies in the traditional fd_set interface with its FD_SETSIZE value and the need to constantly rebuild and scan the sets of descriptors. The costs of doing that are quite negligible when the server is handling a relatively small number of concurrent connections, thus select() is an excellent example of architecture that can easily be applied to epoll, kqueue, Boost.Asio and even other modern event-driven frameworks. [Linux-2] [OpenBSD] [Boost]

One of the most powerful methods to control the overload is to make buffering explicit. Instead of having unlimited queue, the bounded queue makes the application choose how it needs to handle overload. Depending on the workload, the decision could be either to immediately drop requests, admit them with a delay, use rate limiting, shed some of the low priority tasks or use TCP flow-control features. What is important about this approach is that overload is becoming an explicit engineering choice, not an unintentional effect of buffering. [IETF-5]

Once the queue is bounded, however, another question arises: how large should that bound be? An arbitrary capacity may prevent unlimited memory growth, but an excessively large queue can still produce unacceptable waiting times. Queue capacity should therefore be related to the application’s latency objectives and expected workload. We return to this question in the Backpressure and Sizing section.

Correctness of this architecture relies on understanding how the underlying OS works. On Linux, the accepted sockets are not guaranteed to inherit O_NONBLOCK property, which is why [Linux-1]

  accept4(..., SOCK_NONBLOCK | SOCK_CLOEXEC)

should be used as the default setting. Also, select() modifies its descriptor sets and, on Linux, its timeout value on each call. Similarly, recv() and send() may process only part of the data or return EAGAIN status code. Closing a monitored socket from another thread can lead to undefined behavior. Thus, this naturally leads to the architecture where I/O thread manages the lifetimes of sockets and all the network-related I/O operations, and worker threads are solely responsible for request processing and producing responses.

In the following sections we are going to implement this architecture with modern C++.

Framing the problem and design goals

Processing a high volume of requests is not the problem per se. A robust server must continue to operate reliably when its capacity to process incoming work is temporarily surpassed. Under bursty loads, design goals should shift from optimizing for maximum throughput to having bounded memory consumption, predictable latency, explicit overload handling capabilities, and sufficient observability to understand where processing time is being spent. Sometimes, a server that rejects excessive work and stays responsive is far more valuable than a server which keeps consuming work until it inevitably fails.

This article uses select() to keep the event loop simple and portable, making the architecture easier to explain. However, the same considerations apply to epoll, kqueue, Boost.Asio, and other event-loop frameworks. Please note that select() is suitable for relatively small servers only. Linux epoll or BSD/macOS kqueue serve a similar purpose to select(), but scale better when monitoring large numbers of sockets/file descriptors. Now let’s look at the key concepts such as concurrency, ownership, and backpressure.

There are several key principles behind this architecture.

First, the event loop must be light-weight. Its job is to accept new connections, receive data from the sockets [Linux-10], assemble complete request frames, dispatch work to the bounded queue, collect completed responses, and write responses back to the clients. Event loop is fully responsible for managing the lifetime of connections and doing all network I/O. This avoids many synchronization issues and eliminates races caused by concurrent socket reads, writes, or closure. The event loop must never do anything expensive like executing any business logic or performing any blocking operations. Shortening the I/O path gives us enough flexibility to keep the server responsive even when the workers are busy.

Second, we want networking and application logic to be separated. Request processing is performed by a worker pool using modern C++ concurrency primitives such as std::jthread, cooperative cancellation with std::stop_token, and RAII-based thread lifetime management. Workers are only dealing with application data and never touch the socket state directly.

Third, admission must be bounded. Accepting each new request increases the resource utilization (memory consumption, scheduling time, CPU). Accepting new requests forever just shifts the overload from network onto application memory where it grows in queueing delay until latency becomes unacceptably high. Bounded queue forces us to take an explicit decision whenever we run out of capacity like whether to reject new requests, to admit them with delay, to apply some kind of rate-limiting or to drop less important work. Overload becomes a controlled state of our application rather than an accidental outcome of the unlimited buffering.

Finally, cooperative cancellation and observability are important parts of our design. They allow workers to finish their work without accepting any new requests, and expose metrics at each step of request processing. Waiting in the queue, processing time, total latency, queue size and request rejection rates give us much more information about the health of our server than its throughput, helping us to spot overload early on.

One important aspect of our architecture that is often overlooked is that there are two different paths of communication with completely different concurrency semantics. Our request queue has SPMC (single-producer multiple-consumer) model: the event loop adds requests to the request queue, and multiple worker threads process them. The response queue, on the contrary, uses MPSC (multiple producer single consumer) model: multiple workers add completed responses to the response queue, and the event loop retrieves them and sends them to the client’s sockets.

While special queue implementations may optimize both communication patterns separately, the reusable bounded queue constructed with the help of modern C++ synchronization primitives such as std::mutex, std::condition_variable, and RAII locking often turns out to be the most convenient solution providing a great balance between simplicity and correctness. As the server grows, a single event-loop thread may eventually become a bottleneck, requiring a multi-loop architecture. However, multi-loop design is outside the scope of this article. A reusable queue abstraction makes such future changes easier without requiring a redesign of the overall architecture.

At the end of the day, our goal is not to build the fastest server but the most reliable and observable one. These features do not come from any specific I/O multiplexing API but from explicit ownership, bounded resources, clearly defined concurrency boundaries, and overload policies.

System architecture overview

The overall architecture is intentionally simple, with each component responsible for a single well-defined task, as illustrated in Figure 1. Incoming connections are managed by a single event loop built around select(). The event loop parses incoming requests, places them onto a bounded work queue, and later delivers completed responses back to clients. All application-specific processing is delegated to a pool of worker threads, while operational metrics are collected throughout the request lifecycle.

The most important design decision is ownership. The event loop is the sole owner of every connection. It is responsible for accepting new connections, reading incoming data, managing socket state, writing responses, and closing connections. Worker threads never interact with sockets directly; they operate exclusively on parsed requests and produce completed responses. This clear separation significantly reduces synchronization complexity and eliminates an entire class of race conditions associated with concurrent reads, writes, and connection teardown.

All sockets are created in non-blocking mode. On Linux, accept4() is the preferred mechanism because it allows newly accepted sockets to be created atomically with both SOCK_NONBLOCK and SOCK_CLOEXEC. Relying on accept() followed by fcntl() introduces an unnecessary race window, and accepted sockets do not reliably inherit the O_NONBLOCK flag from the listening socket. Configuring non-blocking behavior at creation time results in a simpler and more robust implementation. [Linux-1] [Linux-4]

With non-blocking sockets, operations such as recv() and send() may complete only partially or return EAGAIN (or EWOULDBLOCK) when no progress can be made immediately. Rather than treating these conditions as errors, the event loop simply resumes processing other connections and retries the operation when the socket becomes ready again. This event-driven model allows a single thread to manage many concurrent connections without blocking on any individual client. [Linux-6] [Linux-8]

A natural consequence of this design is that all network writes remain centralized within the event loop. This decision is motivated by correctness rather than performance. Stream-oriented sockets make no guarantee that a call to send() will transmit the entire response, and preserving response ordering while handling partial writes quickly becomes difficult if multiple worker threads are permitted to write directly to the same socket. By routing completed responses through a bounded response queue, the event loop becomes the single writer for every connection, ensuring ordered delivery while greatly simplifying synchronization and buffer management.

The architecture also benefits from treating the event loop as the central coordination point for the entire server. Besides monitoring client sockets, it can observe additional file descriptors that represent internal events. On Linux, an eventfd provides an efficient mechanism for waking the event loop from another thread, for example, during shutdown or configuration changes. timerfd integrates periodic timers and connection timeouts into the same event-driven infrastructure. Monitoring these descriptors through the existing select() loop avoids ad hoc synchronization mechanisms and eliminates the need to interrupt the event loop by closing sockets from another thread, a behavior that POSIX leaves unspecified and that Linux documentation explicitly discourages relying upon. [Linux-3] [Linux-12]

Although the implementation presented in this article uses select() for clarity, the overall architecture is intentionally independent of any particular I/O multiplexing mechanism. Changing the I/O multiplexing mechanism affects the event-notification layer, but not the fundamental design. The fundamental design remains unchanged: a lightweight event loop owns every connection, bounded queues regulate the flow of work, worker threads execute application logic, and explicit ownership keeps synchronization predictable and maintainable.

Implementation of core components

Now that we have the architecture in place, the implementation of other parts is much easier. Implementation includes three parts that cooperate with each other: a bounded queue to manage the workload, a worker pool that processes the workload and an event loop that manages all connections. All the parts have clear responsibilities, thus simplifying the development process and testing as well.

The initial implementation will be done with simplicity and correctness in mind rather than optimizing for maximum performance. Modern C++ includes very efficient mechanisms for synchronization, which are quite good enough for server applications. Complexity can be added only when measurements show if it is needed.

The modern bounded queue

The bounded queue provides the concurrency barrier between the event loop and the worker pool. Its role is equally simple yet vital like transferring the work from one thread to another and limiting the amount of work the application can accept.

While lock-free queues are frequently used in performance-oriented applications, their use comes at the cost of increased complexity in memory ordering, reclamation, and correct operation. It is usually better to first ask the simpler question: does profiling reveal that the queue is the problem area?

In most cases, the performance bottleneck of the system lies elsewhere for example, be it the application itself, database operations, other services, or network latency. A properly designed bounded queue using standard synchronization mechanisms in C++ is thus an excellent start.

Modern C++ allows such an implementation to be both brief and clear. The bounded queue uses std::mutex, std::condition_variable, RAII-style locking, and std::optional to allow blocking consumers, bounded producers, and graceful shutdown of the application. The bounded queue also has another meaning besides serving as a synchronization primitive: the bounded queue defines admission boundary of the application. Once the queue gets full, some kind of decision has to be made: whether to reject new work, to defer admission, to apply backpressure, or to shed lower priority work. By making overload visible, queue helps to build resiliency of the server rather than simply acting as a transport between threads.

The implementation in Listing 1 is an example of bounded queue implemented using modern C++ synchronization primitives.

template<typename T>
class BoundedQueue
{
public:
  explicit BoundedQueue(std::size_t capacity);

  [[nodiscard]]
  bool try_push(T value);

  template<class Rep, class Period>
  [[nodiscard]]
  bool push_for(
    T value,
    std::chrono::duration<Rep, Period> timeout);

  [[nodiscard]]
  std::optional<T> pop();

  [[nodiscard]]
  std::optional<T> try_pop();
  void close();

  [[nodiscard]]
  std::size_t size() const;

  private:
  std::deque<T> queue_;
  mutable std::mutex mutex_;
  std::condition_variable not_empty_;
  std::condition_variable not_full_;
  std::size_t capacity_;
  bool closed_{false};
};

The implementation intentionally omits lock-free techniques for now. Should profiling later demonstrate that queue contention has become a measurable bottleneck, the implementation can be replaced without affecting the surrounding architecture because the queue already provides a clean abstraction boundary.

Building the worker pool

After requests go into the bounded queue, the job is done by the worker pool exclusively. The workers take requests from the queue, process an application-specific handler, and put responses in the response queue to transmit via the event loop.

There is a modern C++ solution to traditional thread pools created using std::thread in the form of std::jthread that has the automatic joining on destruction feature and the cooperative cancellation mechanism provided by std::stop_token. (See Listing 2.)

struct Request
{
  int fd;
  std::string payload;
  std::chrono::steady_clock::time_point admitted;
};

struct Response
{
  int fd;
  std::string payload;
  bool close_after_write{false};
};

class WorkerPool
{
public:
  using Handler = 
    std::move_only_function<Response(Request)>;
  WorkerPool(std::size_t workers,
             BoundedQueue<Request>& requests,
             BoundedQueue<Response>& responses,
             Handler handler);

private:
  void run_worker(std::stop_token token);

  std::vector<std::jthread> workers_;

  BoundedQueue<Request>& requests_;
  BoundedQueue<Response>& responses_;

  Handler handler_;
};

Equally important is what the workers do not do. Worker threads do not access sockets directly. They do not call recv() and send(), and they do not close client connections. Their only job is to transform requests into responses. This separation significantly simplifies synchronization and allows the networking and application logic to evolve independently.

The response queue is naturally multiple-producer, single-consumer (MPSC) topology because there are many producers (worker threads) and one consumer (the event loop).

The event loop

The event loop contains most of the correctness problems in the design of network servers. Luckily, having some simple ownership rules makes the implementation much easier.

The event loop owns each socket during its whole lifecycle. It accepts new connections, takes data from them, reconstructs whole request frames, sends them to the bounded request queue, receives completed responses from the worker pool, writes back to clients, and finally closes connections when it is necessary. There is no way for any other thread to change the socket state.

Such an ownership model is very helpful when using non-blocking sockets. Calls to recv() and send() may process only part of the data or return EAGAIN (or EWOULDBLOCK) when the operation cannot complete immediately. Such behavior is a regular case of event-driven processing.

Moreover, select() requires some additional conditions to be met. First of all, descriptor sets are changed in place and should be reinitialized before each select() call. In Linux OS, the same applies to the timeout structure. Otherwise, there will be some hard-to-detect bugs.

With the help of modern C++, this code becomes more readable but does not change anything in the semantics. One could express timeouts with std::chrono, manage resources with RAII, use cooperative shutdown with std::stop_token.

Listing 3 is the simplified example of the event loop.

using namespace std::chrono_literals;

constexpr auto poll_timeout = 200ms;

while (!stop_token.stop_requested())
{
  rebuild_descriptor_sets();
  drain_completed_responses();
  auto timeout = to_timeval(poll_timeout);
  const int ready =
    ::select(max_fd + 1,
             &read_set,
             &write_set,
             nullptr,
             &timeout);

  if (ready < 0)
  {
    if (errno == EINTR)
        continue;
      throw std::system_error(
        errno,
        std::system_category());
  }

  accept_new_connections();
  read_requests();
  write_pending_responses();
}

Despite being highly oversimplified, this diagram (Figure 1) represents the basic event loop architecture. Its full-fledged implementation is just an expansion of every step with keeping the same ownership rules.

Why centralized I/O is important

One could try to give permission to every worker thread to write its response to the client socket. Although this might seem as decreasing of queuing overhead, it will make synchronization much more complicated.

The stream sockets don’t guarantee that the whole buffer would be sent during one send() call. Partial sends happen quite often, especially in high-load conditions. Allowing multiple worker threads to perform sending responses would make it difficult to preserve responses order and to handle partially sent buffers.

Routings of every completed response through the response queue makes the event loop the single writer for every connection. The buffer handling, partial sends, connections shutting down, and responses order remain in one module, which makes this architecture more understandable and debuggable.

Event loop cooperation

While observing client sockets’ activity, the event loop can watch other events as well. On the Linux platform an eventfd can be used to notify the event loop from other thread, while timerfd helps to integrate timers and connection timeout handling into the event-driven system.

It allows avoiding the undefined behavior by trying to wake up the select() with closing of the monitored descriptor in another thread. The event loop continues to own the lifetime of every socket, while wake-ups and shutting down requests go through synchronization primitives.

Design takeaways

These principles are independent of the particular I/O multiplexing mechanism.

It is this clear separation of responsibilities, which makes the architecture resilient. Event loop takes care of communication, the worker pool takes care of computing, the queues manage the work distribution, and every module has a single responsibility. It leads to creation of the predictable, measurable, and maintainable server.

Backpressure and sizing

Backpressure is not a single mechanism; it is a stack of mechanisms applied at different layers of the system. At the application layer, the server can reject work, defer admission briefly, apply rate limits, or shed lower-priority requests. At the connection layer, it can stop reading from a socket or close the connection. At the transport layer, TCP already provides flow control through receiver windows, retransmission behavior, and zero-window probing.

For HTTP services, overload also has well-defined protocol-level semantics. 503 Service Unavailable is appropriate when the service as a whole is temporarily overloaded, often with a short Retry-After header. 429 Too Many Requests is better reserved for per-client, per-user, or per-principal throttling. 431 Request Header Fields Too Large is useful when admission should fail before work reaches the application layer because the request itself is too large. [IETF-1]

A practical comparison of overload strategies looks like Table 1.

A good rule of thumb for most HTTP servers would be the following: immediately drop a connection when the request queue becomes full; respond with the 503 Service Unavailable status code and a short Retry-After header when overloaded globally; use 429 Too Many Requests for per-client or per-principal overload throttling. [IETF-3]

There are a number of TCP-level tweaks available as well, although they can’t be seen as a substitute for proper admission control at the application level. TCP_NODELAY, for instance, reduces latency for short responses by disabling Nagle’s algorithm [IETF-2]. TCP_DEFER_ACCEPT allows to wake up the server only when data comes into the listening socket. TCP_INFO exposes a bunch of useful Linux TCP statistics. [Linux-11]

On the other hand, more aggressive TCP tweaks like aborting connections that have grown too long are not a proper alternative to the combination of bounded request queues, quantifiable latency and overload handling policy.

When sizing request queues, the starting point should be the latency goals that the server has to achieve. In particular, the Little’s Law [Wikipedia] of queueing theory provides a convenient starting point: L = λW, where L is the average queue length, λ is the arrival rate, and W is the average queue waiting time.

When expressed in terms of W = L/λ, the equation means that the queue size should be set up taking into account both the arrival rate of requests and how much time the server is ready to spend waiting before serving those requests.

Such a relation proves to be a handy starting point in capacity planning. With a known arrival rate and allowable queueing delay, an initial queue-size estimate can be derived. On the other hand, if the queue capacity is fixed, then the wait will grow linearly with the load.

The above relation is an approximation. Real-life loads tend to be bursty, service times are not constant, and p95/p99 (95th and 99th percentile) latencies can be substantially higher than the mean. Nonetheless, Little’s Law gives a much more defensible starting point compared to ‘a few thousands of items just in case’. The queue size can later be tuned through burst tests and adjusted using the measurements of queue waiting time, service time, end-to-end latency and rejection ratio. [Little11]

Similarly, the number of worker threads should be selected taking into consideration the expected service time. If the server spends most of its time waiting for the completion of the tasks on other hosts, its scalability will be quite different from that of CPU-bound processes.

A small overload helper keeps the admission policy explicit (Listing 4).

enum class OverloadPolicy
{
  reject_immediately,
  defer_briefly,
  shed_low_priority
};

[[nodiscard]]
bool admit_or_overload(
  BoundedQueue<Request>& queue,
  Request request,
  OverloadPolicy policy,
  Connection& connection)
{
  using namespace std::chrono_literals;
  auto reject = [&connection] {
    connection.out =
      "HTTP/1.1 503 Service Unavailable\r\n"
      "Connection: close\r\n"
      "Retry-After: 1\r\n"
      "Content-Length: 0\r\n\r\n";
    connection.close_after_write = true;
    return false;
  };

  switch (policy)
  {
    case OverloadPolicy::reject_immediately:
      if (!queue.try_push(std::move(request)))
        return reject();
      return true;

    case OverloadPolicy::defer_briefly:
      if (!queue.push_for(std::move(request),
                          2ms))
        return reject();
      return true;

      case OverloadPolicy::shed_low_priority:
        // In a real system, Request would carry
        // priority, tenant, endpoint, or cost
        // metadata. Low-priority work can then
        // be refused before critical work.
        if (!queue.try_push(std::move(request)))
          return reject();
        return true;
  }
  return reject();
}

The essential thing is not the policy itself which is illustrated above, but that there is an explicit policy at all. Overloading must not be an unintended consequence of infinite queues, kernel buffer storage, or blocking invocations. It must be an intended component of the server’s behavior, measurable in metrics and discernible in the code.

Measurement, observability, and testing

It is impossible to understand a resilient server by reporting just one number, called latency. There are several distinct phases for processing a request that represent different aspects of the server’s behavior. Only the measurement of total response time makes it hard to figure out where the latency occurs and to troubleshoot performance problems.

At least, there should be three independent measures of latency:

• Time in the queue: the period between receiving a request and starting its processing by a worker.
• Service time: time that an application logic executes in the context of the worker thread.
• Total latency: total processing time since the moment when a complete request is received until all bytes of the response have been written to the client.

Each measure reveals a different story. The growth of latency in the queue means that requests arrive too quickly. The increase in service time means that either an application takes more time to execute or it calls a slow external component. The end-to-end latency measure shows how the client sees the server.

Modern C++ allows such instrumentation at low cost. A tiny structure is enough to collect all the necessary timestamps:

  using Clock = std::chrono::steady_clock;
  
  struct Timings
  {
    Clock::time_point admitted;
    Clock::time_point worker_start;
    Clock::time_point worker_done;
    Clock::time_point response_sent;
  };
  // queue_wait   = worker_start - admitted
  // service_time = worker_done  - worker_start
  // end_to_end   = response_sent - admitted

Instead of merely keeping the averages, each measurement needs to be logged into a latency histogram. HdrHistogram [HdrH] is a great open-source tool for doing so since it offers fixed memory logging along with percentile queries which make collecting such histograms feasible in high throughput systems.

Failure modes, shutdown, and security

Several failure modes emerge quite often when implementing hand-written select()-based servers. The first one is that one may assume that a socket, which was reported as readable or writeable, will pass the whole amount of requested data. In reality, recv() can return only the available bytes, and send() may write only part of the data or return EAGAIN on non-blocking sockets. select() itself can return EINTR, and descriptor sets along with timeouts should always be re-built on each iteration. Handling those situations as part of normal execution flow is crucial for creating solid event loops.

Another common issue is that sockets should never be closed from worker threads, because it’s undefined behavior according to POSIX, and Linux does not wake up select() at all in that case. Instead, workers should inform event loop about need to close the connection, and I/O thread will handle removing the descriptor, performing proper shutdown() and closing the socket using close(). Including an eventfd into the event loop helps here to wake up event loops from those coordination messages. [Linux-9]

Graceful shutdown should be taken into account in the architecture too. The server should stop accepting new connections, process all outstanding requests, wait until queues become empty or until some shutdown timeout occurs, flush all pending responses, and shut down connections. Sending shutdown(fd,SHUT_WR) will ensure that any data will be sent out before the socket will be closed, and MSG_NOSIGNAL flag will prevent unnecessary generation of SIGPIPE on writing to already closed socket.

Timeouts should be handled at the level of event loop too, instead of trying to rely on socket-level receive or send timeouts. On Linux, timerfd fits naturally into event-driven architecture, and TCP_USER_TIMEOUT option will allow to control how long stalled peer connections will be kept alive. Similar way, failures at the level of the listening socket should be fixed by optimizing application-level latency, tuning sizes of queues or increasing processing capabilities of application before trying any TCP-level tuning. [Linux-5]

Security is a concern which starts much earlier than applying TLS to a server. Robust request framing and parsing will help to avoid processing of malformed or ambiguous requests. Request headers should have limits, conflicting Content-Length and Transfer-Encoding headers should be rejected, oversized requests should fail with appropriate HTTP status code, like 431 Request Header Fields Too Large. As far as internet-facing services are concerned, TLS should be considered default transport either by the server itself, or by the reverse proxy. [IETF-4]

Recommendations for defaults and production checklist

The proposed architecture deliberately emphasizes simplicity, clarity of ownership, and predictable behavior over excessive optimizations. On Linux, new sockets returned by accept4() should have the flags SOCK_NONBLOCK and SOCK_CLOEXEC set in order to ensure atomicity of both non-blocking mode and close-on-exec semantics. Request and response queues should always have a known finite capacity, which can be calculated on the basis of latency goals of the application. Size of worker pool should be derived from the measurements of service time and adjusted via workload measurements, not request rate. For HTTP-based services, 503 Service Unavailable together with short Retry-After header is a good choice as the default response in the case of global overload of the server, while 429 Too Many Requests is more suitable for the client-based rate limiting.

It is recommended to use the flag MSG_DONTWAIT for keeping the writing process non-blocking and MSG_NOSIGNAL to suppress generation of SIGPIPE when clients unexpectedly close connections. It is usually easier to manage idle connection deadlines within the event loop than via timeouts on the socket level and thus, the former method allows the timeout mechanism to be integrated into the rest of the server architecture. The evaluation of performance should be based on the tool, which is able to generate requests with constant rate, and it is recommended to record latency separately as queue wait time, service time and end-to-end response time histograms.

Before putting a server implemented according to this architecture into production, it is necessary to check several key assumptions about it. All sockets must be configured for non-blocking operations, and the event loop should rebuild its select() descriptor sets and timeouts at the beginning of every iteration. The event loop should be the only owner of socket read/write operations and connection lifetime management, while the worker threads should deal with application data only. The request and response queues should have a known finite capacity, the overload policy should be explicitly specified, and the latency should be measured separately as queue wait time, service time and end-to-end response time.

Request framing should be checked strictly and oversized headers must be rejected properly. Transport security model, in particular, termination point of TLS protocol must be specified – either the application itself or reverse proxy acts as SSL/TLS terminator. Ultimately, implementation details do not matter much in comparison with key principles of the architecture. Keep all queues bounded, make overload evident, provide the single-threaded ownership of connection state, measure queueing latency separately from the service time – those principles will be beneficial for any server regardless of what event loop API is used.

Conclusion

Designing an event-driven server is not just about achieving maximum performance; it is about ensuring consistent behavior once demand begins to exceed processing capabilities. The separation of concerns of network I/O operations and the actual logic, bounded queue sizes, backpressure, and clear delineation of ownership of connections are all necessary elements in developing a highly resilient system. Combined with the capabilities provided by modern C++ concurrency constructs and observability, these practices can result in responsive and measurable systems. While select() is used here to demonstrate the concepts, the same architecture can be applied to epoll, kqueue, and Boost.Asio. The technologies and APIs may change over time, but the fundamental principles of limited resources, admission control, cooperative concurrency, and measurement will always be at the core of building resilient event-driven systems.

References

[Boost] Boost C++ Libraries Overview: https://www.boost.org/doc/libs/latest/doc/html/boost_asio/overview.html

[HdrH] High Dynamic Range Histogram: https://hdrhistogram.github.io/HdrHistogram/

[IETF-1] RFC 6585 – ‘Additional HTTP Status Code’, available at: https://datatracker.ietf.org/doc/html/rfc6585

[IETF-2] RFC 896 – ‘Nagle’s algorithm’, available at: https://datatracker.ietf.org/doc/html/rfc896

[IETF-3] RFC 9110 – ‘HTTP Semantics’, available at: https://datatracker.ietf.org/doc/html/rfc9110

[IETF-4] RFC 9112 – ‘HTTP/1.1’, available at: https://datatracker.ietf.org/doc/html/rfc9112

[IETF-5] RFC 9293 – ‘Transmission Control Protocol (TCP)’, available at: https://datatracker.ietf.org/doc/rfc9293/

[Little11] John D. C. Little ‘Little’s Law as Viewed on Its 50th Anniversary’ in Operations Research 59:3, published May-June 2011, available at https://people.cs.umass.edu/~emery/classes/cmpsci691st/readings/OS/Littles-Law-50-Years-Later.pdf

[Linux-1] accept(), Linux manual page, available at: https://man7.org/linux/man-pages/man2/accept.2.html

[Linux-2] epoll(), Linux manual page, available at: https://man7.org/linux/man-pages/man7/epoll.7.html

[Linux-3] eventfd(), Linux manual page, available at: https://man7.org/linux/man-pages/man2/eventfd.2.html

[Linux-4] fcntl(), Linux manual page, available at: https://man7.org/linux/man-pages/man2/fcntl.2.html

[Linux-5] listen(), Linux manual page, available at: https://man7.org/linux/man-pages/man2/listen.2.html

[Linux-6] recv(), Linux manual page, available at: https://man7.org/linux/man-pages/man2/recv.2.html

[Linux-7] select(), Linux manual page, available at: https://man7.org/linux/man-pages/man2/select.2.html

[Linux-8] send(), Linux manual page, available at: https://man7.org/linux/man-pages/man2/sendmsg.2.html

[Linux-9] shutdown(), Linux manual page, available at: https://man7.org/linux/man-pages/man2/shutdown.2.html

[Linux-10] socket(), Linux manual page, available at: https://man7.org/linux/man-pages/man7/socket.7.html

[Linux-11] TCP, Linux manual page, available at: https://man7.org/linux/man-pages/man7/tcp.7.html

[Linux-12] timerfd_create(), Linux manual page, available at: https://man7.org/linux/man-pages/man2/timerfd_create.2.html

[OpenBSD] kqueue(), OpenBSD manual page, available at: https://man.openbsd.org/kqueue.2

[Wikipedia] Little’s Law theorem: https://en.wikipedia.org/wiki/Little%27s_law

Ajay Pandey is a software engineering leader and distributed systems architect specializing in modern C++, concurrency, and high-frequency event-driven systems. He has over two decades of experience developing C++ backend application servers for Fixed Income trading systems and Intelligent Network platforms in the telecom industry. His interests include scalable architectures, low-latency engineering, event coalescing, and resilient real-time platforms. You can reach him at ajaypandey1@gmail.com or through his LinkedIn profile: https://www.linkedin.com/in/ajay-pandey-76158310/