C++ Exceptions and Memory Allocation Failure

Background

C++ exceptions are habitually disabled in many software projects. A related issue is that these projects also encourage the use of new (nothrow) instead of the more common new, as the latter may throw an exception. This choice is kind of self-deceptive, as people don’t usually disable completely all mechanisms that potentially throw exceptions, such as standard library containers and string. In fact, every time we initialize or modify a string, vector, or map, we may be allocating memory on the heap. If we think that new will end in an exception (and therefore choose to use new (nothrow)), exceptions may also occur when using these mechanisms. In a program that has disabled exceptions, the result will inevitably be a program crash.

However, it seems that the crashes I described are unlikely to occur… When was the last time you saw a memory allocation failure? Before I tested to check this issue, the last time I saw a memory allocation failure was when there was a memory corruption in the program: there was still plenty of memory in the system, but the memory manager of the program could no longer work reliably. In this case, there was already undefined behaviour, and checking for memory allocation failure ceased to make sense. A crash of the program was inevitable, and it was a good thing if the crash occurred earlier, whether due to an uncaught exception, a null pointer dereference, or something else.

Now the question is: If there is no undefined behaviour in the program, will memory allocation ever fail? This seems worth exploring.

Test of memory allocation failure

Due to the limitation of address space, there is an obvious upper limit to the amount of memory that one process can allocate. On a 32-bit system, this limit is 2³² bytes, or 4 GiB. However, on typical 64-bit systems like x64, this limit is not 2⁶⁴ bytes, but 2⁴⁸ bytes instead, or 256 TiB.

On a 32-bit system, when a process’s memory usage reaches around 2 GiB (it may vary depending on the specific system, but will not exceed 4 GiB), memory allocation is guaranteed to fail. The physical memory of a real-world 32-bit system can often reach 4 GiB, so we do expect to see memory allocation failures.

The more interesting question is: What happens when the amount of physical memory is far less than the address space size? Ignoring abnormal scenarios like allocating more memory than the physical memory size at a time (which would likely be a logic error in the program), can a reasonable program still experience memory allocation failures?

The core logic of my test code is shown in Listing 1.

try {
  std::size_t total_alloc = 0;
  for (;;) {
    char* ptr = new char[chunk_size];
    if (zero_mem) {
      memset(ptr, 0, chunk_size);
    }
    total_alloc += chunk_size;
    std::cout << "Allocated "
              << (zero_mem ? "and initialized "
                           : "")
              << total_alloc << " B\n";
  }
}
catch (std::bad_alloc&) {
  std::cout << "Successfully caught bad_alloc "
               "exception\n";
}

The program allocates memory repeatedly – optionally zeroing the allocated memory – until it catches a bad_alloc exception.

(I did not change the new-handler [CppReference-1], as I do not usually do this in projects, and it is not helpful in testing whether memory allocation failures can really happen. When the new-handler is invoked, memory allocation has already failed – unless the new-handler can free some memory and make allocation succeed, it can hardly do anything useful.)

The test shows that Windows and Linux exhibit significantly different behaviour in this regard. These two are the major platforms concerned, and macOS behaves similarly to Linux.

Windows

I conducted the test on Windows 10 (x64). According to the Microsoft documentation, the total amount of memory that an application can allocate is determined by the size of RAM and that of the page file. When managed by the OS, the maximum size of the page file is three times the size of the memory, and cannot exceed one-eighth the size of the volume where the page file resides [Microsoft]. This total memory limit is shared by all applications.

The program’s output is shown below (allocating 1 GiB at a time on a test machine with 6 GiB of RAM):

  Allocated 1 GiB
  Allocated 2 GiB
  Allocated 3 GiB
  …
  Allocated 14 GiB
  Allocated 15 GiB
  Successfully caught bad_alloc exception
  Press ENTER to quit

The outputs are the same, regardless of whether the memory is zeroed or not, but zeroing the memory makes the program run much slower. You can observe in the Task Manager that the memory actually used by the program is smaller than the amount of allocated memory, even when the memory is zeroed; and that when the amount of allocated (and zeroed) memory gets close to that of available memory, the program’s execution is further slowed down, and disk I/O increases significantly – Windows starts paging in order to satisfy the program’s memory needs.

As I mentioned a moment ago, there is an overall memory limit shared by all applications. If a program encounters a memory allocation failure, other programs will immediately experience memory issues too, until the former one exits. After running the program above, if I don’t press the Enter key to quit, the results of newly opened programs are as follows (even if the physical memory usage remains low):

  Successfully caught bad_alloc exception
  Press ENTER to quit

Assuming that a program does not allocate a large amount of memory and only uses a small portion (so we exclude some special types of applications, which will be briefly discussed later), when it catches a memory allocation failure, the total memory allocated will be about 4 times the physical memory, and the system should have already slowed down significantly due to frequent paging. In other words, even if the program can continue to run normally, the user experience has already been pretty poor.

Linux

I conducted the test on Ubuntu Linux 22.04 LTS (x64), and the result was quite different from Windows. If I do not zero the memory, memory allocation will only fail when the total allocated memory gets near 128 TiB. The output below is from a run which allocates 4 GiB at a time:

  Allocated 4 GiB
  Allocated 8 GiB
  Allocated 12 GiB
  …
  Allocated 127.988 TiB
  Allocated 127.992 TiB
  Successfully caught bad_alloc exception
  Press ENTER to quit 

In other words, the program can catch the bad_alloc exception only when it runs out of memory address…

Another thing different from Windows is that other programs are not affected if memory is allocated but not used (zeroed). A second copy of the test program still gets close to 128 TiB happily.

Of course, we get very different results if we really use the memory. When the allocated memory exceeds the available memory (physical memory plus the swap partition), the program is killed by the Linux OOM killer (out-of-memory killer). An example run is shown below (on a test machine with 3 GiB memory, allocating 1 GiB at a time):

  Allocated and initialized 1 GiB
  Allocated and initialized 2 GiB
  Allocated and initialized 3 GiB
  Allocated and initialized 4 GiB
  Allocated and initialized 5 GiB
  Allocated and initialized 6 GiB
  Killed

The program had successfully allocated and used 6 GiB memory, and was killed by the OS when it was initializing the 7th chunk of memory. In a typical 64-bit Linux environment, memory allocation will never fail – unless you request for an apparently unreasonable size (possible only for new Obj[size] or operator new(size), but not new Obj). You cannot catch the memory allocation failure.

Modify the overcommit_memory setting?

We can modify the overcommit_memory setting [Kernel], you probably have shouted out. What I described above was the default Linux behaviour, when /proc/sys/vm/overcommit_memory was set to 0 (heuristic overcommit handling). If its value is set to 1 (always overcommit), memory allocation will always succeed, as long as there is enough virtual memory address space: you can successfully allocate 32 TiB memory on a machine with only 32 GiB memory – this can actually be useful for applications like sparse matrix computations. Yet another possible value is 2 (don’t overcommit), which allows the user to fine-tune the amount of allocatable memory, usually with the help of /proc/sys/vm/overcommit_ratio.

In the don’t-overcommit mode, the default overcommit ratio (a confusing name) is 50 (%), a quite conservative value. It means the total address space commit for the system is not allowed to exceed swap + 50% of physical RAM. In a general-purpose Linux system, especially in the GUI environment, this mode is unsuitable, as it can cause applications to fail unexpectedly. However, for other systems (like embedded ones) it might be the appropriate mode to use, ensuring that applications can really catch the memory allocation failures and that there is little (or no) thrashing.

(Before you ask, no, you cannot, in general, change the overcommit setting in your code. It is global, not per process; and it requires the root privilege.)

Summary of memory allocation failure behaviour

Looking at the test results above, we can see that normal memory allocations will not fail on general Linux systems, but may fail on Windows or special-purpose Linux systems that have turned off overcommitting.

Strategies for memory allocation failure

We can classify systems into two categories:

• Those on which memory allocation will not fail
• Those on which memory allocation can fail

The strategy for the former category is simple: we can simply ignore all memory allocation failures. If there were errors, it must be due to some logic errors or even undefined behaviour in the code. In such a system, you cannot encounter a memory allocation failure unless the requested size is invalid (or when the memory is already corrupt). I assume you must have checked that size is valid for expressions like new Obj[size] or malloc(size), haven’t you?

The strategy for the latter category is much more complicated. Depending on the requirements, we can have different solutions:

1. Use new (nothrow), do not use the C++ standard library, and disable exceptions. If we turned off exceptions, we would not be able to express the failure to establish invariants in constructors or other places where we cannot return an error code. We would have to resort to the so-called ‘two-phase construction’ and other techniques, which would make the code more complicated and harder to maintain. However, I need to emphasize that notwithstanding all these shortcomings, this solution is self-consistent – troubles for robustness – though I am not inclined to work on such projects.
2. Use new (nothrow), use the C++ standard library, and disable exceptions. This is a troublesome and self-deceiving approach. It brings about troubles but no safety. If memory is really insufficient, your program can still blow up.
3. Plan memory use carefully, use new, use the C++ standard library, and disable exceptions; in addition, set up recovery/restart mechanisms for long-running processes. This might be appropriate for special-purpose Linux devices, especially when there is already a lot of code that is not exception-safe. The basic assumption of this scenario is that memory should be sufficient, but the system should still have reasonable behaviour when memory allocation fails.
4. Use new (nothrow), use the C++ standard library, and enable exceptions. When the bad_alloc exception does happen, we can catch it and deal with the situation appropriately. When serving external requests, we can wrap the entire service code with try ...catch, and perform rollback actions and error logging when an exception (not just bad_alloc) occurs. This may not be the easiest solution, as it requires the developers know how to write exception-safe code. But neither is it very difficult, if RAII [CppReference-2] is already properly used in the code and there are not many raw owning pointers. In fact, refactoring old code with RAII (including smart pointers) can be beneficial per se, even without considering whether we want exception safety or not.

Somebody may think: Can we modify the C++ standard library so that it does not throw exceptions? Let us have a look what a standard library that does not throw exceptions may look like.

Standard library that does not throw?

If we do not use exceptions, we still need to have a way to express errors. Traditionally we use error codes, but these have the huge problem that a universal way does not exist: errno encodes errors in its way, your system has your way, and yet a third-party library may have its own way. When you put all things together, you may find that the only thing in common is that 0 means successful…

Assuming that you have solved the problem after tremendous efforts (make all subsystems use a single set of error codes, or adopt something like std::error_code [CppReference-3]), you will still find yourself with the big question of when to check for errors. Programmers that have been used to the standard library behaviour may not realise that using the following vector is no longer safe:

  my::vector<int> v{1, 2, 3, 4, 5};

The constructor of vector may allocate memory, which may fail but it cannot report the error. So you must check for its validity when using v. Something like:

  if (auto e = v.error_status();
      e != my::no_error) {
    return e;
  }
  use(v);

OK… At least a function can use an object passed in by reference from its caller, right?

  my::error_t process(const my::string& msg)
  {
    use(msg);
    …
  }

Naïve! If my::string behaves similarly to std::string and supports implicit construction from a string literal – i.e. people can call this function with process("hello world!") – the constructor of the temporary string object may fail. If we really intend to have complete safety (like in Solution 1 above), we need to write:

  my::error_t process(const my::string& msg)
  {
    if (auto e = msg.error_status();
        e != my::no_error) {
      return e;
    }
    use(msg);
    …
  }

And we cannot use overloaded operators if they may fail. vector::operator[] returns a reference, and it is still OK. map::operator[] may create new map nodes, and can cause problems. Code like the following needs to be rewritten:

  class manager {
  public:
    void process(int idx, const std::string& msg)
    {
      store_[idx].push_back(msg);
    }
  private:
    std::map<int, std::vector<string>> store_;
  };

The very simple manager::process would become many lines in its exception-less and safe version (Listing 2).

class manager {
public:
  error_t process(int idx,
                  const my::string& msg)
  {
    if (auto e = msg.error_status();
        e != my::no_error) {
      return e;
    }
    auto* ptr =
      store_.find_or_insert_default(idx);
    if (auto e = store_.error_status();
        e != my::no_error) {
      return e;
    }
    ptr->push_back(msg);
    return ptr->error_status();
  }
  …
private:
  my::map<int, my::vector<string>> store_;
};

Ignoring how verbose it is, writing such code correctly seems more complicated than making one’s code exception-safe, right? It is not an easy thing just to remember which APIs will always succeed and which APIs may return errors.

And obviously you can see that such code would be in no way compatible with the current C++ standard library. The code that uses the current standard library would need to be rewritten, third-party code that uses the standard library could not be used directly, and developers would need to be re-trained (if they did not flee).

Recommended strategy

I would like to emphasize first that deciding how to deal with memory allocation failure is part of the system design, and it should not be just the decision of some local code. This is especially true if the ‘failure’ is unlikely to happen and the cost of ‘prevention’ is too high. (For similar reasons, we do not have checkpoints at the front of each building. Safety is important only when the harm can be higher than the prevention cost.)

Returning to the four solutions I discussed earlier, my order of recommendations is 4, 3, 1, and 2.

• Solution 4 allows the use of exceptions so that we can catch bad_alloc and other exceptions while using the standard library (or other code). You don’t have to make your code 100% bullet-proof right in the beginning. Instead, you can first enable exceptions and deal with exceptions in some outside constructs, without actually throwing anything in your code. When memory allocation failure happens, you can at least catch it, save critical data, print diagnostics or log something, and quit gracefully (a service probably needs to have some restart mechanism external to itself). In addition, exceptions are friendly to testing and debugging. We should also remember that error codes and exceptions are not mutually exclusive: even in a system where exceptions are enabled, exceptions should only be used for exceptional scenarios. Expected errors, like an unfound file in the specified path or an invalid user input, should not normally be dealt with as exceptions.
• Solution 3 does not use exceptions, while recognizing that memory failure handling is part of the system design, not deserving local handling anywhere in the code. For a single-run command, crashing on insufficient memory may not be a bad choice (of course, good diagnostics would be better, but then we would need to go to Solution 4). For a long-running service, fast recovery/restart must be taken into account. This is the second best to me.
• Solution 1 does not use exceptions and rejects all overhead related to exception handling, time- or space-wise. It considers that safety is foremost and is worth extra labour. If your project requires such safety, you need to consider this approach. In fact, it may be the only reasonable approach for real-time control systems (aviation, driving, etc.), as typical C++ implementations have a high penalty when an exception is really thrown.
• Solution 2 is the worst, neither convenient nor safe. Unfortunately, it seems quite popular due to historical momentum, with its users unaware how bad it is…

Keep in mind that C++ is not C: the C-style check-and-return can look much worse in C++ than in C. This is because C++ code tends to use dynamic memory more often, which is arguably a good thing – it makes C++ code safer and more flexible. Although fixed-size buffers (common in C) are fast, they are inflexible and susceptible to buffer overflows.

Actually, the main reason I wanted to write this article was to point out the problems of Solution 2 and to describe the alternatives. We should not follow existing practices blindly, but make rational choices based on requirements and facts.

Test code

The complete code for testing the memory failure behaviour is available at either:

• http://wyw.dcweb.cn/mem_alloc_test.zip (for downloading)
• http://wyw.dcweb.cn/mem_alloc_test.cpp.html (for browsing)

You can clearly see that I am quite happy with exceptions. ☺

References

[CppReference-1] cppreference.com, std::set_new_handler, https://en.cppreference.com/w/cpp/memory/new/set_new_handler

[CppReference-2] cppreference.com, ‘RAII’, https://en.cppreference.com/w/cpp/language/raii

[CppReference-3] cppreference.com, std::error_code, https://en.cppreference.com/w/cpp/error/error_code

[Kernel] kernel.org, ‘Overcommit accounting’,https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

[Microsoft] Microsoft, ‘How to determine the appropriate page file size for 64-bit versions of Windows’, https://learn.microsoft.com/en-us/troubleshoot/windows-client/performance/how-to-determine-the-appropriate-page-file-size-for-64-bit-versions-of-windows

Wu Yongwei Having been a programmer and software architect, Yongwei is currently a consultant and trainer on modern C++. He has nearly 30 years’ experience in systems programming and architecture in C and C++. His focus is on the C++ language, software architecture, performance tuning, design patterns, and code reuse. He has a programming page at http://wyw.dcweb.cn/